tag:blogger.com,1999:blog-56258273093011208572024-02-20T15:13:50.972+00:00kynesimThis is the corporate blog for Kynesim Ltd - a technology development company in Cambridge, UK. We do fun stuff involving automotive, consumer electronics, IoT, and various local startups.Richardhttp://www.blogger.com/profile/15413093214412156645noreply@blogger.comBlogger74125tag:blogger.com,1999:blog-5625827309301120857.post-26534068987645251992017-06-19T11:29:00.004+01:002017-06-19T11:33:30.519+01:00External Entity Vulnerability in Expat 2.2.0 And Earlier<h2>
</h2>
(First posted on the <a href="https://libexpat.github.io/doc/cve-2017-9233/">LibExpat Documentation Site</a>) <br />
<br />
I'm in the intriguing position of being employed to work on a free software project by <a href="https://www.linuxfoundation.org/">the Linux Foundation</a>. I'm expanding the test coverage of the Expat library under the eagle eye of maintainer Sebastian Pipping, which is an excellent way of discovering how the brilliant, twisted and almost entirely undocumented internals work. (There will be articles. Many articles.)<br />
<br />
One thing you expect to come across when writing tests is the occasional bug. I came across a serious one in the parsing of external parameter entities. CVE-2017-9233 (to give it its formal identification) says that bad XML in an external entity will cause the parser to go into an infinite loop and never return control to the application.<br />
<br />
For those of you who haven't been saturated in XML terminology for the last however long, an example is in order. Suppose you have the following trivial piece of XML:<br />
<br />
<pre> <!DOCTYPE doc SYSTEM "http://example.com/level1.dtd">
<doc/>
</pre>
<br />
And the DTD <span style="font-family: "courier new" , "courier" , monospace;">level1.dtd</span> that it reads:<br />
<br />
<pre> <!ELEMENT doc EMPTY>
<!ENTITY % e SYSTEM "http://example.com/level2.ent">
%e;
</pre>
<br />
And the external entity definition in yet another resource, <span style="font-family: "courier new" , "courier" , monospace;">level2.ent</span>:<br />
<br />
<pre> <!ELEMENT el EMPTY>
<el/>
</pre>
<br />
Now a quick riffle through the XML standards will tell you that ordinary elements such as <span style="font-family: "courier new" , "courier" , monospace;"><el></span> aren't allowed in DTDs. All you can legally use are the DTD declarations <span style="font-family: "courier new" , "courier" , monospace;"><!ELEMENT></span>, <span style="font-family: "courier new" , "courier" , monospace;"><!ATTLIST></span>, <span style="font-family: "courier new" , "courier" , monospace;"><!ENTITY></span> and <span style="font-family: "courier new" , "courier" , monospace;"><!NOTATION></span>. When our entity <span style="font-family: "courier new" , "courier" , monospace;">%e</span> in level.dtd gets substituted in, it will put the <span style="font-family: "courier new" , "courier" , monospace;"><el/></span> element straight into the DTD, leaving us with a malformed DTD. The parser should detect this and reject the whole shebang.<br />
<br />
What actually happened is hard to follow in the source code. (<i>Many </i>articles.) When parsing reached the 'e' of <span style="font-family: "courier new" , "courier" , monospace;"><el/></span>, the tokenizer recognised it as a valid start of an ordinary element (which is true) and returned the appropriate value, <span style="font-family: "courier new" , "courier" , monospace;">XML_TOK_INSTANCE_START</span>.<br />
<br />
The calling code knew how to recognise the limited number of tokens that are legal and most of the tokens that aren't legal in a DTD, but unfortunately it missed this specific case. Without any better instructions from its decision logic, the code assumed that it had found something valid and tried tokenising again, in case there was more text to be parsed.<br />
<br />
It did this expecting that its internal pointers were updated to point to that unparsed text. That would have been true if it had dealt with something legal, but since "<e" isn't a legal part of a DTD the pointers had been left alone. The tokenizer therefore saw "<e" again, returned <span style="font-family: "courier new" , "courier" , monospace;">XML_TOK_INSTANCE_START</span> again, and the whole thing repeated until the application was killed.<br />
<br />
In brief:<br />
<ul>
<li>The lowest levels of the parser recognise "<e" as starting an element.</li>
<li>The next level up correctly doesn't recognise it as a valid case</li>
<li>...but also fails to recognise it as an invalid case.</li>
<li>The parser assumes it was successful and tries again <i>on the same string.</i></li>
</ul>
<br />
<h3>
Why Do I Care?</h3>
"How does this affect me?" you may well ask. "I don't use Expat. I don't even write in C. How can I possibly be affected?" The answer is, you may well be using Expat unknowingly. There are wrapper libraries for Python, Java and many other languages for Expat. Many other <a href="https://libexpat.github.io/doc/users">application and libraries</a> (particularly C and C++ libraries like Poco, libDOM or libwww) use Expat under the hood.<br />
<br />
The libraries often enable external entity parsing, which does potentially allow this bug to bite your application. Some of them will, or used to, download the URIs for you, which is a serious problem. Some of the applications are just parsing local configuration files, but even those might follow a DTD if one was<br />
inserted into the configuration file somehow. Other applications explicitly use external DTDs, leaving you vulnerable if those sites are compromised or malicious.<br />
<br />
In short, it's entirely possible for you to be using Expat and not know it.<br />
<br />
<h3>
What Should I Do?</h3>
First and most obviously, upgrade. The current version of libexpat has patched this bug, and a few other things; read the change log for details.<br />
<br />
The other thing you should always do is consider how you use your XML parser. I am not a security expert by any stretch of the imagination, but even I know not to download arbitrary URIs, for example. Reading "https://www.w3c.org/xml/fluffy.ent" is probably safe; reading "http://evil.haxx0rs.org/xml/i-pwn-u.dtd" probably isn't. You should check that any library you use doesn't automatically download arbitrary URIs for you, and you should be careful about what URIs you do allow to be downloaded.<br />
<br />
<h3>
Conclusions</h3>
If you want a moral from this article: always test. Murphy's Law ensures there will always be bugs in your code, and you should make at least some effort to find them before they annoy other people. And if by chance you should be one of those people annoyed by a bug, let the author know. He or she will usually be glad for the feedback and want it fixed as much as you do!Anonymoushttp://www.blogger.com/profile/05601176772762598727noreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-75309479680630083622016-11-17T22:53:00.001+00:002016-11-17T22:54:18.573+00:00Join us!Just a heads-up to say that we'll be at the <a href="http://www.cl.cam.ac.uk/">Cambridge University Computer Lab</a> recruiting event tomorrow (um, today), showing off a few things we've knocked together and trying to persuade all you lovely people to come and work for us.<br />
<div>
<br /></div>
<div>
Do drop by if you're about - we will literally have cookies - and if you're after work or an internship, get in touch at <a href="mailto:info@kynesim.co.uk">info@kynesim.co.uk</a> .<br />
<br />
In the meantime, here's a photo of an oscilloscope playing (3-d) pong:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbOZi5spw_FjHd7WTeVet9zrX7viDDPI6UzpbkfoJdqhO2ttB66RjLfmXEtDXxshz3AXL2fzKJ0tf9x2B5m7aQvK1EBpQUG65nanNKcaQtqFAZ-kttq3-UXgf1h7FGDgVforbNoczlca96/s1600/pong.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbOZi5spw_FjHd7WTeVet9zrX7viDDPI6UzpbkfoJdqhO2ttB66RjLfmXEtDXxshz3AXL2fzKJ0tf9x2B5m7aQvK1EBpQUG65nanNKcaQtqFAZ-kttq3-UXgf1h7FGDgVforbNoczlca96/s320/pong.jpg" width="320" /></a></div>
<br />
Source code (for a RasPI with the scope tied to audio out and the AC decouplers removed) is at <a href="https://github.com/kynesim/scopedemo">https://github.com/kynesim/scopedemo</a> .<br />
<br /></div>
Richardhttp://www.blogger.com/profile/15413093214412156645noreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-85880182245587650282016-10-09T13:22:00.001+01:002016-10-09T13:29:49.404+01:00Fun facts about ethernet debugging, number 3+4i in an occasional series<p>So, I've been writing a driver for an OS-less machine which uses (for reasons you will be relieved that I am not going to go into) a LAN9221 as its ethernet interface.</p>
<p>Now, this chip is moderately spiffy - in particular, it seems Microchip have now encountered every possible bus-related design cock-up and have a handy register ready for reversing out most of them. However, it requires quite careful FIFO handling and turns out to be exquisitely read-sensitive. So far, so good.</p>
<p>So, one thing I tried was doing a Tx test. Send a packet every couple of seconds, read status back out of the status FIFO, report. Check that all is well. </p>
<p>Now, I didn't have a LAN port that didn't have too much chatter on it, so I used an ASIX AX88178 I had lying around to do the capture (and incidentally, what is it with Linux desktops these days that they just can't seem to shut up? You no sooner plug something in than you have kilobits of traffic asking if, against all knowledge and probability, an ethernet interface with no IP is a link back to some mothership or other. Sheesh).</p>
<p>Anyway, you look at the trace in wireshark, and all is well, except that if you send:</p>
<pre>
0000 68 05 ca 1e 35 18 00 50 9a 00 00 00 08 00 01 02 h...5..P........
0010 03 04 05 76 40 30 00 00 40 00 0b 00 00 10 01 02 ...v@0..@.......
0020 03 04 05 06 07 08 01 02 03 04 05 06 07 08 01 02 ................
0030 03 04 05 06 07 08 01 02 03 04 05 06 07 08 01 02 ................
</pre>
<p>You get your original packet, but immediately after it, you also get:</p>
<pre>
0000 40 00 bf ff 68 05 ca 1e 35 18 00 50 9a 00 00 00 @...h...5..P....
0010 08 00 01 02 03 04 05 76 40 30 00 00 40 00 0b 00 .......v@0..@...
0020 00 10 01 02 03 04 05 06 07 08 01 02 03 04 05 06 ................
0030 07 08 01 02 03 04 05 06 07 08 01 02 03 04 05 06 ................
0040 07 08 01 02 ....
</pre>
<p>which is your packet, with <tt>0x4000bfff</tt> prepended to it. No status word in the Tx status FIFO, no IRQ_SIS TXE bit, nothing. Weird, huh?</p>
<p>It's a replay, so can't be bad FIFO management (well, not obvious bad FIFO management), and it's not the CPU double-writing or you'd get word dups, not packet dups. If you vary the size of your packet, you find that the first byte is the length of the original, and the third and fourth byte are always something like <tt>0xNfff</tt> where <tt>N</tt> seems to be something to do with the top nybble of your packet length.</p>
<p>So, you try sending two packets back to back. Send:</p>
<pre>
0000 68 05 ca 1e 35 18 00 50 9a 00 00 00 08 00 01 02 h...5..P........
0010 03 04 05 77 40 30 00 00 40 00 0c 00 00 10 01 02 ...w@0..@.......
0020 03 04 05 06 07 08 01 02 03 04 05 06 07 08 01 02 ................
0030 03 04 05 06 07 08 01 02 03 04 05 06 07 08 01 02 ................
0000 68 05 ca 1e 35 18 00 50 9a 00 00 00 08 00 01 02 h...5..P........
0010 03 04 05 78 40 30 00 00 40 00 0c 00 00 10 01 02 ...x@0..@.......
0020 03 04 05 06 07 08 01 02 03 04 05 06 07 08 01 02 ................
0030 03 04 05 06 07 08 01 02 03 04 05 06 07 08 01 02 ................
</pre>
<p>Get:</p>
<pre>
0000 40 00 bf ff 68 05 ca 1e 35 18 00 50 9a 00 00 00 @...h...5..P....
0010 08 00 01 02 03 04 05 77 40 30 00 00 40 00 0c 00 .......w@0..@...
0020 00 10 01 02 03 04 05 06 07 08 01 02 03 04 05 06 ................
0030 07 08 01 02 03 04 05 06 07 08 01 02 03 04 05 06 ................
0040 07 08 01 02 40 00 bf ff 68 05 ca 1e 35 18 00 50 ....@...h...5..P
0050 9a 00 00 00 08 00 01 02 03 04 05 78 40 30 00 00 ...........x@0..
0060 40 00 0c 00 00 10 01 02 03 04 05 06 07 08 01 02 @...............
0070 03 04 05 06 07 08 01 02 03 04 05 06 07 08 01 02 ................
0080 03 04 05 06 07 08 01 02 ........
</pre>
<p>Sometimes you get two of these curious runt packets, and sometimes one. Awooga! You then spend all sodding night debugging the cursed thing, convinced that your MMU has somehow rewritten the FIFO, or you've accidentally written it a negative length and it's wrapped, or that you're trying to send a packet in the middle of a reset.</p>
<p>Anyway, finally, in desperation, you plug your ASIX adaptor into a different machine, running kernel 3.13, rather than the 3.11 (sheesh, that old?) on your original box and you get your original packets. Two at a time, all fine and dandy - and my onboard adaptors on two different laptops seem to agree with that.</p>
<p>It seems that older ASIX drivers and/or 3.11 will insert comedy packets into your wireshark captures, for fun and profit. Now, the reason I switched to the ASIX in the first place was because I was seeing these runt packets on another interface, so I'm not sure if I blame the ASIX driver or not. My desktop has both 802.11q and IPv6 enabled, so it may be that some component of the network stack is simply failing to cope with an attempt to configure 802.11q and <tt>IFF_PROMISC</tt> at the same time. Be warned!</p>
<p> Hopefully, if you found this post via google at 3am, you can now change adaptor, go to bed and sleep the sleep of the justly offended.</p>
<p>Gah. Onward to lwip! (which is at least reasonably well-behaved)</p>
Richardhttp://www.blogger.com/profile/15413093214412156645noreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-42857786788529238972016-07-29T20:42:00.000+01:002016-07-29T20:42:44.284+01:00Console application for MTAPI<br />
No, this is still not the power measurement post. Richard will get around to it soon, honest.<br />
<br />
We spend a lot of time playing around with TI's CC2538 Zigbee chip running Z-Stack, which means that we spend a lot of time programming other embedded chips to talk TI's Monitor and Test API serial protocol to it. Needless to say, debugging at more than one CPU's remove can be a tedious exercise, and there's nothing (that I know about, at least) that will allow you to talk MTAPI to a CC2538 from your nice, comfortable Linux development environment.<br />
<br />
Enter MTConsole, in the <a href="http://github.com/kynesim/MTConsole">http://github.com/kynesim/MTConsole</a> repository. In a fit of enthusiasm, and a strong desire to be able to script tests, I've put together a Python program to translate to and from MTAPI. It is limited at the moment, and some areas are, to be blunt, not very pretty. It falls over with an exception on a parse error, for example, because that was what I wanted when testing the parser. That will get visited with fire and the sword when it first annoys me in real use.<br />
<br />
So far I've put a lot of effort into parsing binary input (from the CC2538) into text, and not much into parsing text into MTAPI commands. That will be changing as I need more commands (fire and the sword, people), but at the moment it's not useful for much more than proving that your chip is up and talking to the outside world. Useful as that is, I would eventually like to get to the point where I can script it well enough to mock up complex command sequences.<br />
<br />
Please feel free to grab and use as the mood takes you. Patches, bug reports, comments and suggestions are always welcome, just bring your own sword.Anonymoushttp://www.blogger.com/profile/05601176772762598727noreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-80422315705743754502016-07-20T21:21:00.000+01:002016-07-20T21:25:00.109+01:00Hacking Wireshark for fun and profit<p>We've been working on some low-power Zigbee sensors recently, based on CC2538 / ZStack 1.2.2 and running Zigbee HA 1.2.1 ; whilst we were doing this it occurred to us that Wireshark could usefully decode a few more of the IAS Zigbee messages into something useful.</p>
<p>So we did that.</p>
<p>It also turns out that some of our clients (and compliance test houses) use the very nice <a href="http://www.ubilogix.com/products/ubiqua">Ubiqua</a> protocol analyser tool</p>
<p>However, we are a Wireshark shop.</p>
<p>Step up Vadim <vpanov05@somewhere>, who contributed a bunch of patches in Wireshark <a href="https://www.wireshark.org/lists/wireshark-bugs/201206/msg00846.html">bug 7426</a>. Those had rotted a bit, so I resurrected them and the upshot is that the <a href="http://github.com/kynesim/wireshark">http://github.com/kynesim/wireshark</a> repository now contains a bunch of things that Zigbee hackers might find fun:</p>
<ul>
<li> Better decoding for Zigbee IAS messages - from Rhodri James.
<li> Support for CUBX, TI SmartRF Studio and Ember Insight Desktop file formats as per Vadim's patch.
<li> Support for more recent Ubiqua 3 file formats (at least, on the traces I have here).
<li> A nasty backdoor mechanism so that you can decode Ubiqua traces which don't contain the TC key transport packet in the trace (Ubiqua stashes this in a separate table, and we pass it round the back to the Zigbee packet dissector).
</ul>
<p>One day I will get around to trying to push this lot upstream, but I suspect we will want a better way to do the backdoor key transport than the
ugly hack I have in there at the moment.</p>
<p>Anyway, if you feel minded, grab it, enjoy and do report any bugs you come across (and I will do another post on power measurement for low-power radio, honest).</p>
Richardhttp://www.blogger.com/profile/15413093214412156645noreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-32972876807032540872016-04-12T21:37:00.001+01:002016-04-12T21:37:46.114+01:00It's been a while ..It looks as though the last post on this blog was waaay back in 2013, which goes to show how busy we've been. So, what's happened between then and now?
Well, our open source stuff has migrated to github - <a href="https://github.com/kynesim">https://github.com/kynesim</a> . There you'll find a bunch of stuff you might find useful, including:
<ul>
<li> The venerable <a href="https://github.com/kynesim/tstools">tstools</a> and <a href="https://github.com/kynesim/muddle">muddle</a>.
<li> Our local variants of <a href="https://github.com/kynesim/ccsniffpiper">ccsniffpiper</a> - a program which allows you to sniff Zigbee with a TI CC2531 USB dongle, and <a href="https://github.com/kynesim/wireshark/">wireshark</a> - which contains some patches from Rhodri that help decode Zigbee HA IAS ACE commands in a more friendly way.
<li> The current version of <a href="https://github.com/kynesim/kbus">kbus</a> - still 10% the size of kdbus :-)
<li> <a href="https://github.com/kynesim/upc2">upc2</a> - an easy to use, easy to cross-compile, terminal program that can speak xmodem and Andrew Gordon's semi-proprietory grouch protocol.
</ul>
And lately we've been playing with some automotive powertrain stuff, some very low power Zigbee sensors based on TI ZStack Home 1.2.2a (under some circumstances, you can get battery life projections as long as the shelf life of the batteries), BT LE dynamic advertising with Android, and TI's newer, even lower power CC2650 - but those will have to be the subject of their own posts.
Richardhttp://www.blogger.com/profile/15413093214412156645noreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-35108707259751114052013-10-09T12:17:00.002+01:002013-10-14T12:01:34.290+01:00Changing the "theme" in Enthought Traits UI with Qt as the backendI'm currently writing code for a customer using Enthought Traits, and in particular Traits UI. Traits as a whole is a very nice toolkit for use in scientific programming, but you can find out about that by googling in the normal manner.<br />
<br />
The specific issue, though, is that we've moved to using pyside/Qt4 as the GUI backend, instead of wx(Python) - enabled, for instance, by doing:<br />
<br />
<span style="font-family: "Trebuchet MS", sans-serif;"> try:<br /> from traits.etsconfig.api import ETSConfig<br /> ETSConfig.toolkit = 'qt4'<br /> except ValueError:<br /> pass</span><br />
<span style="font-family: "Trebuchet MS", sans-serif;"></span><br />
Some things are done a little bit differently with the Qt backend, though, and it's not always obvious how. In particular, traits UI has always allowed "theming", and the normal way to do this with the traditional wx backend is using something like:<br />
<br />
<span style="font-family: "Trebuchet MS",sans-serif;"> from traitsui.api import Theme</span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"> ...</span><br />
<span style="font-family: "Trebuchet MS",sans-serif;"> Item('move_to_loading_area_btn', item_theme=Theme('@std:BE5')),</span><br />
<br />
<span style="font-family: inherit;"><span style="font-size: small;"><i> (note that this is <b>not</b> the syntax for the <span style="font-family: "Trebuchet MS",sans-serif;">item_theme</span> argument shown in the documentation, that doesn't seem to work - so this is also "a useful reference").</i></span> </span><br />
<br />
This doesn't work with the Qt backend. It turns out that the way to do it is to specify a Qt stylesheet, which can be done as a string:<br />
<br />
<span style="font-family: "Trebuchet MS",sans-serif;"> Item('move_to_loading_area_btn', style_sheet='* { color: red }'),</span><br />
<br />
Useful Qt references are then:<br />
<ul>
<li>http://qt-project.org/doc/qt-4.8/stylesheet-syntax.html</li>
<li>http://qt-project.org/doc/qt-4.8/stylesheet-reference.html#list-of-properties</li>
<li>http://qt-project.org/doc/qt-4.8/stylesheet-examples.html</li>
</ul>
<i>(blogged here because no-one else seems to have described doing this)</i>Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-5625827309301120857.post-68599014675075428912013-09-29T12:36:00.001+01:002013-09-29T12:53:51.752+01:00A report on PyConUK 2013As usual, this was a very enjoyable conference. I had fun, I met lots of
interesting people, and I learnt things.<br />
This is mostly a report on the program, because that's the easiest thing to
write about, and because some of these notes are for work.<br />
<br />
<a name='more'></a><br />
On Friday Zeth re-instructed us to join up in groups according to where we
come from, and so half-a-dozen of us from East Anglia ended up at a table in
the dining hall. It looks as if Peter Inglesby has been volunteered to try to
put together a weekend meeting some time in the future - but as he was heavily
involved in the programming for the weekend, I'd expect him to need a while to
recover first.<br />
<br />
Also at the table was Pranjal Mittal, who came to the conference from India,
and was giving a talk on <a class="reference external" href="http://code.google.com/p/ganeti/">Ganeti</a> on Saturday. (I'm assuming <a class="reference external" href="http://www.pranjalmittal.in/2013/07/google-summer-of-code-update-1.html">this is he</a>)
This interested me because I'd already figured out from the program notes that
Ganeti might solve some problems for us at work, as colleagues had recently
tried to get OpenStack up and running (following on from hard disk problems
with our virtual machine cluster). It looks like this probably needs the
traditional couple of weeks of learning before it works, which wasn't
feasible, and Ganeti sounds as if it might do what we want (and the <a class="reference external" href="https://github.com/ramereth/vagrant-ganeti/">Vagrant
and Ganeti</a> project may be useful as well). Anyway, talking to Pranjal then,
and watching his talk later convinced me it probably does do just what we
want, and should (hopefully) be relatively simple to get going.<br />
<br />
My prize for aplomb and sheer style has to go to Gautier Hayoun. For some
reason his talk, An overview of Ansible, started early, which meant that a lot
of us (coming from another talk) dithered outside the room assuming the
previous talk had overrun, until we realised that, no, this was the talk we
wanted. However, Gautier had apparently believed he only had time for a 10
minute talk, so it ended shortly after the audience had doubled in size.
At which point, he gave the talk again, and still had time for questions at
the end. Moreover, it was a very good talk, and gave me exactly the
information I wanted.<br />
<br />
<a class="reference external" href="http://www.ansibleworks.com/docs/gettingstarted.html">Ansible</a> is a radically simpler alternative to Puppet and Chef. It's not daemon
based (at either end), and uses YAML playbooks which provide a description of
the state you want your server to be in, by describing what actions
should be taken in what order to get there (as opposed to Puppet, for
instance, which specifies results and leaves the tool to sort out ordering).
It is idempotent - it won't redo an action on a client if that action has
already been completed. Gautier did recommend, though, that one should also
checkout <a class="reference external" href="http://saltstack.com/community.html">Salt</a> if one was considering Ansible (and vice-versa).<br />
<br />
(Aaron Brady's Python for Configuration Management then expanded on that
introduction (although the talks were independent), and reinforced my idea
that Ansible may be something I want to use. His talk is online at
<a class="reference external" href="http://www.insom.me.uk/post/pycon-talk.html">http://www.insom.me.uk/post/pycon-talk.html</a>, and links to a github repository
with examples.)<br />
<br />
The Breaking Things for Money talk by Tony Simpson and Peter Russell from The
Test People was just fun.<br />
<br />
I missed Geoffrey French talking about Ubiquitous Larch, the latest
incarnation of his Python programming environment, but luckily he was willing
to show it to me at the social evening. As I would expect from him, it is
visually attractive (and yes, that matters). I've played a little with the
IPython notebook recently, and the Ubiquitous Larch feels much more polished,
not least because it can rotate 3-d graphs. However, I suspect the killer
feature may be the ability to interact with other people running the same
environment - live changes such as moving a slider on his laptop and seeing
a graph alter both there and on my phone. Put it together with a voip session
and that could be really useful. I do hope that he can get funding to develop
this further.<br />
<br />
(If people are interested in trying it out, then contact him, but his previous
project, The Larch Environment, is available at
<a class="reference external" href="http://larch.pythonanywhere.com/">http://larch.pythonanywhere.com/</a>)<br />
<br />
Larry Hastings did his introduction to Python bytecode (All-Singing
All-Dancing Python Bytecode), which I <em>think</em> I've seen a version of before
(there are versions out there on the web to read/watch, including on <a class="reference external" href="https://lwn.net/Articles/544787/">LWN</a>). I
thought it was a good overview of the subject, and his disassembler <a class="reference external" href="https://bitbucket.org/larry/maynard">Maynard</a>
produces much more useful output than the standard tool.<br />
<br />
Ian Ozsvald's talk on detecting/disambiguating words in tweets was
fascinating. He made what he was doing sound simple, but was managing what may
be useful results when I'd have given up before starting. He's been talking
about <a class="reference external" href="https://github.com/ianozsvald/social_media_brand_disambiguator">the project</a> on his blog (which shows up on the London Python aggregator,
and I think also on Planet Python), but the talk made it all a lot clearer to
me. Talking to him in the hall later, I've also signed up as interested in the
O'Reilly book he's (now) co-writing on <a class="reference external" href="http://ianozsvald.com/2013/09/17/writing-a-high-performance-python-book/">High Performance Python</a>.<br />
<br />
I session chaired on Saturday afternoon. PyConUK speakers tend to make this
easy, as they're generally good at keeping to time, so my hardest job was
getting the microphone to people with questions. One of the reasons for
session chairing is that it can get you to see talks you might not have gone
to otherwise (and this has yet to work out badly for me). I really enjoyed
both of the turtle talks. Mike Sandford gave an introductory talk, presented
<em>as a turtle program</em>, running on a RaspberryPi, which was very cute, and
Simon Davy gave a talk on how he is trying to write a more powerful turtle
library that will allow running thousands of simultaneous turtles, so that
users can emulate flocking and other interesting behaviours.<br />
<br />
My "unexpected find" talk was Matthew Hughes with Teaching Data Science With
Really Scrapable Web App. This didn't start well, as his laptop took a while
to connect properly to the projector, and so he lost five minutes from the
start of his talk. Despite that, he still managed to do a good and coherent
presentation, and have time for questions at the end (heh, I was session
chairing, I was impressed). The talk, though, introduced his "Really Scrapable
Web App" (<a class="reference external" href="https://github.com/matthewhughes/really-scrapable-web-app">RSWA</a>), which is a tool/set of examples for learning how to scrape
the web. As he says in <a class="reference external" href="http://www.matthewhughes.co.uk/really-scrapable-web-app-is-a-new-way-to-learn-about-web-scraping/">his article</a>:<br />
<blockquote>
Borrowing concepts from Project Euler and Code Academy, RSWA runs locally on
the users computer and contains a number of challenges which aim to gently
introduce core skills used by data scientists.</blockquote>
I've looked a little at web scraping in the past, and bounced off because I
didn't have something I <em>needed</em> to scrape as a motivation. This sounds like
just the thing to fill that gap, and playing with it is now on my list of
things to do.<br />
<br />
What else?<br />
<br />
Well, the conference dinner was excellent, as usual (and I need to look out
for <a class="reference external" href="http://shop.oreilly.com/product/0636920030508.do">ntoll's O'Reilly book</a> on futures and deferred in JQuery - it sounds as
if it will be a good read almost independent of the JQuery part, and I love
his <a class="reference external" href="http://www.theguardian.com/info/developer-blog/2013/sep/17/deferreds-and-promises-programming-javascript">tumble dryer analogy</a>).<br />
<br />
The staff were always helpful, too, and kept up a constantly replenished
supply of coffee, and the food for lunch was plentiful (although I think the
attempt to get people to have starters first on Sunday was probably doomed to
failure).<br />
<br />
I liked being given a canvas bag - they're really useful, and the PyConUK bags
are normally attractive, but this year's especially so. The conference T-shirt
is wonderfully daft.<br />
<br />
Other people have mentioned the RaspberryJam happening as a part of the
conference - I only nipped over once for a quick look, but it was very busy,
and there was something good about seeing how many of the adults and children
were wearing the same T-shirt.<br />
<br />
And, of course, the lightning talks were good. I love the fact that Harald
Mass (<a class="reference external" href="http://www.lightningtalkman.com/">http://www.lightningtalkman.com</a>) came over to present them - it wouldn't
be the same without him. Here are some random links from the lightning talks:<br />
<ul class="simple">
<li><a class="reference external" href="http://kivy.org/">http://kivy.org/</a></li>
<li><a class="reference external" href="http://www.carlreynolds.net/">http://www.carlreynolds.net/</a> and <a class="reference external" href="http://nhshackday.com/">http://nhshackday.com/</a></li>
<li>EuroSciPy 2014 will be in Cambridge (<a class="reference external" href="https://www.euroscipy.org/">https://www.euroscipy.org/</a> doesn't say
this yet)</li>
<li>Alex Willmer: when estimating, double and put into the next largest units.
Also, a reference to Harry Potter and the Methods of Rationality.</li>
<li><a class="reference external" href="http://orionrobots.co.uk/">http://orionrobots.co.uk/</a></li>
<li><a class="reference external" href="http://twistedftw.org/">http://twistedftw.org/</a></li>
<li><a class="reference external" href="http://www.structlog.org/">http://www.structlog.org</a></li>
<li>Tim Golden has started a github organisation called “One Screenful of
Python”, which is aimed to present interesting, self-contained pieces of
Python code that will fit on a single screen and can be used as aids in
teaching: <a class="reference external" href="https://github.com/OneScreenfulOfPython/screenfuls">https://github.com/OneScreenfulOfPython/screenfuls</a></li>
</ul>
Other reports on the conference, as well as links to talks online and so on,
are being collected at <a class="reference external" href="http://pyconuk.net/PostConf">http://pyconuk.net/PostConf</a>.<br />
<br />
Next year's conference will be at the same place, and on the weekend of 19th
September. I hope to be there.<br />
<br />
Tibs, 2013-09-29<br />
<br />
(This is also posted to the Cambridge and East Anglia Python User's Group, at <a href="https://groups.google.com/forum/#!topic/campug/0CM0oVVQ8i0">https://groups.google.com/forum/#!topic/campug/0CM0oVVQ8i0</a>)<br />
<!-- vim: set filetype=rst tabstop=8 softtabstop=2 shiftwidth=2 expandtab: -->
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-282329175156206732013-08-26T16:50:00.000+01:002013-08-26T16:50:44.363+01:00Ambient backscatter radioIt seems that some folk at U. Washington have managed to implement an idea that I've been playing around with for a while - ambient backscatter. Now what we need are far, far better reflectors - their system only really works well at distances of 2-3 feet in free air, and really what we need is something that can get through walls.<br />
<br />
On the other hand, if their aerials can be made smaller, they could reflect WiFi signals, which have much higher local intensities than distant TV transmitters and whose reflection characteristics are fairly well known thanks to all the folk doing diversity with them.<br />
<br />
I wonder if a semi-intelligent aerial which spots beacon frames in h/w is a good thing to have - on the one hand, it provides a known timing reference for the receiver to try and wake up to, but on the other you will have all of the distance-related wakeup window issues.<br />
<br />
The other thing one can do over the UW study in a home hub environment is, of course, to have a pulsed interrogation transmitter - notably if you operate in 868MHz or 2.4GHz that transmitter can be on the same frequency as your ambient radio, so you can work in a wait_for_reflect-pulse-wait_for_result mode, which allows you to get quite high power intensities without interfering too much with WiFi (particularly if you are listening to the WiFi and know when its slots are).<br />
<br />
This (in some form) is obviously the right technology for things like window sensors - now you only need to change batteries in one room, not one every sensor.<br />
<br />
Anyway, have a link - it's an interesting read: <a href="http://abc.cs.washington.edu/files/comm153-liu.pdf">http://abc.cs.washington.edu/files/comm153-liu.pdf</a><br />
<br />Richardhttp://www.blogger.com/profile/15413093214412156645noreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-52158293919772126522013-03-14T11:08:00.001+00:002013-03-14T11:08:56.814+00:00Muddle v2.5.1 - change in how "muddle pull" works<br />
If "<span style="font-family: Trebuchet MS, sans-serif;">muddle pull</span>" is given a command line that includes the build description (implicitly or explicitly), it will pull the build description first, and if that pull changed things, will reload the build description and re-evaluate the command line, before pulling the other checkouts requested.<br />
<br />
This means that you no longer need to check if "<span style="font-family: Trebuchet MS, sans-serif;">muddle pull</span>" changed the build description and then <i>do it again</i> if it did.<br />
<br />
See "<span style="font-family: Trebuchet MS, sans-serif;">muddle </span><span style="font-family: Trebuchet MS, sans-serif;">help pull</span>" for a more complete description.<br />
<br />
If you do want the old behaviour, use "<span style="font-family: Trebuchet MS, sans-serif;">muddle pull -noreload</span>".<br />
<br />
As a consequence of this change, the "<span style="font-family: Trebuchet MS, sans-serif;">_just_pulled</span>" value is now set by "<span style="font-family: Trebuchet MS, sans-serif;">muddle </span><span style="font-family: Trebuchet MS, sans-serif;">checkout</span>" as well as by "<span style="font-family: Trebuchet MS, sans-serif;">muddle pull</span>".<br />
<div>
<div>
<br /></div>
<div>
This resolves issue 218.</div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-17060466114913027132013-02-13T13:25:00.001+00:002013-02-13T13:25:42.475+00:00Muddle v2.5 released<br />
Upgrade to the latest version of muddle in the normal manner:<br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;"> $ cd <muddle-directory></muddle-directory></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"> $ git pull</span><br />
<br />
I believe that it all works, but if you have problems with v2.5 (which has had some substantial code rewrites in various areas), consider reverting to <span style="font-family: Trebuchet MS, sans-serif;">Just-before-2.5</span><span style="font-family: inherit;">, which was the last checkin on the old "master" branch::</span><br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;"> $ cd <muddle-directory></muddle-directory></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"> $ git checkout Just-before-2.5</span><br />
<br />
Changes in v2.5 since v2.4.4 are listed after the break:<br />
<br />
<a name='more'></a><br />
First, changes you'll already have if you've been keeping up-to-date with the current version of muddle:<br />
<br />
<ul>
<li>New squashfs and romfs deployments. Also, the underlying code for the "muddled.deployments" modules has been rewritten, and CPIO deployments can now be made to a package (specifically, to a package install directory), removing one of the causes of packages depending on deployments.</li>
<li>Minor tweaks in the release mechanism, based on experience of using it. These include adding a "<span style="font-family: Trebuchet MS, sans-serif;">-test</span>" switch to "<span style="font-family: Trebuchet MS, sans-serif;">muddle release</span>", and allowing "<span style="font-family: Trebuchet MS, sans-serif;">muddle stamp release</span>" to "guess" the next version number to use for a release stamp.<br />See "<span style="font-family: Trebuchet MS, sans-serif;">muddle help releas</span>e" and "<span style="font-family: Trebuchet MS, sans-serif;">muddle help stamp release</span>".</li>
<li>A new "quickstart" section at the beginning of the documentation, for those who've just been given a build using muddle for the first time.<br />See <a href="http://muddle.readthedocs.org/en/latest/quickstart.html">http://muddle.readthedocs.org/en/latest/quickstart.html</a> ("Quick start: so you’ve been asked to work on a project using muddle")</li>
<li>The <span style="font-family: Trebuchet MS, sans-serif;">Invocation</span> class is no more, it has been folded into <span style="font-family: Trebuchet MS, sans-serif;">Builder</span>. Thus <span style="font-family: Verdana, sans-serif;">builder.invocation.<i>XXX</i></span> is now identical to <span style="font-family: Trebuchet MS, sans-serif;">builder.<i>XXX</i></span> in all cases. This basically makes build descriptions a bit simpler, and removes the problem of how to remember whether a particular method had <span style="font-family: Trebuchet MS, sans-serif;">.invocation</span> in front of its name or not.</li>
<li>There has been an internal rewrite of the version control handler support and much of the "memory" of how checkouts work has been moved from there to the <span style="font-family: Trebuchet MS, sans-serif;">Database</span> class and db.py. This was a prerequisite of the lifecycle work, below.</li>
<li>There have been various minor fixes in license support, including not propagating GPL licenses to/through GPL licenses.</li>
</ul>
<br />
Secondly, changes that have just been introduced by merging a long development<br />
branch back into "master":<br />
<br />
<ul>
<li>Lifecycle support. I've been working on this for a while. Basically, it allows branching a whole build tree for maintenance support.<br /><br />New commands are:<br /><span style="font-family: Trebuchet MS, sans-serif;"> muddle init -branch <i><branch> <repo></repo></branch></i> <i><build-desc></build-desc></i><br /> muddle branch-tree <i><new-branch></new-branch></i><br /> muddle query checkout-branches<br /> muddle query checkout-id<br /> muddle sync</span><br />and <span style="font-family: Trebuchet MS, sans-serif;">muddle stamp version</span> behaves slightly differently with a branched build tree.<br /><br />Stamping and unstamping of checkouts on other branches or revisions is now more robust and consistent as well.<br /><br />See the new lifecycle documentation (a new chapter in the normal muddle documentation, <a href="http://muddle.readthedocs.org/en/latest/lifecycle.html">http://muddle.readthedocs.org/en/latest/lifecycle.html</a>), and also "<span style="font-family: Trebuchet MS, sans-serif;">muddle help</span>" for each of the new commands above.</li>
<li>Repository specifications no longer treat a revision argument of the form <span style="font-family: Trebuchet MS, sans-serif;">"a:b"</span> as meaning <span style="font-family: Trebuchet MS, sans-serif;"><i><branch>:<revision></revision></branch></i></span>. This was a legacy from early repository support. and clashes with such things as specifying Bazaar revision ids of the form <span style="font-family: Trebuchet MS, sans-serif;">"branch:<i><path></path></i>"</span> or <span style="font-family: Trebuchet MS, sans-serif;">"revno:<i><number></number></i>"</span>.</li>
<li>The <span style="font-family: Trebuchet MS, sans-serif;">"no_follow"</span> VCS option has been added for Subversion and Bazaar, and<br /><span style="font-family: Trebuchet MS, sans-serif;"> builder.db.set_checkout_vcs_option(co_label, name, value)</span>may now be used as another way to set options.</li>
<li>A rewritten and more powerful <span style="font-family: Trebuchet MS, sans-serif;">muddle doc</span> command. See <span style="font-family: Trebuchet MS, sans-serif;">muddle help doc</span>, and try:<br /><span style="font-family: Trebuchet MS, sans-serif;"> muddle doc Label<br /> muddle doc Label -pydoc<br /> muddle doc -contains Directory</span></li>
<li>A variety of other tidyings and bugfixes.</li>
</ul>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-86682834457426480252013-01-28T09:30:00.000+00:002013-01-28T09:30:49.038+00:00muddle - CPIO support in packagesThere is a new version of muddle, which addresses issue 201: Allow CPIO file creation by a package, not just by a deployment, and also has some internal amendments which should not be visible.<br />
<br />
<a name='more'></a><br />
<br />
This new version of muddle is tagged package-cpio-support. It is available on both Google Code and github, in the normal manner (just "git pull").<br />
<br />
The traditional way of creating a CPIO archive with muddle is via a deployment, something like:<br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;">import muddled.deployments.cpio as cpio</span><br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;"> fw = cpio.create(builder, 'firmware.cpio', 'deployment-name')</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"> fw.copy_from_role(role1, '', '/')</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"> fw.copy_from_role(role2, 'bin', '/bin')</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"> fw.done()</span><br />
<br />
which creates a file called <span style="font-family: Trebuchet MS, sans-serif;">deploy/deployment-name/firmware.cpio</span>.<br />
<br />
This is well and good, but sometimes a build wants the CPIO archive to use in a package. The most obvious example is when one builds a kernel and wants to attach a CPIO archive (or worse, one of several CPIO archives) to it as an initrd filesystem.<br />
<br />
In the previous versions of muddle, the only way to do this was to make the kernel-building package depend upon a deployment which built the CPIO archive. This doesn't work very well, as muddle was never designed to support packages depending on deployments (see issue 235). Dependency tracking "through" the deployment doesn't work as one might hope, so doing a proper rebuild of the kernel-building package can be quite difficult.<br />
<br />
One solution would be to fix muddle to handle dependency tracking through deployments, but that's non-trivial, not least because one would need to work out what it <i>means</i> in the general case.<br />
<br />
Luckily, it turns out to be simple to enhance the CPIO creation mechanism to support packages.<br />
<br />
The second argument of cpio.create is amended so that it can take a deployment name (as before), or a deployment or package label. If it is given a deployment label, it acts just as before, but if it is given a package label:<br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;">from muddled.depend import package</span><br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;"> fw = cpio.create(builder, 'firmware.cpio', package('firmware', 'firmware-role'))</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"> fw.copy_from_role(role1, '', '/')</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"> fw.copy_from_role(role2, 'bin', '/bin')</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"> fw.done()</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">then it now creates the CPIO file as </span><span style="font-family: Trebuchet MS, sans-serif;">install/firmware-role/firmware.cpio</span><span style="font-family: inherit;">.</span><br />
<span style="font-family: inherit;"><br /></span>
Note that it is still possible to specify a subdirectory in the CPIO file name, so <span style="font-family: Trebuchet MS, sans-serif;">'fred/firmware.cpio'</span> would create <span style="font-family: Trebuchet MS, sans-serif;">install/firmware-role/fred/firmware.cpio</span>.<br />
<br />
The "virtual" package <span style="font-family: Trebuchet MS, sans-serif;">package:firmware{firmware-role}/*</span> is created automatically, and depends on all the packages in roles <span style="font-family: Trebuchet MS, sans-serif;">role1</span> and <span style="font-family: Trebuchet MS, sans-serif;">role2</span>, just as was the case when using a deployment. This does, however, mean that the firmware package cannot be in the same role as any of the packages it depends on, or we would end up with a circular dependency, which is not allowed. I think that I would recommend having a specific role for CPIO file generation, and if multiple CPIO files are to be constructed, disambiguate them via name and subdirectory (using subdirectories to split then will work well if one is trying to deploy them via a collecting deployment).<br />
<br />
The other change is a restructuring of the internals of checkout/VCS handling within muddle. This should be entirely transparent to the normal user (although anyone using upstream repositories should keep a careful eye on their use, as that's the part of the code I'm least confident of). It's been done in aid of the lifecycle-branching work I'm doing in a development branch, but I believe is a general improvement in the code as well.<br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-37527257235919899302013-01-03T10:28:00.001+00:002013-01-03T10:28:35.585+00:00muddle - "no invocation"Muddle is updated, but you shouldn't be able to tell the difference...<br />
<br />This update is tagged "no-invocation", and is mostly an internal technical change.<br />
<br />
Short description: wherever you have written "<span style="font-family: Trebuchet MS, sans-serif;">builder.invocation.</span>" you can now write "<span style="font-family: Trebuchet MS, sans-serif;">builder.</span>", leading to shorter lines in your build description, which is a Good Thing. However, "<span style="font-family: Trebuchet MS, sans-serif;">builder.invocation.</span>" will continue to work.<br />
<br />
Long description: after the cut.<br />
<br />
<a name='more'></a><br />
I've merged the Invocation class into the Builder class (both are/were in <span style="font-family: Trebuchet MS, sans-serif;">muddled/mechanics.py</span>).<br />
<br />
When muddle was first being designed, it was expected that the Builder class would know how to build things, and the Invocation class would hold information about the particular build description in force - there may have been an intent to allow more than one Invocation per Builder in the back of people's minds.<br />
<br />
However, in practice this distinction has never been very well maintained, and has just led to lines of build description code which are unnecessarily long, because they have "<span style="font-family: Trebuchet MS, sans-serif;">builder.invocation.</span>" instead of "<span style="font-family: Trebuchet MS, sans-serif;">builder.</span>". Methods have also been added to either class with no particular rhyme or reason, as well, which doesn't help.<br />
<br />
So, I've merged the Invocation class into the Builder class. This does lead to a class that will make <span style="font-family: Trebuchet MS, sans-serif;">pylint</span> seriously unhappy (way too many methods in Builder, to its mind). So that we don't break existing build descriptions, we also set:<br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;">self.invocation = self</span><br />
<br />
inside the Builder __init__ method - i.e., you <i>can</i> still reference "<span style="font-family: Trebuchet MS, sans-serif;">builder.invocation.<whatever></whatever></span>" and have it continue to work.<br />
<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-91155329240454643632012-12-17T15:09:00.003+00:002012-12-17T15:09:41.557+00:00New muddle "quickstart" The muddle documentation has now gained a "Quick start" section, at <a href="http://muddle.readthedocs.org/en/latest/quickstart.html">http://muddle.readthedocs.org/en/latest/quickstart.html</a>. I've also added an introduction, which more-or-less matches the front page text on Google Code.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-2168202719226975892012-12-14T15:21:00.002+00:002012-12-14T15:21:17.449+00:00Change to behaviour of "$ muddle" in src/<br />
I have just changed muddle so that the behaviour of a "bare" muddle<br />
anywhere under <span style="font-family: Trebuchet MS, sans-serif;">src/</span> has changed.<br />
<br />
<a name='more'></a><br />
Before, if you were in <span style="font-family: Trebuchet MS, sans-serif;">src/</span>, just typing "muddle" at the prompt would<br />
do a REBUILD of all the checkouts "below" you (if you were in a<br />
specific checkout source directory, then just of that checkout).<br />
Rebuild unsets all the "built" flags, so has to look at each package<br />
in turn and run its Makefile.<br />
<br />
Now, that same "muddle" does a BUILD instead.<br />
<br />
This means that if you are in <span style="font-family: Trebuchet MS, sans-serif;">src/</span> and do "muddle", and one of the<br />
packages doesn't build, so you fix it, and then type "muddle" in the<br />
same place, the building will resume from where it stopped, rather than<br />
having to rebuild all the packages that had already built.<br />
<br />
We believe this is an improvement. Let me know if it is not.<br />
<br />
Other uses of a "bare" muddle continue the same as before - see<br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;">$ muddle help label absent</span><br />
<br />
(I split the muddle help for labels up into more manageable chunks).<br />
<br />
Just for safety, the git repository before this change is tagged<br />
"<span style="font-family: Trebuchet MS, sans-serif;">muddle-does-rebuild</span>", and the commit for this change is tagged<br />
"<span style="font-family: Trebuchet MS, sans-serif;">muddle-does_build-in-src</span>" (the "<span style="font-family: Trebuchet MS, sans-serif;">_</span>" in there is a typo, sorry).<br />
<br />
Tibs<br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-8914157067754188952012-07-26T11:36:00.000+01:002012-07-26T11:37:39.865+01:00Muddle v2.4.4I've just pushed muddle v2.4.4. Changes below the cut.<br />
<br />
<a name='more'></a>This could, arguably, have been considered a big enough change to merit 2.5, <span style="background-color: white;">but I'm saving that for the next big release, which has more internal code </span><span style="background-color: white;">changes.</span><br />
<span style="background-color: white;"><br /></span><br />
<span style="background-color: white;">The main changes are:</span><br />
<span style="background-color: white;"><br /></span><br />
<span style="background-color: white;"><i>Support for making releases.</i> This is a new mechanism for handling customer releases, using a variant stamp file to document exactly what is being released, a new mechanism to specify what should be built for the release, and a method for generating the final release tarball.</span><br />
<span style="background-color: white;"><i><br /></i></span><br />
<span style="background-color: white;">The support includes:</span><br />
<ul>
<li><span style="background-color: white;">new command "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle stamp release <name> <version></version></name></span>" to produce a release</span><span style="background-color: white;"> spec.</span></li>
<li><span style="background-color: white;">new "special" build argument, "<span style="font-family: 'Trebuchet MS', sans-serif;">_release</span>", which is defined in the build</span><span style="background-color: white;"> description as "the labels to build for a release"</span></li>
<li><span style="background-color: white;">new command "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle release</span>" to unstamp and build the release, and tar it</span><span style="background-color: white;"> up.</span></li>
<li><span style="background-color: white;">new function "<span style="font-family: 'Trebuchet MS', sans-serif;">release_from(builder, release_dir)</span>" in the build</span><span style="background-color: white;"> description, which copies files into the release directory</span></li>
</ul>
<span style="background-color: white;">See documentation at </span><a href="http://muddle.readthedocs.org/en/latest/release.html" style="background-color: white;">http://muddle.readthedocs.org/en/latest/release.html</a><span style="background-color: white;"> (and let me say again how wonderful the ReadTheDocs people are, for providing that service), </span><span style="background-color: white;">and also:</span><br />
<ul>
<li><span style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;">muddle help stamp release</span></span></li>
<li><span style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;">muddle help query release</span></span></li>
<li><span style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;">muddle help release</span></span></li>
</ul>
<span style="background-color: white;"><i>New unstamp variant "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle unstamp -update</span>"</i>, can be used to </span><span style="background-color: white;">update the current build tree to match the stamp file. See "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle help</span></span><span style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;"> unstamp</span>".</span><br />
<br />
<i>New switch "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle stamp save -before</span></i>", can be used to select a<span style="background-color: white;"> date/time for the stamp. See "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle help stamp save</span>".</span><br />
<br />
<i>"<span style="font-family: 'Trebuchet MS', sans-serif;">muddle stamp diff</span>" now allows stamp files or build tree as arguments</i>, and<span style="background-color: white;"> changes its default comparison method. Also, output can always be directed</span><span style="background-color: white;"> to a file. See "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle help stamp diff</span>".</span><br />
<br />
<i>Rewritten help text for "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle subst</span>"</i>, and new test code for the same. Also <span style="background-color: white;">better error reporting and a bug fix.</span><span style="background-color: white;"> See "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle help subst</span>"</span><br />
<br />
<i>New help "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle help environment</span>"</i>, which explains the environment variables<span style="background-color: white;"> muddle sets for muddle Makefiles (and elsewhere)</span><br />
<br />
<i>New "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle query checkout-id</span>"</i>, for finding the revision (as would go in a <span style="background-color: white;">stamp file) for a checkout. See "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle help query checkout-id</span>"</span><br />
<br />
<i>New environment variable "<span style="font-family: 'Trebuchet MS', sans-serif;">MUDDLE_OBJ_BIN</span>"</i> (finally).<br />
<br />
And a variety of technical changes, including:<br />
<ul>
<li><span style="background-color: white;"> Stamp files are now documented, in the header comment for</span><span style="background-color: white;"> <span style="font-family: 'Trebuchet MS', sans-serif;">version_stamp.py</span>. See "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle doc version_stamp</span>".</span></li>
<li><span style="background-color: white;">Stamp files can contain comment lines, which do not contribute to the</span><span style="background-color: white;"> hash for the file. The time stamps (for when the file was created) are</span><span style="background-color: white;"> now comments, so two identical stamp files created at different times</span><span style="background-color: white;"> will now have the same hash value.</span></li>
<li><span style="background-color: white;">The premature binding of the current "<span style="font-family: 'Trebuchet MS', sans-serif;">builder</span>" into <span style="font-family: 'Trebuchet MS', sans-serif;">VersionControlHandler</span></span><span style="background-color: white;"> instances has finally been fixed, leading to saner code in various places.</span></li>
<li><span style="background-color: white;">Some improvements to the code for bazaar. In particular, choosing a</span><span style="background-color: white;"> particular revision now works.</span></li>
<li><span style="background-color: white;">Some minor git fixes, including fixing stamping of detached HEAD</span><span style="background-color: white;"> revisions.</span></li>
<li><span style="background-color: white;">The Builder method "<span style="font-family: 'Trebuchet MS', sans-serif;">checkout_path</span>" no longer accepts None as an arguement </span><span style="background-color: white;">(this was deprecated ages back).</span></li>
<li><span style="background-color: white;">The Builder method "<span style="font-family: 'Trebuchet MS', sans-serif;">deploy_path</span>" now takes a label as its argument.</span></li>
<li><span style="background-color: white;">The muddle exceptions, <span style="font-family: 'Trebuchet MS', sans-serif;">GiveUp</span> and friends, now contain a return code,</span><span style="background-color: white;"> which is used at the top level to set muddle's exit code.</span></li>
<li><span style="background-color: white;">New test script, <span style="font-family: 'Trebuchet MS', sans-serif;">tests/all_tests.py</span>, automates running the available</span><span style="background-color: white;"> test scripts.</span></li>
</ul>
<span style="background-color: white;">and various other bugfixes.</span><br />
<br />
Also, muddle is now mirrored on github (<a href="https://github.com/tibs/muddle.mirror">https://github.com/tibs/muddle.mirror</a>), <span style="background-color: white;">although it will not necessarily always be up-to-date.</span><br />
<br />
This release fixes the following issues:<br />
<ul>
<li><span style="background-color: white;">Issue 94 - muddle stamp diff should be able to compare with current</span><span style="background-color: white;"> repository, and with remote stamp file</span></li>
<li><span style="background-color: white;">Issue 164 - Add MUDDLE_OBJ_BIN</span></li>
<li><span style="background-color: white;">Issue 212 - Implement version 2 stamp files - document the format</span></li>
<li><span style="background-color: white;">Issue 221 - Provide a way to generate a version number for a build tree</span></li>
<li><span style="background-color: white;">Issue 226 - muddle stamp update <stamp-file></stamp-file></span></li>
<li><span style="background-color: white;">Issue 230 - Stamp files record bzr revno, but should record revision-id</span></li>
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-48825041983637825952012-06-20T18:23:00.000+01:002012-06-20T18:30:44.065+01:00Cross-compiling Python for arm, with muddleThis week I needed to cross-compile Python. The "local" platform is an x86, and the target is an ARM. My specific purpose is to be able to use the KBUS Python bindings on the target platform, which means I also care about things like ctypes.<br />
<br />
Unfortunately, Python is not terribly simple to cross-compile, partly because it relies on building itself twice. Luckily, someone else has already documented what to do - namely Paul Gibson at <a href="http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html">http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html</a>. That blog post describes the problem, provides the necessary patches for several versions of Python, and then describes how to build it at the command line.<br />
<br />
(At time of writing, it doesn't have a patch for Python 2.7.3, but 2.7.2 is recent enough for our purposes, and still available from <a href="http://www.python.org/download/releases/2.7.2">http://www.python.org/download/releases/2.7.2</a>)<br />
<br />
So the issue becomes simply how to amend the procedure for a muddle environment.<br />
<br />
<a name='more'></a>As is explained in the RandomSplat blog post, we need to build Python twice, once for the host platform, and once for the target. In muddle, that means we need two roles, so our muddle build description needs to contain something like:<br />
<div>
<br /></div>
<div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> # Python. Once on the host and once for the target</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> make.twolevel(builder, 'python', [HOST_TOOLS], 'thirdparty', 'Python-2.7.2')</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> make.twolevel(builder, 'python', [THIRDPARTY], 'thirdparty', 'Python-2.7.2')</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> pkg.append_env_for_package(builder, 'python', [THIRDPARTY],</span></div>
<div>
<span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;"> 'TARGET_CONFIGURE_TARGET', '</span><span style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;">arm-none-linux-gnueabi</span></span><span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;">')</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> # We need the host Python to build the target Python</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> do_depend(builder, 'python' , [THIRDPARTY],</span><span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;"> [ ('python', HOST_TOOLS) ])</span></div>
</div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></div>
<div>
<span style="font-family: inherit;">The </span><span style="font-family: 'Trebuchet MS', sans-serif;">TARGET_CONFIGURE_TARGET</span><span style="font-family: inherit;"> environment variable is used to tell the muddle Makefile what our target architecture actually is.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
Whilst we are at it, it is also worth telling other packages where to put their Python packages - <span style="font-family: 'Trebuchet MS', sans-serif;">site-packages</span> is traditional, but we have to say where that is on the target system. So something like:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> # Tell KBUS where to put its Python module</span></div>
<div>
<span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;"> pkg.append_env_for_package(builder, 'kbus', [SUPPORT],</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> 'TARGET_PYTHON_SITE_DIR',</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> </span><span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;">'/usr/lib/python2.7/site-packages'</span><span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;">)</span></div>
</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">is useful - the muddle Makefile for KBUS can then use the </span><span style="font-family: 'Trebuchet MS', sans-serif;">TARGET_PYTHON_SITE_DIR</span><span style="font-family: inherit;"> environment variable in its </span><span style="font-family: 'Trebuchet MS', sans-serif;">install</span><span style="font-family: inherit;"> target (and I hope eventually to update the muddle Makefile we supply with the KBUS in google code to include use of such a value).</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">So, I downloaded the Python package, unpacked it into its source directory, and made it available to the build - for instance:</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> $ pushd src/thirdparty</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> $ tar -zxvf Python-2.7.2.tgz</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> $ cd Python-2.7.2</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> $ git init</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> $ git add *</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"> $ git commit -m 'Initial commit of Python 2.7.2'</span></div>
<div>
<span style="background-color: white;"><br /></span></div>
<div>
<span style="background-color: white;"><span style="font-family: inherit;">I t</span>hen applied the appropriate patches from the RandomSplat blog, and commited them. I carefully included the URL of the blog post in the commit message, for future reference.</span></div>
<div>
<br /></div>
<div>
(I also needed to make a remote repository available for Python 2.7.2, and do the "<span style="font-family: 'Trebuchet MS', sans-serif;">muddle import; muddle push</span>" dance to link up to it, and so on - but I assume you know how to do that.)</div>
<div>
<br /></div>
<div>
Finally, we need a muddle Makefile to drive all of this. Given we've got two roles, we could specify a muddle Makefile for each of them, but I think it is actually easier, in this case, to have everything in a single file. So we start with the traditional:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"># Muddle makefile for python.</span></div>
<div>
<span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;">CFLAGS += $(MUDDLE_INCLUDE_DIRS:%=-I%)</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;">LDFLAGS += $(MUDDLE_LIB_DIRS:%=-L%)</span></div>
</div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></div>
<div>
<span style="font-family: inherit;">and then some comments explaining why we've got two roles.</span><br />
<blockquote class="tr_bq">
<span style="font-family: inherit;">(By the way, blogger doesn't seem to know what to do with tabs in my quotes, so I'm afraid there are many spaces instead. Still, you weren't going to cut-and-paste my Makefile examples anyway, were you?)</span></blockquote>
<span style="background-color: white; font-family: inherit;">Then we handle the first (host) role:</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;">ifeq ($(MUDDLE_ROLE), host-tools)</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"># Building on the host</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;">all:</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><span style="white-space: pre;"> </span>(cd $(MUDDLE_OBJ_OBJ); make python Parser/pgen)</span></div>
<div>
<span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;"># When Python builds, it compiles all the .py files in $(MUDDLE_SRC)/Lib,</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"># even if we're otherwise doing an out-of-tree build.</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"># I can't see a way around it, so we'll copy the whole tree instead.</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;">config:</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><span style="white-space: pre;"> </span>-mkdir -p $(MUDDLE_OBJ_OBJ)</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>$(MUDDLE) copywithout $(MUDDLE_SRC) $(MUDDLE_OBJ_OBJ) .git</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>(cd $(MUDDLE_OBJ_OBJ); PYTHONPATH= PYTHONHOME= ./configure)</span></div>
<div>
<span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Trebuchet MS', sans-serif;">install:</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><span style="white-space: pre;"> </span># No need to install the host-tools Python</span></div>
<div style="font-family: inherit;">
<br /></div>
</div>
<div>
<span style="font-family: inherit;">As the comments say, it seemed necessary to copy the source tree because of </span><span style="font-family: 'Trebuchet MS', sans-serif;">.py</span><span style="font-family: inherit;"> files getting compiled. Ah well.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Next we have the target part of the build. In the actual makefile, this includes some comments referencing the RandomSplat blog post, and repeating the actual command lines suggested there for building. This is useful context for the lines I <i>actually</i> use, but I shan't repeat here. So:</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;">else</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"># Cross compiling</span></div>
<div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
# We use the host-tools Python we assume we already built</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
HOSTPYTHONDIR = $(shell $(MUDDLE) query objdir package:python{host-tools})/obj</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
HOSTPYTHON = $(HOSTPYTHONDIR)/python</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
HOSTPGEN = $(HOSTPYTHONDIR)/Parser/pgen</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span style="background-color: white;"><br /></span></div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span style="background-color: white;">all:</span></div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>(cd $(MUDDLE_OBJ_OBJ); \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span> <span style="white-space: pre;"> </span>make \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>PYTHONPATH= \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>PYTHONHOME= \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>CFLAGS='$(CFLAGS)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>LDFLAGS='$(LDFLAGS)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>HOSTPYTHON='$(HOSTPYTHON)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>HOSTPGEN='$(HOSTPGEN)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>CROSS_COMPILE_TARGET=yes \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>CROSS_COMPILE=arm-none-linux-gnueabi- \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>HOSTARCH=$(TARGET_CONFIGURE_TARGET) \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>BUILDARCH=$(shell uname -m) \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>)</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span style="background-color: white;"><br /></span></div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span style="background-color: white;"># Interestingly, building this doesn't try to amend $(MUDDLE_SRC)/Lib, so</span></div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
# we don't need to copy the sources. Of course, the host-tools Python we</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
# use in our build will be using its copy of the sources when it needs them,</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
# but they are identical to those in $(MUDDLE_SRC), so we don't mind.</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
#</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
# To constrain the configure stage to only look at libraries in our</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
# MUDDLE_LIB_DIRS, we set LD_LIBRARY_PATH to MUDDLE_PKGCONFIG_DIRS_AS_PATH</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
config:</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span style="white-space: pre;"> </span>-mkdir -p $(MUDDLE_OBJ_OBJ)</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>(cd $(MUDDLE_OBJ_OBJ); \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>PYTHONPATH= \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>PYTHONHOME= \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>HOSTPYTHON='$(HOSTPYTHON)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>CFLAGS='$(CFLAGS)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>LDFLAGS='$(LDFLAGS)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>LD_LIBRARY_PATH=$(MUDDLE_PKGCONFIG_DIRS_AS_PATH) \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span> <span class="Apple-tab-span" style="white-space: pre;"> </span>$(MUDDLE_SRC)/configure \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>--host=$(TARGET_CONFIGURE_TARGET) \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>--prefix=/usr \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>)</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span style="background-color: white;"><br /></span></div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span style="background-color: white;">install:</span></div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span style="white-space: pre;"> </span>(cd $(MUDDLE_OBJ_OBJ); \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span> <span class="Apple-tab-span" style="white-space: pre;"> </span>make install \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>PYTHONPATH= \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>PYTHONHOME= \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>CFLAGS='$(CFLAGS)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>CPPFLAGS='$(CFLAGS)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>LDFLAGS='$(LDFLAGS)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>HOSTPYTHON='$(HOSTPYTHON)' \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span> <span class="Apple-tab-span" style="white-space: pre;"> </span>DESTDIR=$(MUDDLE_INSTALL) \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>CROSS_COMPILE_TARGET=yes \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>CROSS_COMPILE=arm-none-linux-gnueabi- \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>HOSTARCH=$(TARGET_CONFIGURE_TARGET) \</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>)</div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
endif</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">The thing I had wrong here, for a while, was not specifying </span><span style="background-color: white;"><span style="font-family: 'Trebuchet MS', sans-serif;">BUILDARCH</span>. This is needed by the patches used to tell the <span style="font-family: 'Trebuchet MS', sans-serif;">setup.py</span> code used to build <span style="font-family: 'Trebuchet MS', sans-serif;">_ctypes</span> what to do, and not specifying it caused ctypes support not to be compiled. This is the sort of mistake one needs to look out for when taking someone else's command lines (which work) and removing things from them "because they're not needed" - sometimes that's not so!</span></div>
<div style="font-family: 'Trebuchet MS', sans-serif;">
<br /></div>
<div>
<span style="font-family: inherit;">Finally, we have some common trailing code:</span></div>
</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;">clean:</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><span style="white-space: pre;"> </span>(cd $(MUDDLE_OBJ_OBJ); make clean)</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;">distclean:</span></div>
<div>
<span style="font-family: 'Trebuchet MS', sans-serif;"><span style="white-space: pre;"> </span>rm -rf $(MUDDLE_OBJ_OBJ)</span></div>
<div style="font-family: inherit;">
<br /></div>
</div>
<div>
<span style="font-family: inherit;">just as you might expect.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Meanwhile, I note that there was a talk on cross-compiling Python at PyCon 2012: </span><span style="background-color: white;"><a href="https://us.pycon.org/2012/schedule/presentation/11/">https://us.pycon.org/2012/schedule/presentation/11/</a>. There doesn't seem to be anything other than the abstract for the talk, but I do note that the last item in it is "</span><span style="background-color: white;">Invitation to discuss ways to make Python more accessible to embedded developers", so maybe there's some impetus towards a solution building up.</span></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-12002280108682967062012-05-15T14:39:00.001+01:002012-05-15T14:39:47.682+01:00More muddle improvementsTo wit:<br />
<br />
<br />
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">visdep</span> (in the sandbox) now uses '<span style="font-family: 'Trebuchet MS', sans-serif;">xdot'</span> to display, not '<span style="font-family: 'Trebuchet MS', sans-serif;">xdot.py</span>'. This means that on <span style="font-family: 'Trebuchet MS', sans-serif;">apt-get</span> based systems, <span style="font-family: 'Trebuchet MS', sans-serif;">sudo apt-get install xdot</span> should suffice to install the dependencies needed for <span style="font-family: 'Trebuchet MS', sans-serif;">visdep</span> to work.</li>
<li>The output of CPIO file generation is tidied up and made easier to understand (although still terribly verbose), and the irritating (at least to me) warnings have been fixed (this was a bug in applying instructions, in fact).</li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">muddle status</span> will now detect if a git checkout is <i>behind</i> the remote repository, without needing a <span style="font-family: 'Trebuchet MS', sans-serif;">git fetch</span>, and also reports the checkouts that need attention at the end of its output. See <span style="font-family: 'Trebuchet MS', sans-serif;">muddle help status</span>.</li>
<li>A long-standing bug has been fixed in cleaning deployments - this makes packages depending on deployments (which Richard introduced a while ago) work that bit better.</li>
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-31093827318705721532012-05-01T16:40:00.000+01:002012-05-01T16:40:20.419+01:00muddle 2.4.1 - "just pulled"Muddle 2.4.2 is now in the repository. It adds support for the <span style="font-family: "Trebuchet MS", sans-serif;">_just_pulled</span> command line argument, which allows you to do things like:<br />
<br />
<span style="font-family: "Trebuchet MS", sans-serif;"> muddle pull _all</span><br />
<span style="font-family: "Trebuchet MS", sans-serif;"> muddle rebuild _just_pulled</span><br />
<br />
See the documentation at <a href="http://muddle.readthedocs.org/en/latest/cmdline.html#special-command-line-arguments">http://muddle.readthedocs.org/en/latest/cmdline.html#special-command-line-arguments</a> for how it actually works.<br />
<br />
<span style="font-size: x-small;">(What, you may ask, was 2.4.1? Well, it was the version of upstream repository support which actually worked properly.)</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-32895086952772934602012-03-14T20:37:00.000+00:002012-03-14T20:37:03.911+00:00Version 2.4 of muddle is releasedThe main changes are below the cut.<br />
<br />
<a name='more'></a>There is a new command "<span style="font-family: "Trebuchet MS", sans-serif;">muddle distribute</span>", which allows distribution of
muddle build trees, for making a source or binary release, for GPL license
compliance, and so on. To support this, there is new funcitonality in muddle
to specify distributions, and to specify the licensing of checkouts and
packages. All of this is documented in the user manual, and in "<span style="font-family: "Trebuchet MS", sans-serif;">muddle help
distribute</span>".<br />
<br />
Note that "muddle distribute" and the distribution and license functions may
still change slightly as actual use reveals any raw edges.
<br />
<br />
There is also new support for upstream repositories, with corresponding new
commands "<span style="font-family: "Trebuchet MS", sans-serif;">muddle push-upstream</span>" and "<span style="font-family: "Trebuchet MS", sans-serif;">muddle pull-upstream</span>". This allows one
to name upstream (or remote) repositories and use them, as is commonly done
with Linux.
<br />
<br />
There are also a variety of other changes:<br />
<div>
<ul>
<li>Better exception handling when an error occurs in a build description,
especially in subdomain build descriptions - basically, not showing
tracebacks of uninteresting parts of the muddle infrastructure.</li>
<li>The help categories have been changed slightly, so "<span style="font-family: "Trebuchet MS", sans-serif;">muddle help categories</span>"
can take account of "<span style="font-family: "Trebuchet MS", sans-serif;">muddle distribute</span>".</li>
<li>"<span style="font-family: "Trebuchet MS", sans-serif;">muddle help</span>" doesn't fail when checking the column width if output is not
to a terminal.</li>
<li>The "<span style="font-family: "Trebuchet MS", sans-serif;">muddled.depend</span>" module provides new convenience functions "<span style="font-family: "Trebuchet MS", sans-serif;">checkout</span>",
"<span style="font-family: "Trebuchet MS", sans-serif;">package</span>" and "<span style="font-family: "Trebuchet MS", sans-serif;">deployment</span>" for creating labels - for instance,
"<span style="font-family: "Trebuchet MS", sans-serif;">checkout('fred')</span>".</li>
<li>Various internal fixes have been made.</li>
</ul>
As ever, <a href="http://muddle.readthedocs.org/en/latest/index.html">the muddle documentation</a> at ReadTheDocs has been automatically updated. There are new sections on:<br />
<ul>
<li><div>
Repositories (and in particular, upstream repositories and how to use them)</div>
</li>
<li><div>
Distributions and licenses</div>
</li>
<li>How instruction files are written</li>
</ul>
and don't forget the "not very frequently asked quesstions" in the Jottings section.</div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-57350042353905680392012-02-27T16:43:00.004+00:002012-02-27T16:54:45.756+00:00Kids! Want to use the ARM PMU event counters on DM3730?<p>There is a long-standing bug that prevents DM3730 users from using the ARM performance monitor counters on OMAP3 (my machine is a Beagle xM with a DM3730 on it).</p>
<p>The PMU counters are described in ARM DDI0344K, s. 3.2.42-3.2.51 and allow one to count interesting events such as data fetches, TLB and cache misses, and so on. There are four PMU counters and one cycle counter.</p>
<p>In order to access these from userspace, one must persuade a kernel module (or the kernel itself) to perform:</p>
<pre> {
int fred = 1;
asm("MCR p15,0,%[r], c9, c14, 0" : : [r] "r" (fred));
}
</pre>
<p>However, if one does this in a 'stock' 3.2.0 kernel (or before, I guess), one will find that only the cycle counter works - the others are all stuck at whatever value you last wrote into them (not necessarily zero, oddly enough).</p>
<p>This is because the four performance unit counters are clocked from the EMU clock domain, which is in turn in the EMU power domain, which is turned off by the kernel shortly after boot, since nothing is currently using it.</p>
<p>The "right way" to turn it on again would be by calling <tt>clk_enable()</tt> on <tt>emu_src_clk</tt>. However, a bug in the power management subsystem means that whilst this will attempt to turn on the EMU domain clocks, it will fail to turn on the EMU power domain and the counters will remain stuck.</p>
<p>Since that bit of the kernel is in flux right now, a quick hack to get you going is to find <tt>arch/arm/mach-omap2/clockdomains3xxx_data.c</tt> and change:
<pre>
static struct clockdomain emu_clkdm = {
.name = "emu_clkdm",
.pwrdm = { .name = "emu_pwrdm" },
.flags = /* CLKDM_CAN_ENABLE_AUTO | */ CLKDM_CAN_SWSUP,
.clktrctrl_mask = OMAP3430_CLKTRCTRL_EMU_MASK,
};
</pre>
<p>to</p>
<pre>
static struct clockdomain emu_clkdm = {
.name = "emu_clkdm",
.pwrdm = { .name = "emu_pwrdm" },
.flags = /* CLKDM_CAN_ENABLE_AUTO | *//*CLKDM_CAN_SWSUP*/ 0,
.clktrctrl_mask = OMAP3430_CLKTRCTRL_EMU_MASK,
};
</pre>
<p>Thus stopping the kernel from ever suspending that domain. No guarantees that this works over suspend/resume, but it at least allows you to clock the counters.</p>Richardhttp://www.blogger.com/profile/15413093214412156645noreply@blogger.com1tag:blogger.com,1999:blog-5625827309301120857.post-88328544214669051752012-02-03T17:02:00.000+00:002012-02-03T17:02:00.653+00:00muddle version 2 stamp file supportThe latest version of muddle now supports version 2 stamp files. See "<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">muddle help stamp</span>" (and the associated subcommands). There's a tag "<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">stamp-v2-support</span>" if you like that sort of thing.<br />
<br />
<a name='more'></a><br />
Version 2 stamp files support all the information stored in the new Repository class we're now using, and also store checkout options (i.e., typically the "shallow checkout" option for git).<br />
<br />
There's legacy support for outputting version 1 stamp files (just in case) - this makes a good try at working, but is obviously not guaranteed.<br />
<br />
There's somewhat more trusted support for <i>reading</i> version 2 stamp files, although again it may get corner cases wrong.<br />
<br />
I've still to finish writing up the actual format (in the doc string for <span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">muddled/verstion_stamp.py</span>), but I figured getting the support out for v2 stamps was worth doing, as otherwise we have the potential of losing information.<br />
<br />
Other changes:<br />
<br />
<ul>
<li>"<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">muddle --version</span>" will now report the branch as well, if it's not <span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">master</span></li>
<li>the odd support for handling branches in bzr by tacking their name on the end of the repository URL has been removed. I don't believe this should affect anyone.</li>
<li>there are various bugfixes, in the normal manner, and some extra methods on Repository</li>
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-45553288433027578192012-01-24T15:21:00.002+00:002012-01-24T15:24:36.498+00:00...and muddle continues to evolvemuddle v2.3.1 fixes a couple of bugs, and beyond that, head-of-tree adds "<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">muddle --version</span>", which uses git to report a sensible version string, and also the directory that muddle is being run from. For instance:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">$ muddle --version</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">muddle v2.3.1-3-g34519ca in /home/tibs/sw/muddle</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: inherit;">(That's the last tag (v2.3.1), the number of commits since then (3) and the start of the SHA1 string for this commit (34519ca))</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-54690107914951258852012-01-19T13:57:00.003+00:002012-01-19T13:57:47.930+00:00KBUS on Google Code now using GitThe KBUS repositories on <a href="http://code.google.com/p/kbus/">http://code.google.com/p/kbus/</a> have now been moved from Mercurial to Git.<br />
<br />
The "default" (userspace) and<b> kernel</b> repositories are now recombined. The changes from the kernel submission work on github have also been folded in.<br />
<br />
See <a href="http://code.google.com/p/kbus/wiki/DevelopmentHistory">http://code.google.com/p/kbus/wiki/DevelopmentHistory</a> for more information.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5625827309301120857.post-3009973549792936662012-01-12T21:52:00.003+00:002012-01-12T22:01:46.037+00:00Version 2.3 of muddle is releasedI took a branch of muddle out in September 2011 to write tests for domain support, which had "bit rotted" through disuse. As it turned out, my need for domain support went away again, and instead I started to use the branch for rewriting the help texts for each command. Of course, in order to do that, I had to understand exactly what each command did, which led to rewriting the main command structure so that this was <i>consistent</i>, and hopefully also <i>predictable</i>, and thus easier to document. So it all took rather longer than planned.<br />
Meanwhile, another project recently necessitated adding support for git repositories on Google Code, which in turn required rewriting the internal repository and checkout support. So that also got folded in.<br />
This version of muddle is now stable enough to merge back into the mainline. I've been using it for real development work throughout its development, and there are more tests than there used to be (although still not enough subdomain tests!). However, clearly, if you come across a bug or infelicity that I've introduced, do let me know (preferably by raising an issue at <a class="reference external" href="http://code.google.com/p/muddle/issues/list">http://code.google.com/p/muddle/issues/list</a>, but otherwise by email to <a class="reference external" href="mailto:tibs@kynesim.co.uk">tibs@kynesim.co.uk</a>).<br />
Since there's a large amount of sudden change, much of it to significant internal structure, I'm giving this new version of muddle a version tag, and calling it version 2.3 (SVN revision 638 (back when muddle was stored in subversion) had tag 2_2).<br />
<blockquote>(The tag "end_of_v2.2" marks the end of version 2.2, and there is also a branch "v2.2-legacy" from the same location, which can be conveniently used to checkout the end of the 2.2 sequence, and also provides a place for any maintenance changes if such prove necessary.)</blockquote>The main changes are:<br />
<ul><li><div class="first">Much of the the internals of the command line processing has been rewritten, and this should give more robust command handling. It also means that commands of a common "category" (see 'muddle help categories') share much more common code, and thus have better consistency.</div></li>
<li><div class="first">Much of the help text has been reworked, and there are new topics:</div><ul class="simple"><li>'muddle help categories'</li>
<li>'muddle help labels'</li>
<li>'muddle help subdomains'</li>
</ul>Also, help text is now paged (much as is done by git), using the pager specified by $PAGER, or else 'more'. There is a command line switch, '--nopager', to rever to the old behaviour.</li>
<li><div class="first">An attempt has been made to make the "bare" muddle command (i.e., 'muddle' with no arguments) follow the DWIM (Do What I Mean) principles described in the documentation. See 'muddle help labels' for more information on how this works.</div></li>
<li><div class="first">Similarly, muddle checkout/package/dependency commands with no label arguments (e.g., 'muddle build') now try to decide consistently what to do. Again, see 'muddle help labels'.</div></li>
<li><div class="first">All checkout/package/dependency commands now know how to cope with any type of label. See 'muddle help labels' for details.</div></li>
<li><div class="first">Most commands that take labels as arguments can now take partial labels (label fragments) instead, which is a lot simpler for a human being. This includes the 'visdep' script in the sandbox. See 'muddle help labels' for more information.</div></li>
<li><div class="first">'muddle -n' now works for all commands that do something (it used to be very unpredictable about this). It is now a reliable tool, and may be used as such. Also, note that it reports a single label per line, which makes it much easier to read.</div></li>
<li><div class="first">As well as the special '_all' argument, there are now also '_default_deployments' and '_default_roles', which expand to all the default deployment labels and all the package labels in the default roles, respectively. Thus the top-level "bare" 'muddle' command is identical to 'muddle buildlabel _default_deployments _default_roles'.</div>However, note that '_all' is now treated as a special value for use on the commandline, and not as a "pseudo-wildcard" label name. Thus it is no longer meaningful to ask for <tt class="docutils literal">package:_all{x86}</tt> when one meant <tt class="docutils literal"><span class="pre">package:*{x86}</span></tt> (the former will report that there is no package called <tt class="docutils literal">_all{x86}</tt>).</li>
<li><div class="first">What 'muddle where' says has changed in detail, and there is now 'muddle where -detail' for use in scripts.</div></li>
<li><div class="first">When building things, or "killing" things, muddle is significantly less verbose - it doesn't normally report ALL the labels it will affect.</div></li>
<li><div class="first">More and hopefully more useful error messages have been introduced. In particular, I've tried to catch occasions where muddle unexpectedly does nothing, and suggest why this might be (liking trying to do 'muddle build' in a checkout directory when the only role for the corresponding package is not one of the default roles - this one has taken me too long to work out in the past). Some of these error reports are more verbose, or not as easy to understand, as they should be, but this is hoped to be an important improvement.</div></li>
<li><div class="first">When a syntax or other error is found in a build description, it is no longer buried in a traceback concerned with how the build description was being imported.</div></li>
<li><div class="first">muddle commands now try harder to catch cases where the user has specified a role for a checkout or deployment label, and grumble.</div></li>
<li><div class="first">Various command aliases have been removed:</div><ul class="simple"><li>'muddle query deps' - use 'needed-by'</li>
<li>'muddle query instructions' - use 'inst-files'</li>
<li>'muddle query makeenv' - use 'make-env'</li>
<li>'muddle query preciseenv' - use 'precise-env'</li>
<li>'muddle query results' - use 'needs'</li>
</ul>Note that 'muddle query envs' is still an alias for 'muddle query all-envs', and this is still different than 'muddle query env'.<br />
Also, as it turned out, 'muddle uncheckout' and 'muddle removed' did exactly the same thing. 'muddle removed' has thus been removed.</li>
<li><div class="first">Some commands have been changed:</div><ul class="simple"><li>'muddle dependencies' is replaced by 'muddle query dependencies'</li>
<li>'muddle depends' is replaced by 'muddle query depends' (this is actually the same as the previous item)</li>
<li>'muddle vcs' is replaced by 'muddle query vcs'</li>
<li>'muddle root' has gone. Instead use 'muddle query root', which already existed. 'muddle query root' now just reports the root (of the build tree), and not the "default domain", since this concept has essentially gone away.</li>
<li>'muddle query default-labels' is now the more accurate 'muddle query default-deployments'</li>
</ul></li>
<li><div class="first">'muddle query package-roles' will also include domains in the reported names, if appropriate.</div></li>
<li><div class="first">'muddle pull' is once again preferred over 'muddle fetch' or 'muddle update', but the other variants are still there.</div></li>
<li><div class="first">The 'utils.Directory' classes now set $PWD (as top-level muddle itself does, and for the same reason). They also now have a useful 'join' method (see the classes for its use). A version of these classes has been pulled out to a separate repository on github - see <a class="reference external" href="https://github.com/tibs/withdir">https://github.com/tibs/withdir</a></div></li>
<li><div class="first">There is now more testing! (particularly of the 'muddle -n' commands, and of subdomain handling - necessary because the latter had "bit rotted").</div></li>
<li><div class="first">Various obscure bugs have been found and fixed by the new tests. Some of them were really <i>quite</i> obscure.</div></li>
<li><div class="first">Various things that were to be deprecated have now been removed. Hopefully no-one will notice.</div></li>
<li><div class="first">'NullPackage' and 'null_package()' have been added to muddled/pkg.py. Their docstrings explain why they might be useful.</div></li>
<li><div class="first">'Rule' classes now have an 'action' member, instead of an 'obj' member. I believe this to be much clearer, but it is an unannounced incompatible change.</div></li>
<li><div class="first">Repository handling has been rewritten, separating out the concerns of naming a (remote) repository and naming (the location of) a checkout.</div>In particular, and importantly, it now understands how to use git repositories at google code, where the repository name is appended to the "main" URL with a dot, rather than a "/".<br />
The old commands for setting things up (checkouts.twolevel and so on) are still there, but one can now do:<br />
<pre class="literal-block">from muddled.repository import Repository
from muddled.version_control import checkout_from_repo
repo = Repository('git', 'https://example.com', 'useful-stuff',
branch='very-useful')
checkout_from_repo(builder, 'useful-stuff', repo)
</pre>althogh there's actually more to it than that - in particular, one can specify a different <co_dir> and <co_leaf> to checkout_from_repo, giving one full control over where the repository is actually checked out:<br />
<pre class="literal-block">checkout_from_repo(builder, 'useful-stuff', repo, co_dir='jim',
co_leaf='fred')
</pre>checks the repository out into 'src/jim/fred', but still using the label 'checkout:useful-stuff/checked_out'. I'd claim this is significantly easier to understand than the twoleve/multilevel functions.<br />
See the documentation for Repository via "muddle doc repository.Repository", or in its source file.<br />
There is now thus a new command "muddle query checkout-repos" to report on the repositories. See its help for details.<br />
This new mechanism is believed to understand muddle stamp files generated by previous versions of muddle, and to be able to produce muddle stamp files that previous versions of muddle should understand.</li>
<li><div class="first">Internal changes that may be of interest:</div><ul class="simple"><li>'builder.invocation.default_labels' is now 'builder.invocation.default_deployment_labels'.</li>
<li>Similarly, the method 'add_default_label()' becomes 'add_default_deployment_label()'. It also now checks that the label <i>is</i> a deployment label.</li>
</ul></li>
</ul>Also, of course, many minor code tidies have been made.<br />
<div class="section" id="particular-issues-fixed"><h1>Particular issues fixed</h1><ul class="simple"><li>Fixes issue 4. Bugs in description files are not handled elegantly</li>
<li>Fixes issue 67. Checkout and package inference should know about domains</li>
<li>Fixes issue 104. No feedback on undefined label name</li>
<li>Fixes issue 112. Python 2.7 (I'm now using that for testing as well)</li>
<li>Fixes issue 126. Provide coherent branch support in the VCS plugins</li>
<li>Fixes issue 166. test_checkouts.py broken by fix for issue 161</li>
<li>Fixes issue 175. Repository name should be independent of local checkout name</li>
<li>Fixes issue 177. 'muddle checkout package:xxx' should checkout what is needed for package xxx</li>
<li>Fixes issue 179. revisit the implementation of the simple/twolevel/multilevel checkout code</li>
<li>Fixes issue 184. All the muddle commands that take labels should support domains</li>
<li>Fixes issue 185. Muddle unstamp should check its build against domain/checkout name, not just checkouts name</li>
<li>Fixes issue 186. Deprecate use of 'all_packages', use 'all_package_labels' instead</li>
<li>Fixes issue 189. Make sure all commands do something vaguely sensible for '-n'</li>
<li>Fixes issue 194. 'muddle pull' of a toplevel checkout in a subdomain attempts to clone</li>
<li>Fixes issue 195. If there's nothing to do, say so</li>
<li>Fixes issue 198. Muddle should actually honour the DWIM section from the README</li>
<li>Fixes issue 199. Helpful error behaviour of fetch/pull unexpected (well, ok, this is in master as well)</li>
<li>Fixes issue 200. Find all targets for deployments</li>
<li>Fixes issue 207. Muddle assumes too much about how RootRepository and Description are joined<b><span style="font-size: large;"></span></b></li>
</ul><br />
<ul class="simple"></ul></div><div class="section" id="updating-to-the-new-version"><h1>Updating to the new version</h1>...is simple, just <span style="font-family: "Courier New", "Courier", monospace;">cd</span> into your muddle source directory and <span style="font-family: "Courier New", "Courier", monospace;">git pull</span> in the normal manner.<br />
<br />
If you find some serious bug and need to go back to version 2.2, then <span style="font-family: "Courier New", "Courier", monospace;">cd</span> into the muddle source directory and do <span style="font-family: "Courier New", "Courier", monospace;">git checkout v2.2-legacy</span><span style="font-family: inherit;"> (and, obviously, report the problem).</span></div>Unknownnoreply@blogger.com0