kynesim

...and muddle continues to evolve

2012-01-24T15:21:00.002Z

muddle v2.3.1 fixes a couple of bugs, and beyond that, head-of-tree adds "muddle --version", which uses git to report a sensible version string, and also the directory that muddle is being run from. For instance:

$ muddle --version
muddle v2.3.1-3-g34519ca in /home/tibs/sw/muddle

(That's the last tag (v2.3.1), the number of commits since then (3) and the start of the SHA1 string for this commit (34519ca))

KBUS on Google Code now using Git

2012-01-19T13:57:00.003Z

The KBUS repositories on http://code.google.com/p/kbus/ have now been moved from Mercurial to Git.

The "default" (userspace) and kernel repositories are now recombined. The changes from the kernel submission work on github have also been folded in.

See http://code.google.com/p/kbus/wiki/DevelopmentHistory for more information.

Version 2.3 of muddle is released

2012-01-12T21:52:00.003Z

I took a branch of muddle out in September 2011 to write tests for domain support, which had "bit rotted" through disuse. As it turned out, my need for domain support went away again, and instead I started to use the branch for rewriting the help texts for each command. Of course, in order to do that, I had to understand exactly what each command did, which led to rewriting the main command structure so that this was consistent, and hopefully also predictable, and thus easier to document. So it all took rather longer than planned.
Meanwhile, another project recently necessitated adding support for git repositories on Google Code, which in turn required rewriting the internal repository and checkout support. So that also got folded in.
This version of muddle is now stable enough to merge back into the mainline. I've been using it for real development work throughout its development, and there are more tests than there used to be (although still not enough subdomain tests!). However, clearly, if you come across a bug or infelicity that I've introduced, do let me know (preferably by raising an issue at http://code.google.com/p/muddle/issues/list, but otherwise by email to tibs@kynesim.co.uk).
Since there's a large amount of sudden change, much of it to significant internal structure, I'm giving this new version of muddle a version tag, and calling it version 2.3 (SVN revision 638 (back when muddle was stored in subversion) had tag 2_2).

(The tag "end_of_v2.2" marks the end of version 2.2, and there is also a branch "v2.2-legacy" from the same location, which can be conveniently used to checkout the end of the 2.2 sequence, and also provides a place for any maintenance changes if such prove necessary.)

The main changes are:

Much of the the internals of the command line processing has been rewritten, and this should give more robust command handling. It also means that commands of a common "category" (see 'muddle help categories') share much more common code, and thus have better consistency.
Much of the help text has been reworked, and there are new topics:
- 'muddle help categories'
- 'muddle help labels'
- 'muddle help subdomains'
Also, help text is now paged (much as is done by git), using the pager specified by $PAGER, or else 'more'. There is a command line switch, '--nopager', to rever to the old behaviour.
An attempt has been made to make the "bare" muddle command (i.e., 'muddle' with no arguments) follow the DWIM (Do What I Mean) principles described in the documentation. See 'muddle help labels' for more information on how this works.
Similarly, muddle checkout/package/dependency commands with no label arguments (e.g., 'muddle build') now try to decide consistently what to do. Again, see 'muddle help labels'.
All checkout/package/dependency commands now know how to cope with any type of label. See 'muddle help labels' for details.
Most commands that take labels as arguments can now take partial labels (label fragments) instead, which is a lot simpler for a human being. This includes the 'visdep' script in the sandbox. See 'muddle help labels' for more information.
'muddle -n' now works for all commands that do something (it used to be very unpredictable about this). It is now a reliable tool, and may be used as such. Also, note that it reports a single label per line, which makes it much easier to read.
As well as the special '_all' argument, there are now also '_default_deployments' and '_default_roles', which expand to all the default deployment labels and all the package labels in the default roles, respectively. Thus the top-level "bare" 'muddle' command is identical to 'muddle buildlabel _default_deployments _default_roles'.
However, note that '_all' is now treated as a special value for use on the commandline, and not as a "pseudo-wildcard" label name. Thus it is no longer meaningful to ask for package:_all{x86} when one meant package:*{x86} (the former will report that there is no package called _all{x86}).
What 'muddle where' says has changed in detail, and there is now 'muddle where -detail' for use in scripts.
When building things, or "killing" things, muddle is significantly less verbose - it doesn't normally report ALL the labels it will affect.
More and hopefully more useful error messages have been introduced. In particular, I've tried to catch occasions where muddle unexpectedly does nothing, and suggest why this might be (liking trying to do 'muddle build' in a checkout directory when the only role for the corresponding package is not one of the default roles - this one has taken me too long to work out in the past). Some of these error reports are more verbose, or not as easy to understand, as they should be, but this is hoped to be an important improvement.
When a syntax or other error is found in a build description, it is no longer buried in a traceback concerned with how the build description was being imported.
muddle commands now try harder to catch cases where the user has specified a role for a checkout or deployment label, and grumble.
Various command aliases have been removed:
- 'muddle query deps' - use 'needed-by'
- 'muddle query instructions' - use 'inst-files'
- 'muddle query makeenv' - use 'make-env'
- 'muddle query preciseenv' - use 'precise-env'
- 'muddle query results' - use 'needs'
Note that 'muddle query envs' is still an alias for 'muddle query all-envs', and this is still different than 'muddle query env'.
Also, as it turned out, 'muddle uncheckout' and 'muddle removed' did exactly the same thing. 'muddle removed' has thus been removed.
Some commands have been changed:
- 'muddle dependencies' is replaced by 'muddle query dependencies'
- 'muddle depends' is replaced by 'muddle query depends' (this is actually the same as the previous item)
- 'muddle vcs' is replaced by 'muddle query vcs'
- 'muddle root' has gone. Instead use 'muddle query root', which already existed. 'muddle query root' now just reports the root (of the build tree), and not the "default domain", since this concept has essentially gone away.
- 'muddle query default-labels' is now the more accurate 'muddle query default-deployments'
'muddle query package-roles' will also include domains in the reported names, if appropriate.
'muddle pull' is once again preferred over 'muddle fetch' or 'muddle update', but the other variants are still there.
The 'utils.Directory' classes now set $PWD (as top-level muddle itself does, and for the same reason). They also now have a useful 'join' method (see the classes for its use). A version of these classes has been pulled out to a separate repository on github - see https://github.com/tibs/withdir
There is now more testing! (particularly of the 'muddle -n' commands, and of subdomain handling - necessary because the latter had "bit rotted").
Various obscure bugs have been found and fixed by the new tests. Some of them were really quite obscure.
Various things that were to be deprecated have now been removed. Hopefully no-one will notice.
'NullPackage' and 'null_package()' have been added to muddled/pkg.py. Their docstrings explain why they might be useful.
'Rule' classes now have an 'action' member, instead of an 'obj' member. I believe this to be much clearer, but it is an unannounced incompatible change.
Repository handling has been rewritten, separating out the concerns of naming a (remote) repository and naming (the location of) a checkout.
In particular, and importantly, it now understands how to use git repositories at google code, where the repository name is appended to the "main" URL with a dot, rather than a "/".
The old commands for setting things up (checkouts.twolevel and so on) are still there, but one can now do:
```
from muddled.repository import Repository
from muddled.version_control import checkout_from_repo

repo = Repository('git', 'https://example.com', 'useful-stuff',
                  branch='very-useful')
checkout_from_repo(builder, 'useful-stuff', repo)
```
althogh there's actually more to it than that - in particular, one can specify a different <co_dir> and <co_leaf> to checkout_from_repo, giving one full control over where the repository is actually checked out:
```
checkout_from_repo(builder, 'useful-stuff', repo, co_dir='jim',
                   co_leaf='fred')
```
checks the repository out into 'src/jim/fred', but still using the label 'checkout:useful-stuff/checked_out'. I'd claim this is significantly easier to understand than the twoleve/multilevel functions.
See the documentation for Repository via "muddle doc repository.Repository", or in its source file.
There is now thus a new command "muddle query checkout-repos" to report on the repositories. See its help for details.
This new mechanism is believed to understand muddle stamp files generated by previous versions of muddle, and to be able to produce muddle stamp files that previous versions of muddle should understand.
Internal changes that may be of interest:
- 'builder.invocation.default_labels' is now 'builder.invocation.default_deployment_labels'.
- Similarly, the method 'add_default_label()' becomes 'add_default_deployment_label()'. It also now checks that the label is a deployment label.

Also, of course, many minor code tidies have been made.

Particular issues fixed

Fixes issue 4. Bugs in description files are not handled elegantly
Fixes issue 67. Checkout and package inference should know about domains
Fixes issue 104. No feedback on undefined label name
Fixes issue 112. Python 2.7 (I'm now using that for testing as well)
Fixes issue 126. Provide coherent branch support in the VCS plugins
Fixes issue 166. test_checkouts.py broken by fix for issue 161
Fixes issue 175. Repository name should be independent of local checkout name
Fixes issue 177. 'muddle checkout package:xxx' should checkout what is needed for package xxx
Fixes issue 179. revisit the implementation of the simple/twolevel/multilevel checkout code
Fixes issue 184. All the muddle commands that take labels should support domains
Fixes issue 185. Muddle unstamp should check its build against domain/checkout name, not just checkouts name
Fixes issue 186. Deprecate use of 'all_packages', use 'all_package_labels' instead
Fixes issue 189. Make sure all commands do something vaguely sensible for '-n'
Fixes issue 194. 'muddle pull' of a toplevel checkout in a subdomain attempts to clone
Fixes issue 195. If there's nothing to do, say so
Fixes issue 198. Muddle should actually honour the DWIM section from the README
Fixes issue 199. Helpful error behaviour of fetch/pull unexpected (well, ok, this is in master as well)
Fixes issue 200. Find all targets for deployments
Fixes issue 207. Muddle assumes too much about how RootRepository and Description are joined

Updating to the new version

...is simple, just cd into your muddle source directory and git pull in the normal manner.

If you find some serious bug and need to go back to version 2.2, then cd into the muddle source directory and do git checkout v2.2-legacy (and, obviously, report the problem).

How to discuss our open source projects?

2011-09-26T16:59:00.001+01:00

So we now have a collection of Kynesim open source projects:

muddle
KBUS
cdatecalc
tstools
grump

and another one to be added soon. What we don't have is any way of making announcements about them (other than this blog - not really ideal) or holding discussion on them (issue tracking I reckon is OK, as I quite like the Google Code issue mechanisms).

Mailing lists are a pain, and none of them make using their archives particularly nice.

Google groups are a possibility, although they're not perfect either.

Google+ won't work because I won't use it (so that discussion is moot).

Any good ideas?

Still improooving muddle - some commands slightly changed.

2011-09-26T16:16:00.000+01:00

I'm still trying to fix various problems in how muddle handles subdomains, and also to add some more testing. However, I've just pushed a somewhat incompatible change to the way that some of the command line tools work.

In the past, if you did something like muddle fetch at the top level of the build tree, it didn't actually do anything, and you had to do muddle fetch _all. This was, in fact, a bug, and those commands which take checkout arguments will now, in general, always default to the checkouts "below" them, including at the top level (where all checkouts are "below").

Of course, it's also an incompatible change. On the other hand, I don't know how many other people still had the habit of typing muddle fetch (etc.) with the vague feeling that they should be doing something that they weren't...

As normal, please raise any issues on the Google Code issues page (and I get an email at home when you do so).

Finally, muddle has a (VCS) status command

2011-09-09T11:22:00.000+01:00

It's been a long wait, but Richard and I both really needed it...

You can now say muddle status and get a report on the subversion/bazaar/git status of checkouts (i.e., the version control systems we currently support). It tries not to say anything for checkouts with no interesting status. See muddle help status for details.

KBUS and muddle documentation now on ReadTheDocs

2011-08-30T21:03:00.000+01:00

The HTML documentation for both KBUS and muddle is now being hosted on Read the Docs, who are providing a wonderful service to those of us using Sphinx for our documentation.

See http://kbus.readthedocs.org/ and http://muddle.readthedocs.org/ respectively.

The documentation should be rebuilt each time a push is made to the appropriate repository. Including, of course, a change to the documentation. So I've no excuse, then...

muddle is now stored in git

2011-08-29T18:21:00.000+01:00

As of just now, the muddle Google Code repository has been converted to git.

If you're still accessing muddle via subversion, then you are using a "frozen" version, which will not be developed further. The last subversion revision is revision 638. Also note that due to difficulties in converting our subversion repository to git (strange stuff, honestly), only trunk has been converted, and at that only since revision 78 (around when something strange seems to have happened, probably to do with trying to merge stuff in subversion).

Cloning muddle is done as one might expect:

git clone https://code.google.com/p/muddle/

Why do this? Well, it makes it easier to development locally (distributed version control systems, yeah), and much easier to branch (last time I tried that in subversion was a nightmare).

Any problems, please contact me and ask.

Muddle dependency visualisation

2011-06-17T15:42:00.003+01:00

For about as long as I've been working with muddle, I've felt that it was lacking the ability to get a big-picture view of how a build tree fits together. This led to issue 48. It's not that I have anything against build descriptions being declarative-type programming; just that I sometimes find them hard to follow from scratch and prefer to get an overview.

Over time I hatched a plan to actually do something about this. Well, muddle knows everything it is possible to know about the dependencies; couldn't I put that knowledge to use?

Betraying myself as more of a perl hacker, I initially wrote a perl script which invoked muddle depend user-short, scraped the output and output in dot format. It was grotty and inelegant in a number of ways, but it served as a valid - if slow - proof of concept. Later, I rewrote it properly in python, directly examining the build description. So I had given myself the ability to create dot graphs of a build tree, but it still wasn't all that useful: every label was a separate node, and there were far too many of them, making the output cluttered and hard to understand.

The key insight in reducing the output was that while a muddle package usually gave rise to five labels - preconfig, configured, build, installed, postinstalled - these always appear in strict order. So why not conflate the nodes on the graph? It wasn't quite as simple as unconditionally merging them, as (for example) sometimes one package's built step depends on another package's configured, so I had it search for such dependencies and only reduce the graph as far as it could without throwing away information.

The script was duly committed to the sandbox directory, and languished there for a while. Just recently, Tibs has picked it up again. The graphs output by the script now sport colour-coding to differentially highlight checkouts, packages and deployments - and all of a sudden it's much more legible!

Alongside visualise-dependencies.py there is now a visdep.py wrapper which feeds the output into José Fonseca's xdot viewer script. The wrapper adds the options to choose amongst the various alternative layout filter algorithms (we find fdp often works well), and to pass the graph through tred (transitive reduction of vertices, e.g. if A depends on B, B on C, and A directly on C, you can drop the A-C dependency as it is implicit from the other two).

For example, a build tree I've been working on recently has a beagle-boot deployment, so here is the output from ~/muddle/sandbox/visdep.py deployment:beagle-boot/deployed :

If you want to have a play, you'll find it in the sandbox directory. For the time being you will need to install xdot.py somewhere on your PATH (see the comment at the top of visdep.py). Then invoke visdep.py, telling it the label(s) you are interested in.

The script is not perfect, being a bit hamstrung by how well dot and friends cope (or fail to cope) with complex graphs: particularly in nontrivial build trees, the output is frequently too cluttered or too sparse. Nevertheless, as you can see from the example above, it's capable of doing a very smart job! We dream of creating a more interactive viewer, perhaps one which allows the user to zoom around a three-dimensional rendering of the graph, but that will have to wait for now...

Muddle logo

2011-06-17T15:00:00.004+01:00

Muddle has a new logo!

It came about when I was scrawling notes during a discussion. I found myself instinctively writing an M in a circle as a shorthand for muddle. I showed it around and it was well received.

So a few minutes work in Inkscape cobbling things together, and hey presto! A logo is born. It's a derivative of the FreeSans font ('@' + 'm') so is covered by the FreeFont license, which is GPLv3 with an exception which does not cause documents using it to become bound by the GPL.

KBUS surgery in progress

2011-01-12T17:35:00.007Z

If you've browsed the kbus source on googlecode recently, you may have noticed that it has grown a few more repositories - kernel, cppkbus, jkbus and web. These are in flux at the moment; the eventual goal is to split them and their history out of the 'default' repository. In the meantime, we're going to leave the default repository untouched until we're ready to throw the Big Switch - do not adjust your set!

Mercurial's subrepository support is rather neat. At the moment the plan is to have the kbus 'default' repository check out everything you need to use it (kernel module, libkbus, tools, jkbus and cppkbus) along with the docs source; the built HTML docs, talks and slides will be hived off to their own repository so you don't have to download them if you don't want to.

KBUS Updates

2011-01-03T17:39:00.000Z

There are now C++ and Java bindings for KBUS. Also, the repository has been moved to Mercurial.

The C++ API was sketched out by Richard Watts, fleshed out by myself, and then bugfixed and used for an actual project by Richard. It provides a complete interface to the KBUS functionality. However, it does not use exceptions at all (or RTTI) since it is known that some users will need to be able to compile it on embedded systems where such cannot be used. This leads to a slightly different API than might otherwise be expected. Richard also introduced two innovations which may be backported to the other APIs at some point - a specific Device class to represent a KBUS device, which makes a lot of sense when doing non-Ksock specific operations, and the ability to make a message a Reply or Request at "send".

The Java interface was originally implemented by Gareth Bailey (who did the native methods which interface to the KBUS C library). I then expanded it somewhat, and Richard has added the functionality he needed. This is not a complete KBUS API, but provides all the normal functions for day-to-day use.

Both are present in the repository as an initial commit and then a "lump" commit of the results of the development off-line for the project concerned. Apologies for that, but it was easier to just work with the internal repositories during development. The disadvantage, of course, is loss of detailed VCS history.

That commit was then the last commit to KBUS as an SVN repository. The project has now been converted to Mercurial. As normal with Google-hosted projects, the SVN repository will continue to be available "read-only" in its final state. I'm hoping use of Mercurial should aid development in various ways (I'd actually have gone for git if I could).

The next change to the project is likely to be moving the documentation (certainly the generated files and the various PDF and Keynote files) to a separate repository - this should bring down the size of the KBUS source download considerably.

Gingerbread branch appears to be open..

2011-01-02T01:33:00.005Z

Those of you who watch the android open-source tree will have noticed a couple of interesting additions lately; specifically, the gingerbread manifest branch and platform/external/chromium.git.

I also notice that someone has been paying attention to the OOM killer and KVM in the android common kernel tree. Sadly, I think KVM is probably a bit heavyweight for this generation of devices, but containers could be a bit of a contender for allowing you to run multiple virtual STBs at the same time - might be a nice way to store your STB (and PVR) in the cloud and have it virtualised onto whatever hardware you happened to be watching at the moment.

For what it's worth, head seems to be building fine for me and the changes thus far seem quite easy to live with. Certainly much nicer to random undocumented API users like me than Linux was at the corresponding stage in its development ..

Also, Happy New Year!

Interesting facts about USB, number 2312 in an occasional series.

2010-12-31T17:21:00.008Z

Ah, USB 2.0. How I don't love thee in any way at all.

I've recently been hacking the USB drivers on an SoC we have here. The initial problem was that a USB mouse connected via a high-speed hub was acting erratically.

[Update: for what it's worth, it turns out that none of the below actually works. But I do now have some nice clean traces showing a high-speed USB hub apparently blatantly failing to report transaction results back to the host. Hey ho. Work continues.. ]

It turns out that this behaviour is common to all mice and all the high-speed hubs we had about, and to our USB-IR widgets, which would occasionally send key-down without key-up, leading to fun with Android deciding that key-repeat was clearly what you meant and spinning your keypresses repeatedly.

We asked the manufacturer for support and they reckoned it worked fine at their end with one of these.

So: next things to do were to bug a couple of the hubs that seemed to work, get ourselves a hub we knew wasn't broken, pull out the USB analysers and see what we could see.

Fact #0: A Totalphase Beagle 480 analyser will mysteriously claim 'Host Disconnected' at the slightest provocation. This includes when your hub power supply is 4.97V - nothing else will care, but the Beagle 480 will look as though you've broken it. So that took me two days to work out .. (amusingly, a Beagle 12 seems fine)

Fact #1: The Slim 4 port USB2.0 hubs that worked are full-speed.

Fact #2: The problem still exists on a self-built board using SMSC9514 (we wanted ethernet too) - my go-to 'it really does work' high-speed hub chip, as used on the Beagle xM.

Fact #3: The problem goes away if you connect the hub to a PC rather than our embedded board.

Fact #4: Our SoC uses the Designware USB 2.0 OTG host controller IP. This is an out-of-tree driver from our vendor, though it's been submitted for inclusion in the mainline kernel - here, but the DENX people have made a few fixes which we'd merged into the vendor-supplied driver. So the host side USB driver is even more of a mess than the stock Designware drivers, which are in themselves grotty in the extreme.

Fact #5: If you attach a Beagle 480 to the host side of the hub, you get:

IN txn (SPLIT) [ 1117 POLL ] 00 02 00 28 00 00 00 00 ..

whereas if you attach the same analyser to the device side, you find that our USB-IR widget (for it is he) sends:

IN txn [ 105 POLL ] 00 02 00 28 00 00 00 00 [4 SOF] IN txn 00 01 00 00 00 00 00 00

So: the USB-IR widget is sending key-up, but it's never being received on the high-speed bus.

At this point, a little background. DANGER WILL ROBINSON! This is highly simplified and almost certainly contains mistakes - the USB 2.0 standard has a good (if long-winded) explanation of transaction translation which it is highly recommended that you read if you ever have to seriously care about these things.

USB 2.0 runs much faster than USB 1.x. USB 1.x timing is divided into 1ms frames whose timing is broadcast by SOF tokens. In USB 2.x we further subdivide frames into 125uS subframes.

USB is a host-controlled token-based bus; the host issues tokens (IN, OUT .. ) and devices respond.

Now, obviously, if you have a high-speed and a full-speed device attached to a high-speed hub you don't want to be waiting around for the full-speed device to respond to your token before servicing the high-speed device. So there is a token called SPLIT, further subdivided into SSPLIT (Start SPLIT) and CSPLIT (Complete SPLIT).

When a host wants to issue a full-speed transaction to a downstream device, it issues SSPLIT to the (high-speed) hub port in some given micro-frame. This instructs a bit of mechanism in the hub called the transaction translator to initiate a full-speed transaction on a given port, send the packet (IN/OUT) that came with the ssplit, collect the answer and wait for the host to issue a CSPLIT to retrieve it.

Now, there are some caveats about split transactions:

There is a limited amount of buffer space per TT. Old transactions will be thrown away when the limit is reached.

Replies will be discarded when 4 or more microframes have passed since the SSPLIT that issued them.

CSPLITs for which the hub can find no replies will get NYET (forever if the hub will never get a reply or has discarded it).

You must not issue an SSPLIT in subframe 6.

You can have one TT per hub or one TT per port (the hub descriptor tells you which).

You may have multiple concurrent SPLITs , but you must issue CSPLITs in the order in which you issued SSPLITS or the hub will assume it has lost your CSPLIT and discard transaction results.

The hub takes some number of FS bit times (8 for the 9514) to issue an FS transaction after a CSPLIT - this is signalled in the hub descriptor.

To make life even more fun, HID devices like keyboards and mice are periodically polled, so there is something in the host called the periodic scheduler which attempts to make sure that each device is polled at least as often as its descriptor would like it to be. Luckily this is a best-effort polling mechanism.

.. oh, and for added amusement, you can also have multiple transactions per subframe though you can ignore this for the purposes of this article. There are also things called isochronous transfers which do need to be real-time, and I have probably broken what little support there was for them in the synopsys drivers.

The upshot of all this is that you need to be extremely careful when issuing splits - particularly periodic splits. Ideally you want to do it in hardware. Less ideally, you need to make sure your periodic scheduler knows about splits and respects the in-order and 'not in microframe 6' constraints.

So. Our next bet was to tie a Beagle 480 to the hub side of the system under test and a Beagle 12 to the device side; this doesn't give you as much information as you might like because a Beagle 12 is only capable of aggregate (rather than sequential) captures and so presents its results out of order, but it's good enough for now.

Monitoring both sides of the bus simultaneously we discover that the synopsys drivers are guilty of several sins:

1. Issuing SSPLITs in frame 6 (believe it or not, the 9514 actually cares about this and will NYET not the frame 6 SSPLIT but any SSPLIT in frame 7. It's allowed to, but this seems odd)

2. Issuing CSPLITs out of order from the SSPLITs that created them, causing the results of previous SSPLITs to be discarded.

3. There is some logic in the synopsys drivers intended to avoid runaway CSPLITs such that if you are about to issue a CSPLIT in a different full-speed frame from the SSPLIT that created it, you abort the transaction. This causes SSPLITs issued in subframe 7 to never get a corresponding CSPLIT (and a similar slight problem for those in subframe 5, though the chances are that if you're going to get an answer you will have got it by subframe 7)

4. Skipping two subframes between SSPLIT and CSPLIT - so you get SSPLIT:0, CSPLIT:3 ..; it's unclear why that CSPLIT gets NYETed - the 9514 ought to keep your results for you until the start of subframe 4 - but nevertheless, not issuing a CSPLIT in subframes 1 and 2 is a standards violation.

These all (indirectly) arise from the fact that the designware IP attempts to handle SPLIT/CSPLIT transfers using the same mechanism it uses for other transfers - the drivers schedule a transaction (if periodic), then they queue it on a hardware channel for transmission, then if it is a split, once you get the reply token (NYET, ACK, etc.) you decide whether to immediately reschedule for transmission.

So: now we (think we) know what's wrong:

The designware IP tries to handle case 1 by pushing an SSPLIT into subframe 0. I'm not really sure why this doesn't work.

It can't avoid 2, because the periodic scheduler doesn't know about splits - if a periodic transaction is ready, it'll be scheduled, in whatever order the list (for it is held in a kernel list) happens to be. If that list is somehow jumbled during scheduling - e.g. because you have two periodic transfers and they keep getting scheduled in the opposite order to each other, so a periodic transfer gets accepted half-way through your split - that's what happens.

3 is a simple bug, and probably shouldn't arise given our ostensible policy of only ever scheduling a split in subframe 0. However, easily fixed by allowing a frame's grace - this sacrifices bandwidth, but it's good enough for now.

4 is a pig; the host is simply too slow to run all that code and always get back in time for its CSPLIT.

So: now we think we know what's going wrong -

Problem 2 is solved by adding logic to allow only one CSPLIT at a time. This also, obscurely, solves problem 1 (perhaps some kind of hardware channel tx conflict? hard to tell without knowing what the hardware channels are actually supposed to do).

Problem 3 is easy

Problem 4 is a pig. I've tried to get around it by adding a fast path to the driver which spots a split in progress and immediately reschedules the transfer on the same host channel without going through the scheduling mechanism. However, the interrupt handling in that driver is a mess so I'm not sure if this is stable, and I'm certainly not sure it solves the problem.

All seems promising so far, though more testing is needed with various peripherals and I have done nothing yet with non-periodic splits, which probably suffer from many of the same problems.

As a side-note, I did have a quick look to see if ftrace would help me work out whether someone else was running for long periods with interrupts disabled, but all it did was crash my kernel. Maybe next time ..

So: there you go. A whirlwind tour of how to get seriously confused about high-speed hubs; hope someone finds it useful. Questions, comments, etc. welcome - do let me know if you'd like the patches - they're to a vendor-only version so I shall have to get permission to release them until there's a formal source release for the product.

Muddle version 2.2 now available

2010-12-16T13:43:00.001Z

I have just merged my "new_vcs" branch of muddle into trunk. This is tagged as "2_2", and thus is muddle version 2.2. Changes include:

VCS changes
The version control support has been rewritten. The actual code to write for a new version control system is now smaller, and does not rely on any particular knowledge of muddle. This should make it much easier to add a new version control system, and also simpler to maintain/extend the existing support.

Changed commands: Pull/Update replaced by Fetch/Merge
'muddle pull' and 'muddle update' are now deprecated (partly because no-one could remember what the difference between them was, and partly because different version control systems use those terms very differently). Instead we now have 'muddle fetch' and 'muddle merge'.
'muddle fetch' will pull changes from a remote repository, but will only merge those changes if it can be done with no interaction. 'muddle merge' will go ahead and do any merges, even if interaction is required. See the the help for these commands for more information.

(It is assumed that, in practice, most people will use 'muddle fetch', and that if more complex merging is required, people are likely to use the specific version control system commands to do this. However, it is conceivable that 'muddle merge' may be enough, so it is still, for the moment at least, provided.)

Note that both 'muddle fetch _all' and 'muddle merge _all' now default to processing all checkouts, and then (re)reporting any problems at the end. The old behaviour (of stopping at the first problem) is available with the '-s' or '-stop' switch.

For the moment, 'muddle pull' and 'muddle update' are both still available, but they are just aliases for 'muddle fetch'.

Changed commands: 'package' support in 'checkout' commands
Those muddle commands that expect checkout names can now also be given arguments of the form 'package:' or 'package:{role}' (and so on). Such arguments will be interpreted as "all checkouts required by the named package(s)". This means that the "dep-" and "pkg-" variants of some commands have gone away, as they are no longer needed.

Changed commands: Help
The help infrastructure has been rewritten. The main user-visible change is that help for subcommands is now available per subcommand - for instance:

muddle help query

now reports on what subcommands are available for 'query', and:

muddle help query rules

gives help for the single subcommand. See 'muddle help' for more details on how the help command works.

Changed commands: Query subcommands
Various of the query subcommands now have aliases, and in some cases what is the alias and what is the main command has been changed. See 'muddle help query' for more information.

There are some new queries: 'deployments', 'default-labels', 'default-roles' and 'package-roles'.

Internal changes
Apart from the above, the following:

'Dependable' has become 'Action' throughout (and similarly for all classes with Dependable in their names). All things called "dependable" have also been renamed as "action". For the moment, all the old names have been retained as aliases, in case of other code relying on them.
The checkout tag names in utils.py have been changed, and code using them modified.
The pkg.py::add_checkout_rules() function has been changed to fit the new command names. Things that should be transient have been made so. Various things have been simplified. In particular, 'Push' and 'Commit' no longer depend on 'CheckedOut'. This means that "muddle push" on a checkout that has not yet been checked out will no longer check it out (which sounds like a good idea).
Since I can never remember the difference between utils.Error and utils.Failure (indeed, I tend to guess that they are the other way round than intended), they have been replaced with utils.MuddleBug and utils.GiveUp (although Error and Failure are retained as aliases for the moment, just in case other code is relying on them).
'pylint' and 'pyflakes' have been run over the code, and many changes (mostly minor, but a few clear bugs) have been made.
When starting up, only the specific command that has been requested is instantiated from its class - this may give a small speedup, especially as subcommands are now split out as separate classes.
utils.DirType is now a named tuple, and the output of 'muddle whereami' is slightly prettier.

So: this Android thing...

2010-10-24T23:50:00.007+01:00

Y'know, one day I'm going to post regularly. When that happens, watch out for snowballs from the sun.

Anyway, I've been quietly porting Android 2.2 to one of the dev boards we have lying around recently and it's surprisingly nice; quite apart from all the hype, Froyo is a neat little embedded Linux which requires fairly minimal porting.

Things that particularly caught my eye:

It provides a sensible(ish) RPC system - if you haven't investigated Binder, I suggest it - it's a good complement to kbus.
A libc lighter than the leviathan that is GNU libc - and hacking bionic to reflect a new kernel is easier than you think, though you do have to play guess-the-header-file a bit.
A libstdc++ which is all you should ever want from an embedded Linux
Decent Java and Javascript implementations
A WebKit port
A decent screen manager (seriously: replace com.android.internal.policy.* and watch your windows go where you asked).
The build system is actually quite nice once you've got used to its oddities - likewise the ANT helper tasks. Admittedly, you do need to go hacking in the SDK and NDK before you even vaguely understand how they do what they do..
.. and a sensible and lightweight init.

Oh, and an SDK with eclipse support. There's little not to like - even the kernel patches are pretty sane and easy to apply once you've got your head around what they're doing.

Networking is, slightly unexpectedly, tricky - the device is pretty convinced it's a mobile phone so trying to persuade it that it has ethernet, DHCP and nothing else causes the odd argument. Also, multi-screen support is only semi-there in Froyo - I can see how to do it, but not how to do it nicely. Hopefully Gingerbread will go some way to solving this one.

The launcher seems unhappy about being made to run at 720p, but then it probably wasn't expecting that big of a display, so I'll forgive it.

Resource use seems reasonable - 88Mbyte for my 'doing nothing' Android. A bit high for a stripped embedded Linux, but not high enough to cause me pain. I'm thinking this is a good candidate for my favourite embedded Linux du jour

So: upward and onward.. hope people have fun at ELC this week - I made the decision to come along just after they'd sold out. D'oh.

CELF Embedded Linux Conference Europe

2010-10-22T17:07:00.000+01:00

I shall be at the Embedded Linux Conference in Cambridge next week, on Wednesday and Thursday. On Thursday I shall be sitting at a desk in the Technical Showcase session between 12:35 and 14:30 (hmm, I assume there's some time for lunch in there as well), available to talk about KBUS, muddle and tstools.

I'm assuming my badge will say "Tony Ibbs" rather than "Tibs"...

Muddle FAQ: Efficient minimal rebuilds

2010-10-14T13:17:00.005+01:00

"I've made a change deep within some package. How do I perform an efficient minimal rebuild?"

Coming to muddle from a mostly make-driven world, I really wanted to be able to stand at the top of a tree, type make and have it rebuild only what has changed. Unfortunately, life isn't as simple as that: it only works properly if package dependencies are correctly defined. This sometimes seems to be beyond individual package developers with their own software, let alone complicated inter-package dependencies, so clearly we need a different plan.

The missing link is provided by muddle labels. These are a bit like the build-stamps I'm familiar with from working with make: if a label is asserted, a certain task been completed for a particular component of your tree. For example, a checkout may be checked_out and a package may be configured, built and installed, and as you might imagine these terms imply a natural dependency ordering. Whenever you invoke muddle at the top level it figures out what needs to be done based on the state of those labels.

However, labels are not wired into the build system of any checkout: in other words, you don't get those make-like semantics. This would be difficult to implement in the general case; muddle would have to get down and extremely dirty with the build system of every checkout - and don't forget that they might be built with make, ant, jam or some build system you've never heard of.

Now, labels may be manually retracted and asserted if necessary. Therefore, one might at first think that it would be reasonable to simply retract the built label for the package in question. However, that is not a safe course of action: in the general case a single checkout might be built into multiple packages, so your change might have multiple effects without you realising it. The better plan is to retract the checked_out tag for the checkout you have changed, and then immediately reassert it. Retracting a label has the obvious semantics on labels depending on it, so this would cause all packages depending on that checkout to be rebuilt - what you actually wanted!

Figuring out the label

So, how do you determine the label to retract and reassert? You might have read the documentation and know that it looks like checkout:YOUR-CHECKOUT/checked_out - but what is the checkout actually called?

At the moment you have to invoke muddle query checkout_dirs and look down the list of directories for one which matches your working directory. (Side note: There is no requirement that a checkout label bears any resemblance to the directory path, though by convention they usually do; I imagine that somebody who named their checkout "vaeGhoo3ohRoh3Qu" would find likely themselves on the receiving end of violence - or, at least, extreme sarcasm - from their colleagues.)

So, in this case, having found out that you were working in the frobozz checkout, you would then run:

muddle retract checkout:frobozz/checked_outmuddle assert checkout:frobozz/checked_out

You can then rebuild and redeploy in the usual way.

More generally, if you want to examine the complete list of dependency labels, start with muddle depend user-short.

Pretty pictures

2010-10-14T12:46:00.001+01:00

I keep meaning to write the "introduction to muddle labels" article, and I keep not having time.

I did, however, get round to drawing some diagrams (using OmniGraffle, which is really good). So it is probably worth putting them somewhere that other people can see them.

First off, we have a simple Label:

If you need to know about domains, this becomes:

If we have a simple build where each package is built from a single checkout, and some of the packages are then deployed, then we can represent that as:

This does not show the roles of the packages (checkouts and deployments don't have roles, but for this purpose I'm ignoring that). We can add roles to the diagram as follows:

The association of the parts of a label with the different axes is perhaps more obvious with the final diagram, which is otherwise the same as the one we've just shown:

Of course, those last three images are subtly misleading about how deployments (in particular) actually work, but that's one of the reasons the real article is needed.

"with Directory" makes testing easier

2010-10-14T10:12:00.002+01:00

One of the problems with writing tests for muddle was finding a clean way to write them in Python.
Testing muddle involves a lot of directory creation and moving between directories. Using os.chdir() and friends doesn't lead to easily readable code, and there is always the problem of forgetting to move back out of a particular directory.
My first attempt to make this easier was to write simple pushd() and popd() functions, which maintained a stack of directories. This was a little better, but still didn't solve the "forgetting" problem.
Of course, the solution is obvious -- use with.
Thus we have:

class Directory(object):
    """A class to facilitate pushd/popd behaviour

    It is intended for use with 'with', as in::

        with Directory('~'):
            print 'My home directory contains'
            print ' ',' '.join(os.listdir('.'))
    """
    def __init__(self, where, verbose=True):
        self.start = normalise(os.getcwd())
        self.where = normalise(where)
        self.verbose = verbose
        os.chdir(self.where)
        if verbose:
            print '++ pushd to %s'%self.where

    def close(self):
        os.chdir(self.start)
        if self.verbose:
            print '++ popd to %s'%self.start

    def __enter__(self):
        return self

    def __exit__(self, etype, value, tb):
        if tb is None:
            # No exception, so just finish normally
            self.close()
        else:
            # An exception occurred, so do any tidying up necessary
            if self.verbose:
                print '** Oops, an exception occurred - %s tidying up'%self.__class__.__name__
            # well, there isn't anything special to do, really
            self.close()
            if self.verbose:
                print '** ----------------------------------------------------------------------'
            # And allow the exception to be re-raised
            return False

and its obvious friends:

class NewDirectory(Directory):
    """A pushd/popd directory that gets created first.

    It is an Error if the directory already exists.
    """
    def __init__(self, where, verbose=True):
        where = normalise(where)
        if os.path.exists(where):
            raise Error('Directory %s already exists'%where)
        if verbose:
            print '++ mkdir %s'%where
        os.makedirs(where)
        super(NewDirectory, self).__init__(where, verbose)

class TransientDirectory(NewDirectory):
    """A pushd/popd directory that gets created first and deleted afterwards

    If 'keep_on_error' is True, then the directory will not be deleted
    if an exception occurs in its 'with' clause.

    It is an Error if the directory already exists.
    """
    def __init__(self, where, keep_on_error=False, verbose=True):
        self.rmtree_on_error = not keep_on_error
        super(TransientDirectory, self).__init__(where, verbose)

    def close(self, delete_tree):
        super(NewDirectory, self).close()
        if delete_tree:
            if self.verbose:
                # The extra space after 'rmtree' is so the directory name
                # left aligns with a previous 'popd to' message
                print '++ rmtree  %s'%self.where
            shutil.rmtree(self.where)

    def __exit__(self, etype, value, tb):
        if tb is None:
            # No exception, so just finish normally
            self.close(True)
        else:
            # An exception occurred, so do any tidying up necessary
            if self.verbose:
                print '** Oops, an exception occurred - %s tidying up'%self.__class__.__name__
            # but don't delete the tree if we've been asked not to
            self.close(self.rmtree_on_error)
            if self.verbose:
                print '** ----------------------------------------------------------------------'
            # And allow the exception to be re-raised
            return False

and I can now write code like:

root_repo = 'file://' + os.path.join(root_dir, 'repo')
with NewDirectory('test_build1'):
    banner('Bootstrapping checkout build')
    muddle(['bootstrap', 'git+%s'%root_repo, 'test_build'])
    cat('src/builds/01.py')

    banner('Setting up src/')
    with Directory('src'):
        with Directory('builds'):
            touch('01.py', CHECKOUT_BUILD)
            git('add 01.py')
            git('commit -m "New build"')
            git('push %s/builds HEAD'%root_repo)

        with NewDirectory('checkout1'):
            touch('Makefile.muddle', MUDDLE_MAKEFILE)
            git('init')
            git('add Makefile.muddle')
            git('commit -m "Add muddle makefile"')
            git('push %s/checkout1 HEAD'%root_repo)
            muddle(['assert', 'checkout:checkout1/checked_out'])

which seems to fit with how I want to think about the tests.

New verson of muddle

2010-10-13T21:12:00.001+01:00

Today I have merged my development branch of muddle back into the trunk. This is the same version of muddle that I was using as "m3" in previous posts. There are a few new commands, but on the whole this is a change for maintainability, and progress towards muddle3, rather than user features.

See the main muddle page (http://code.google.com/p/muddle/) for information on new tags and branches.

muddle and its directories - part 7 - end

2010-09-29T20:47:00.001+01:00

If you've read all of that, and it made some degree of sense, then I really do recommend you also go and read "Welcome to Muddle", online at:

http://muddle.googlecode.com/svn/trunk/muddle/docs/_build/html/README.html

If you've read it before, it should now make a different sort of sense, and if you haven't, you really should. Really.

It also mentions several topics that have not been addressed in these posts.

muddle's help is meant to be useful (muddle help), although it sometimes lags behind reality. If you find problems or inaccuracies in the help, please do raise an issue on the muddle issues page describing the shortcomings - these do get attended to.

Also, I am contactable by email, and will try to help with specific problems.

Finally, if all else fails, we will develop muddle for money!

muddle and its directories - part 6 - domains

2010-09-29T20:46:00.003+01:00

Contents

Domains

Domains

Domains are part of advanced muddle use, and are probably not relevant to most builds. Regardless, it is probably worth reading at least What domains are for from this section, as useful background, and so that you can recognise when they might be applicable.

The way domains work follows naturally from the rest of muddle, but the implementation is still being tidied up - hence the muddle3_label branch.

What domains are for

Domains allow one to include one build description inside another, in much the same way as a Python module can import another.

A simple example might be when building a system with WebKit and X11 support. We might have two builds, constructed as follows:

def UI_Build:
   import WebKit_build
       import X11_build

and:

def Login_build:
    import X11_build

Alternatively, we might have some high-level software for handling internet television, and want to build two systems on different architectures. In this case, our contrasting builds might be:

def IPTV_stack:
    import ARM_based_stack

and:

def IPTV_stack:
    import MIPS_based_stack

How domains work

A build includes another build (a subdomain) using the include_domain() function. This takes the same arguments as the muddle init command (that is, a repository specification and a build description therein), plus a name for the domain.

The following sequence of actions is then performed:

Create a directory called domains/<name>/, where <name> is the name of the new domain.
cd into that new directory, and perform (the equivalent of) a muddle init command therein, using the repository specification and build description given.
Read in this new build, giving a stand-alone build tree datastructure.
Find all of the labels in the new build tree datastructures, and change them to add the domain name. So, for instance, checkout:fred/checked_out would become checkout:(<name>)fred/checked_out.
Incorporate this amended build tree datastructure into the original ("top level") build.
Create a .muddle/am_subdomain file in the domain. This allows muddle to distinguish the actual (final) top-level build from any subdomains within it, which is a useful optimisation.

Note that the build description for a domain does not itself know that it is not the top-level of a muddle build. Indeed, this is an important property of domains - it means that any build description can potentially be included in any other.

For most purposes, the subdomain works just as if it were a "normal" top-level build. In particular, it uses its own .muddle/ directory to record its build state, and writes things to its own obj/ and install/ directories.

Also note that, as a consequence of the way domains work (and this is a good thing), it is only possible for a build to refer to labels in subdomains, not to those in sibling or parent build trees.

To allow for more flexibility in how domains are used, there are tools in the muddled package for manipulating the merged label-space. For instance, if one is using subdomains that originated as two different set-top-box builds, both might provide a linux kernel, and the top-level build probably wants to amend the labels such that both subdomains use one of the kernel packages (broadly, saying "this label should be used instead of that label").

A worked example of using domains

So, we shall start in a new empty directory:

$ cd ..
$ mkdir example2
$ cd example2

and start with the same build that we used before:

$ m3 init svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio builds/01.py
> Make directory /home/tibs/sw/m3/example2/.muddle
Initialised build tree in /home/tibs/sw/m3/example2
Repository: svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
Build description: builds/01.py


Checking out build description ..

> Make directory /home/tibs/sw/m3/example2/src
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/builds builds
A    builds/01.py
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example2/.muddle/tags/checkout/builds
Done.

So, just as before, we now have the following directory structure:

.
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       `-- checkout/
|           `-- builds/
|               `-- checked_out
`-- src/
    `-- builds/
        |-- 01.py
        `-- 01.pyc

We can now edit the src/builds/01.py file to include a subdomain as part of the build. As described above, we need to specify how to retrieve the domain, giving the same information as for the "muddle init" command. We shall call our domain b (after the example we're taking it from):

from muddled.mechanics import include_domain
include_domain(builder,
               domain_name = "b",
               domain_repo = "svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b",
               domain_desc = "builds/01.py")

We also have to say how to include the result into our deployment - a simple way to do this is:

import muddled.deployments.collect as collect
collect.deploy(builder, "everything")
collect.copy_from_role_install(builder, "everything",
                               role = "x86",
                               rel = "", dest = "",
                               domain = None)
collect.copy_from_role_install(builder, "everything",
                               role = "x86",
                               rel = "", dest = "usr",
                               domain = "b")

and then deploy the new deployment "everything", instead of the old "cpio_dep". Note that we're not deploying to a CPIO archive this time, but just as normal files.

We should probably also change the introductory comment to:

# An example of building with a subdomain

So the "top level" build description is now:

#!  /usr/bin/env python
#
# An example of building with a subdomain

import muddled
import muddled.pkgs.make
import muddled.deployments.cpio
import muddled.checkouts.simple
import muddled.deployments.collect as collect
from muddled.mechanics import include_domain

def describe_to(builder):
    # Checkout ..
    muddled.checkouts.simple.relative(builder, "cpio_co")
    muddled.pkgs.make.simple(builder, "pkg_cpio", "x86", "cpio_co")

    include_domain(builder,
                   domain_name = "b",
                   domain_repo = "svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b",
                   domain_desc = "builds/01.py")

    collect.deploy(builder, "everything")
    collect.copy_from_role_install(builder, "everything",
                                   role = "x86",
                                   rel = "", dest = "",
                                   domain = None)
    collect.copy_from_role_install(builder, "everything",
                                   role = "x86",
                                   rel = "", dest = "usr",
                                   domain = "b")

    builder.invocation.add_default_role("x86")
    builder.by_default_deploy("everything")

# End file.

If we now do a "checkout _all":

$ m3 checkout _all
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle
Initialised build tree in /home/tibs/sw/m3/example3/domains/b
Repository: svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b
Build description: builds/01.py


Checking out build description ..

> Make directory /home/tibs/sw/m3/example3/domains/b/src
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/b/builds builds
A    builds/01.py
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle/tags/checkout/builds
There is no rule to build label checkout:b_co/checked_out
> Building checkout:cpio_co/checked_out
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/cpio_co cpio_co
A    cpio_co/hello_world.c
A    cpio_co/Makefile
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/checkout/cpio_co

giving us:

$ ls -AF
domains/  .muddle/  src/

or, in more detail (omitting .svn directories):

.
|-- domains/
|   `-- b/
|       |-- .muddle/
|       |   |-- am_subdomain
|       |   |-- Description
|       |   |-- RootRepository
|       |   `-- tags/
|       |       `-- checkout/
|       |           `-- builds/
|       |               `-- checked_out
|       `-- src/
|           `-- builds/
|               |-- 01.py
|               `-- 01.pyc
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       `-- checkout/
|           |-- builds/
|           |   `-- checked_out
|           `-- cpio_co/
|               `-- checked_out
`-- src/
    |-- builds/
    |   |-- 01.py
    |   `-- 01.pyc
    `-- cpio_co/
        |-- hello_world.c
        `-- Makefile

Apart from the am_subdomain file in its .muddle/ directory, the domains/b/ directory looks like a perfectly normal build.

If we then do:

$ m3 build _all
Building package:(b)pkg_b{x86}/postinstalled package:pkg_cpio{x86}/postinstalled
> Building checkout:(b)b_co/checked_out
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/b/b_co b_co
A    b_co/hello_world.c
A    b_co/Makefile
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle/tags/checkout/b_co
> Building package:(b)pkg_b{x86}/preconfig
> Make directory /home/tibs/sw/m3/example3/domains/b/obj/pkg_b/x86
> Make directory /home/tibs/sw/m3/example3/domains/b/install/x86
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle/tags/package/pkg_b
> Building package:(b)pkg_b{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:(b)pkg_b{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example3/domains/b/obj/pkg_b/x86/hello_world hello_world.c
> Building package:(b)pkg_b{x86}/installed
> make  -f Makefile install
install -m 0755 /home/tibs/sw/m3/example3/domains/b/obj/pkg_b/x86/hello_world /home/tibs/sw/m3/example3/domains/b/install/x86/hello_world
> Building package:(b)pkg_b{x86}/postinstalled
> Building package:pkg_cpio{x86}/preconfig
> Make directory /home/tibs/sw/m3/example3/obj/pkg_cpio/x86
> Make directory /home/tibs/sw/m3/example3/install/x86
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/package/pkg_cpio
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example3/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example3/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example3/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example3/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example3/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example3/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

we end up with the subdomain built in its directory structure:

domains/
`-- b/
    |-- install/
    |   `-- x86/
    |       `-- hello_world*
    |-- .muddle/
    |   |-- am_subdomain
    |   |-- Description
    |   |-- RootRepository
    |   `-- tags/
    |       |-- checkout/
    |       |   |-- b_co/
    |       |   |   `-- checked_out
    |       |   `-- builds/
    |       |       `-- checked_out
    |       `-- package/
    |           `-- pkg_b/
    |               |-- x86-built
    |               |-- x86-configured
    |               |-- x86-installed
    |               |-- x86-postinstalled
    |               `-- x86-preconfig
    |-- obj/
    |   `-- pkg_b/
    |       `-- x86/
    |           `-- hello_world*
    `-- src/
        |-- b_co/
        |   |-- hello_world.c
        |   `-- Makefile
        `-- builds/
            |-- 01.py
            `-- 01.pyc

and the top-level build in its:

`-- install/
|   `-- x86/
|       |-- bin/
|       |   `-- hello_world*
|       `-- hello_world.c
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       |-- checkout/
|       |   |-- builds/
|       |   |   `-- checked_out
|       |   `-- cpio_co/
|       |       `-- checked_out
|       `-- package/
|           `-- pkg_cpio/
|               |-- x86-built
|               |-- x86-configured
|               |-- x86-installed
|               |-- x86-postinstalled
|               `-- x86-preconfig
|-- obj/
|   `-- pkg_cpio/
|       `-- x86/
|           `-- hello_world*
`-- src/
    |-- builds/
    |   |-- 01.py
    |   `-- 01.pyc
    `-- cpio_co/
        |-- hello_world.c
        `-- Makefile

If we ask after packages, we have both sets:

$ m3 query packages
pkg_b
pkg_cpio

(this should arguably report the domain of each package name as well - there should probably be an option to select this).

If we ask what domains we have, we get:

$ m3 query domains

b

(the current printout of domains includes the "empty" or top-level domain in its listing, which is slightly unobvious, and will probably be fixed at some time).

Our dependency rules are now extended to include those from the subdomain as well:

$ m3 depend user-short
-----
checkout:builds/changes_committed <-VcsCheckoutBuilder-- [ checkout:builds/up_to_date[T] ]
checkout:builds/changes_pushed <-VcsCheckoutBuilder-- [ checkout:builds/changes_committed ]
checkout:builds/pulled <-VcsCheckoutBuilder-- [ checkout:builds/checked_out, checkout:builds/up_to_date[T] ]
checkout:builds/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:builds/checked_out ]
checkout:cpio_co/changes_committed <-VcsCheckoutBuilder-- [ checkout:cpio_co/up_to_date[T] ]
checkout:cpio_co/changes_pushed <-VcsCheckoutBuilder-- [ checkout:cpio_co/changes_committed ]
checkout:cpio_co/pulled <-VcsCheckoutBuilder-- [ checkout:cpio_co/checked_out, checkout:cpio_co/up_to_date[T] ]
checkout:cpio_co/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:cpio_co/checked_out ]
checkout:(b)b_co/changes_committed <-VcsCheckoutBuilder-- [ checkout:(b)b_co/up_to_date[T] ]
checkout:(b)b_co/changes_pushed <-VcsCheckoutBuilder-- [ checkout:(b)b_co/changes_committed ]
checkout:(b)b_co/pulled <-VcsCheckoutBuilder-- [ checkout:(b)b_co/checked_out, checkout:(b)b_co/up_to_date[T] ]
checkout:(b)b_co/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:(b)b_co/checked_out ]
checkout:(b)builds/changes_committed <-VcsCheckoutBuilder-- [ checkout:(b)builds/up_to_date[T] ]
checkout:(b)builds/changes_pushed <-VcsCheckoutBuilder-- [ checkout:(b)builds/changes_committed ]
checkout:(b)builds/pulled <-VcsCheckoutBuilder-- [ checkout:(b)builds/checked_out, checkout:(b)builds/up_to_date[T] ]
checkout:(b)builds/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:(b)builds/checked_out ]
deployment:everything/deployed <-CollectDeploymentBuilder-- [ package:*{x86}/postinstalled, package:(b)*{x86}/postinstalled ]
package:pkg_cpio{x86}/built <-MakeBuilder-- [ package:pkg_cpio{x86}/configured ]
package:pkg_cpio{x86}/configured <-MakeBuilder-- [ package:pkg_cpio{x86}/preconfig ]
package:pkg_cpio{x86}/installed <-MakeBuilder-- [ package:pkg_cpio{x86}/built ]
package:pkg_cpio{x86}/postinstalled <-MakeBuilder-- [ package:pkg_cpio{x86}/installed ]
package:pkg_cpio{x86}/preconfig <-MakeBuilder-- [ checkout:cpio_co/checked_out ]
package:(b)pkg_b{x86}/built <-MakeBuilder-- [ package:(b)pkg_b{x86}/configured ]
package:(b)pkg_b{x86}/configured <-MakeBuilder-- [ package:(b)pkg_b{x86}/preconfig ]
package:(b)pkg_b{x86}/installed <-MakeBuilder-- [ package:(b)pkg_b{x86}/built ]
package:(b)pkg_b{x86}/postinstalled <-MakeBuilder-- [ package:(b)pkg_b{x86}/installed ]
package:(b)pkg_b{x86}/preconfig <-MakeBuilder-- [ checkout:(b)b_co/checked_out ]
-----

and our deployment has the expected build order:

$ m3 query deps deployment:everything/deployed
Build order for deployment:everything/deployed ..
checkout:cpio_co/checked_out
checkout:(b)b_co/checked_out
package:(b)pkg_b{x86}/preconfig
package:pkg_cpio{x86}/preconfig
package:(b)pkg_b{x86}/configured
package:pkg_cpio{x86}/configured
package:(b)pkg_b{x86}/built
package:pkg_cpio{x86}/built
package:(b)pkg_b{x86}/installed
package:pkg_cpio{x86}/installed
package:(b)pkg_b{x86}/postinstalled
package:pkg_cpio{x86}/postinstalled
deployment:everything/deployed

Which means that when we deploy:

$ m3 deploy
Building deployment:everything/deployed
> Building deployment:everything/deployed
> Make directory /home/tibs/sw/m3/example3/deploy/everything
Copying /home/tibs/sw/m3/example3/install/x86/ to /home/tibs/sw/m3/example3/deploy/everything/
Copying /home/tibs/sw/m3/example3/domains/b/install/x86/ to /home/tibs/sw/m3/example3/deploy/everything/usr
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/deployment/everything

we end up with everything deployed in the deploy/ directory at the top-level:

deploy/
`-- everything/
    |-- bin/
    |   `-- hello_world*
    |-- hello_world.c
    `-- usr/
        `-- hello_world*

bin/hello_world comes from the top-level build, and usr/hello_world from the subdomain. The hello_world.c is also from the top-level build (the example Makefile for the cpio package copies the source file into the install/ directory).

Version stamps and domains

Version stamps work with domains as well, although one may only generate them for the top-level in a particular build tree.

Thus we can try:

$ m3 stamp version
Finding all checkouts... found 4
Processing Svn checkout 'builds'
builds: 'svnversion' reports checkout has revision '459M'
Processing Svn checkout 'cpio_co'
Processing Svn checkout '(b)b_co'
Processing Svn checkout '(b)builds'
Found domains: set([DomainTuple(name='b', repository='svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b', description='builds/01.py')])

Unable to work out revision ids for all the checkouts
- although we did work out 3 of 4
Problems were:
* builds: 'svnversion' reports checkout has revision '459M'
Problems prevent writing version stamp file

OK, that makes sense, since we had indeed edited that checkout, and had not commited the changes. Luckily, it is sufficient for our purposes to save the state without that edit:

$ m3 stamp save -force
Forcing original revision ids when necessary
Finding all checkouts... found 4
Processing Svn checkout 'builds'
builds: 'svnversion' reports checkout has revision '459M'
Processing Svn checkout 'cpio_co'
Processing Svn checkout '(b)b_co'
Processing Svn checkout '(b)builds'
Found domains: set([DomainTuple(name='b', repository='svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b', description='builds/01.py')])

Unable to work out revision ids for all the checkouts
- although we did work out 3 of 4
Problems were:
* builds: 'svnversion' reports checkout has revision '459M'
Writing to working.stamp
Wrote revision data to working.stamp
File has SHA1 hash 6425284f002c81c9849f2f23add025a710207509
Renaming working.stamp to 6425284f002c81c9849f2f23add025a710207509.partial

and that at least shows us that the stamp file does record details of any subdomains:

[ROOT]
description = builds/01.py
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio

[DOMAIN b]
description = builds/01.py
name = b
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b

[CHECKOUT (b)b_co]
domain = b
name = b_co
relative = b_co
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b
revision = 459

[CHECKOUT (b)builds]
domain = b
name = builds
relative = builds
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b
revision = 459

[CHECKOUT cpio_co]
name = cpio_co
relative = cpio_co
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
revision = 459

[PROBLEMS]
problem1 = builds: 'svnversion' reports checkout has revision '459M'

muddle and its directories - part 5 - adding and stamps

2010-09-29T20:46:00.001+01:00

Contents

Adding a new checkout/package
Version stamps

Adding a new checkout/package

Let's suppose we want to add a new package, based on a new checkout. There is more than one way to do this, but the following shows a sensible approach.

We shall create our new package by just copying one we've already got:

$ cp -a src/cpio_co src/fred

We must remember to remove the outdated repository information:

$ rm -rf src/fred/,svn/

And make this checkout do something different:

$ mv src/fred/hello_world.c src/fred/bye_world.c
$ sed -e s/hello_world/bye_world/g --in-place src/fred/Makefile
$ sed -e s/Hello/Byebye/g --in-place src/fred/bye_world.c

muddle has no knowledge of this checkout yet, so we need to add it to the build description. We can simply edit src/builds/01.py to add the lines:

muddled.checkouts.simple.relative(builder, "fred")

and:

muddled.pkgs.make.simple(builder, "fred", "x86", "fred")

(yes, that leaves us with a package and checkout with the same name, but that's not a problem, and is closer to the more conventional usage).

That's enough to give us:

$ m3 query checkouts
builds
cpio_co
fred

and:

$ m3 query deps deployment:cpio_dep/deployed
Build order for deployment:cpio_dep/deployed ..
checkout:fred/checked_out
checkout:cpio_co/checked_out
package:pkg_cpio{x86}/preconfig
package:fred{x86}/preconfig
package:pkg_cpio{x86}/configured
package:fred{x86}/configured
package:fred{x86}/built
package:pkg_cpio{x86}/built
package:fred{x86}/installed
package:pkg_cpio{x86}/installed
package:pkg_cpio{x86}/postinstalled
deployment:cpio_dep/deployed

However, if we were to try to build:

$ m3
Building deployment:cpio_dep/deployed
> Building checkout:fred/checked_out
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/fred fred
svn: URL 'http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/fred' doesn't exist
Can't build deployment:cpio_dep/deployed - Command 'svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/fred fred' execution failed - 1

This makes sense, because (a) we have not told muddle that the new checkout is present in its directory structure, and (b) we have not actually checked it in to the subversion repository it wants to look in.

Since we've not yet finished developing this new package, we don't want to check it in to the repository yet (and, in particular in our case, we do not want to permanently add it to our example).

So we need to make muddle believe that it has been checked out.

As you might expect, there are two ways to do this. The one that you may guess is to manipulate the .muddle/` directory structure directly - so we could just do:

$ mkdir .muddle/tags/checkout/fred
$ touch .muddle/tags/checkout/fred/checked_out

However, that's a bit clumsy to type, so muddle provides a convenient command for asserting a particular tag (just assuming you are comfortable with the label syntax):

$ m3 assert checkout:fred/checked_out
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/checkout/fred

and now the tag file is present:

$ ls .muddle/tags/checkout/fred
checked_out

and we can then do:

$ m3
Building deployment:cpio_dep/deployed
> Building package:fred{x86}/preconfig
> Make directory /home/tibs/sw/m3/example3/obj/fred/x86
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/package/fred
> Building package:fred{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:fred{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example3/obj/fred/x86/bye_world bye_world.c
> Building package:fred{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example3/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example3/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example3/obj/fred/x86/bye_world /home/tibs/sw/m3/example3/install/x86/bin/bye_world
install -m 0644 bye_world.c /home/tibs/sw/m3/example3/install/x86/bye_world.c
> Building deployment:cpio_dep/deployed
> Make directory /home/tibs/sw/m3/example3/deploy/cpio_dep
Collecting package:*{x86}/*  for deployment to / ..
h = ---Roots---
/ -> [ / (fs /home/tibs/sw/m3/example3/install/x86) mode = 40755 uid = 7007 gid = 7007 kids = /bin /hello_world.c /bye_world.c]
---Map---
/bin/bye_world -> [ /bin/bye_world (fs /home/tibs/sw/m3/example3/install/x86/bin/bye_world) mode = 100755 uid = 7007 gid = 7007 kids = ]
/bin -> [ /bin (fs /home/tibs/sw/m3/example3/install/x86/bin) mode = 40755 uid = 7007 gid = 7007 kids = /bin/bye_world /bin/hello_world]
/hello_world.c -> [ /hello_world.c (fs /home/tibs/sw/m3/example3/install/x86/hello_world.c) mode = 100644 uid = 7007 gid = 7007 kids = ]
/bin/hello_world -> [ /bin/hello_world (fs /home/tibs/sw/m3/example3/install/x86/bin/hello_world) mode = 100755 uid = 7007 gid = 7007 kids = ]
/bye_world.c -> [ /bye_world.c (fs /home/tibs/sw/m3/example3/install/x86/bye_world.c) mode = 100644 uid = 7007 gid = 7007 kids = ]
/ -> [ / (fs /home/tibs/sw/m3/example3/install/x86) mode = 40755 uid = 7007 gid = 7007 kids = /bin /hello_world.c /bye_world.c]
---

base = /
Scanning instructions for role x86, domain None ..
> Writing /home/tibs/sw/m3/example3/deploy/cpio_dep/my_archive.cpio ..
> Packing / ..
> Packing /bin ..
> Packing /bin/bye_world ..
> Packing /bin/hello_world ..
> Packing /hello_world.c ..
> Packing /bye_world.c ..
> Packing TRAILER!!! ..
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/deployment/cpio_dep

This just leaves actually adding the new package to revision control somewhere - in this case, the build description is saying (implicitly) that this package is stored in the same subversion repository as the rest of the example, and so one would need to add it there by the normal means (which is a topic for another time).

(The fact that this is much easier to do for distributed revision control systems like bazaar, mercurial or git is in itself a good reason for using them!)

Version stamps

Version stamps allow one to produce a simple text file (actually, an INI file) representing the current state of a build. This can be useful for various purposes, but its main intent is to allow saving a build state so that it can be accurately recreated at a later date (for instance, so one can rebuild an earlier release to customers).

The muddle stamp and unstamp commands have documentation via muddle help stamp and help unstamp.

However, it is worth giving a quick example of creating a version stamp:

$ m3 stamp version
Finding all checkouts... found 2
Processing Svn checkout 'builds'
Processing Svn checkout 'cpio_co'
Creating directory /home/tibs/sw/m3/example/versions
Writing to /home/tibs/sw/m3/example/versions/_temporary.stamp
Wrote revision data to /home/tibs/sw/m3/example/versions/_temporary.stamp
File has SHA1 hash 189085d413217cb1865868b5d9916085d22e6f50
Renaming /home/tibs/sw/m3/example/versions/_temporary.stamp to /home/tibs/sw/m3/example/versions/01.stamp

As indicated above, we now have a new versions/ directory:

$ ls -AFt
versions/  deploy/  install/  obj/  src/  .muddle/

containing the stamp file:

versions
`-- 01.stamp

Modern build descriptions are encouraged to specify a build name (this is as simple as adding a line of the form:

builder.build_name = 'ExampleBuild'

to the Python code). If this is given, then the version stamp file will use that build name as its filename, otherwise it defaults to the filename of the main build description file (01 in this case).

The versions/ directory is suitable for putting into a version control system, as one might expect, and future versions of muddle are likely to provide more support for handling this (see issue 117 for some ideas on this).

The contents of a stamp file is deliberately human readable:

[ROOT]
description = builds/01.py
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio

[CHECKOUT builds]
name = builds
relative = builds
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
revision = 458

[CHECKOUT cpio_co]
name = cpio_co
relative = cpio_co
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
revision = 458

As normal. muddle will not stop you writing or editing stamp files. This may conceivably be useful.

muddle and its directories - part 4 - building

2010-09-29T20:45:00.001+01:00

Contents

Building a package
Deployment
Rebuilding things
Do what I say, not what I do

Building a package

Whilst it is possible for a package to be built from more than one checkout, this is relatively uncommon, and it is quite usual for a package to correspond directly to a checkout. As it does in this case.

If one tries to build a package that doesn't exist, one will be told so:

$ m3 build cpio
Building package:cpio{x86}/postinstalled
There is no rule to build label package:cpio{x86}/postinstalled

It's simple to find out what packages there are:

$ m3 query packages
pkg_cpio

So let's build it:

$ m3 build pkg_cpio
Building package:pkg_cpio{x86}/postinstalled
> Building package:pkg_cpio{x86}/preconfig
> Make directory /home/tibs/sw/m3/example/obj/pkg_cpio/x86
> Make directory /home/tibs/sw/m3/example/install/x86
> Make directory /home/tibs/sw/m3/example/.muddle/tags/package/pkg_cpio
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

We now have two new directories:

$ ls -AFt
install/  obj/  src/  .muddle/

The src/ directory has not changed.

The .muddle/ directory has gained some new tags in .muddle/tags/package/pkg_cpio/, reflecting the new state of the build:

.muddle/
|-- Description
|-- RootRepository
`-- tags/
    |-- checkout/
    |   |-- builds/
    |   |   `-- checked_out
    |   `-- cpio_co/
    |       `-- checked_out
    `-- package/
        `-- pkg_cpio/
            |-- x86-built
            |-- x86-configured
            |-- x86-installed
            |-- x86-postinstalled
            `-- x86-preconfig

The obj/ directory contains the results of building packages. Thus we have a new directory called obj/pkg_cpio/x86.

obj/
`-- pkg_cpio/
    `-- x86/
        `-- hello_world*

Specifically, within obj/ there is a directory named after each package, within which there is a directory named after each role for that package. For convenience, the makefile can use the environment variable $(MUDDLE_OBJ) to refer to <root_dir>/obj/pkg_cpio/x86 (<root_dir> indicates the absolute path to the top-level build directory).

There would normally be more use of subdirectories such as $(MUDDLE_OBJ)/bin in a "real" build, and muddle provides some other environment variables to help with these.

The install/ contains the results of installing packages. Each {role} gets a separate directory in install/:

install/
`-- x86/
    |-- bin/
    |   `-- hello_world*
    `-- hello_world.c

Again, the makefile can use the environment variable $(MUDDLE_INSTALL) to refer to this role-specific directory.

As an aside, using the MUDDLE environment variables in this way means that if two packages (e.g., package:a{b}/* versus package:c{d}/*, or even package:a{b}/* versus package:a{d}/*) both use the same checkout, and thus the same makefile, the results of building the two packages will still end up in different, predictable directories.

It is possible to find out the entire environment muddle passes to a makefile with the makeenv query:

$ m3 query makeenv package:pkg_cpio{x86}/built
MUDDLE=/home/tibs/sw/m3/muddle3_labels/muddle/muddled/__main__.py
MUDDLE_INCLUDE_DIRS=
MUDDLE_INSTALL=/home/tibs/sw/m3/example/install/x86
MUDDLE_INSTRUCT=/home/tibs/sw/m3/muddle3_labels/muddle/muddled/__main__.py instruct pkg_cpio{x86}
MUDDLE_KIND=package
MUDDLE_LABEL=package:pkg_cpio{x86}/built
MUDDLE_LD_LIBRARY_PATH=
MUDDLE_LIB_DIRS=
MUDDLE_NAME=pkg_cpio
MUDDLE_OBJ=/home/tibs/sw/m3/example/obj/pkg_cpio/x86
MUDDLE_OBJ_INCLUDE=/home/tibs/sw/m3/example/obj/pkg_cpio/x86/include
MUDDLE_OBJ_LIB=/home/tibs/sw/m3/example/obj/pkg_cpio/x86/lib
MUDDLE_OBJ_OBJ=/home/tibs/sw/m3/example/obj/pkg_cpio/x86/obj
MUDDLE_PKGCONFIG_DIRS=
MUDDLE_PKGCONFIG_DIRS_AS_PATH=
MUDDLE_ROLE=x86
MUDDLE_ROOT=/home/tibs/sw/m3/example
MUDDLE_SRC=/home/tibs/sw/m3/example/src/cpio_co
MUDDLE_TAG=built
MUDDLE_TARGET_LOCATION=/
MUDDLE_UNINSTRUCT=/home/tibs/sw/m3/muddle3_labels/muddle/muddled/__main__.py instruct pkg_cpio{x86}

Builds specifying cross-compilation toolchains and other options may have much longer environments passed down.

Deployment

Deployment is the process of copying the files for the various roles from the install/ directories to the deploy/ directories.

Deployment is commonly the stage that prepares the "blob" that will be put onto the final device (if one is building an embedded device), and perhaps also aggregates the tools for doing so.

In this build, we are producing a CPIO archive.

(CPIO is a very old Unix archive format. For our purposes ("us" being muddle), its advantage is that it is possible to flag files within the archive with particular permission bits, to request creationg of device nodes, and other thing that would require superuser privileges if done "live" in the build tree.)

In this build we only have a single deployment target, so we don't need to specify its name.

$ m3 deploy
Building deployment:cpio_dep/deployed
> Building deployment:cpio_dep/deployed
> Make directory /home/tibs/sw/m3/example/deploy/cpio_dep
Collecting package:*{x86}/*  for deployment to / ..
h = ---Roots---
/ -> [ / (fs /home/tibs/sw/m3/example/install/x86) mode = 40755 uid = 7007 gid = 7007 kids = /bin /hello_world.c]
---Map---
/bin/hello_world -> [ /bin/hello_world (fs /home/tibs/sw/m3/example/install/x86/bin/hello_world) mode = 100755 uid = 7007 gid = 7007 kids = ]
/ -> [ / (fs /home/tibs/sw/m3/example/install/x86) mode = 40755 uid = 7007 gid = 7007 kids = /bin /hello_world.c]
/bin -> [ /bin (fs /home/tibs/sw/m3/example/install/x86/bin) mode = 40755 uid = 7007 gid = 7007 kids = /bin/hello_world]
/hello_world.c -> [ /hello_world.c (fs /home/tibs/sw/m3/example/install/x86/hello_world.c) mode = 100644 uid = 7007 gid = 7007 kids = ]
---

base = /
Scanning instructions for role x86, domain None ..
> Writing /home/tibs/sw/m3/example/deploy/cpio_dep/my_archive.cpio ..
> Packing / ..
> Packing /bin ..
> Packing /bin/hello_world ..
> Packing /hello_world.c ..
> Packing TRAILER!!! ..
> Make directory /home/tibs/sw/m3/example/.muddle/tags/deployment/cpio_dep

That leaves us with a new deploy/ directory:

$ ls -AFt
deploy/  install/  obj/  src/  .muddle/

containing our CPIO archive:

deploy/
`-- cpio_dep/
    `-- my_archive.cpio

and the appropriate tag has been set in the .muddle/ directory tree:

.muddle/tags/deployment/
`-- cpio_dep/
    `-- deployed

Rebuilding things

muddle provides a convenience command to rebuild a package:

$ m3 rebuild pkg_cpio
Killing package:pkg_cpio{x86}/built
Clearing tags: package:pkg_cpio{x86}/built package:pkg_cpio{x86}/installed package:pkg_cpio{x86}/postinstalled package:pkg_cpio{x86}/built
Building package:pkg_cpio{x86}/postinstalled
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

As you can see, this "kills" (deletes) the tags saying that the package has been built, installed and postinstalled, and then rebuilds the /postinstalled tag. This does not, however, do anything about things that depend on this package.

It is frequently more useful to use the distrebuild command, which is a conflation of the distclean and build commands:

$ m3 distrebuild pkg_cpio
Building: package:pkg_cpio{x86}/distclean ..
> Building package:pkg_cpio{x86}/distclean[T]
> make  -f Makefile distclean
rm -f /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world
Distclean is just a clean
Killing: package:pkg_cpio{x86}/preconfig ..
Clearing tags: package:pkg_cpio{x86}/preconfig package:pkg_cpio{x86}/configured package:pkg_cpio{x86}/preconfig package:pkg_cpio{x86}/installed package:pkg_cpio{x86}/postinstalled package:pkg_cpio{x86}/built
Building package:pkg_cpio{x86}/postinstalled
> Building package:pkg_cpio{x86}/preconfig
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

The distclean also removes the tags for any packages that depended upon this package (not very obvious in this case where we only had one, of course).

(Beware - it unsets the state of other packages, but not the deployment.)

You can also, quite legally, do the same thing by direct manipulation of the tag files in the .muddle/ directories:

$ rm .muddle/tags/package/pkg_cpio/*
$ m3 build pkg_cpio
Building package:pkg_cpio{x86}/postinstalled
> Building package:pkg_cpio{x86}/preconfig
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

That is, muddle doesn't care if you remove the tags directly, instead of via the command line tool.

You can even do things like:

$ rm .muddle/tags/checkout/builds/checked_out
$ m3 checkout _all
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/builds builds
Checked out revision 459.

This is quite deliberate. Whilst the muddle command line tool provides a variery of useful shorthand commands (such as "distclean", "distrebuild" and "redeploy"), it is quite possible and even sensible to exert finer control over a build by deleting (or, indeed, "touch"ing) tag files in the .muddle directory.

Do what I say, not what I do

Please note that although I have been explicitly requesting indiviual builds, rebuilds, deployments and so on above, this is not necessarily the normal way of using muddle. The muddle command line tool is carefully designed so that it normally does "the right thing", according to which directory you are in.

So, if one is at the top level of the build, and has checked out everything, it is more colloquial just to give the muddle command with no arguments than to be overly specific.

This is perhaps most easily shown by, well, showing it.

We can revert to the just checked out state quite easily, by just removing everything except the .muddle/ and src/ directories:

$ ls -AF
deploy/  install/  .muddle/  obj/  src/
$ rm -rf deploy/ install/ obj/
$ rm -rf .muddle/tags/package/ .muddle/tags/deployment/

Once we're back to the stage just after init, we can just do:

$ m3
Building deployment:cpio_dep/deployed
> Building package:pkg_cpio{x86}/preconfig
...
> Building package:pkg_cpio{x86}/configured
...
> Building package:pkg_cpio{x86}/built
...
> Building package:pkg_cpio{x86}/installed
...
> Building package:pkg_cpio{x86}/postinstalled
> Building deployment:cpio_dep/deployed
...
> Writing /home/tibs/sw/m3/example/deploy/cpio_dep/my_archive.cpio ..
...
> Make directory /home/tibs/sw/m3/example/.muddle/tags/deployment/cpio_dep

(I've shortened the log above, because by now it should be all too familiar.)

Contrariwise, if we clean it all up again:

$ rm -rf deploy/ install/ obj/
$ rm -rf .muddle/tags/package/ .muddle/tags/deployment/

and go into a particular checkout directory:

$ pushd src/cpio_co
$ m3
Killing package:pkg_cpio{x86}/built
Clearing tags: package:pkg_cpio{x86}/built package:pkg_cpio{x86}/installed package:pkg_cpio{x86}/postinstalled package:pkg_cpio{x86}/built
Building package:pkg_cpio{x86}/*
> Building package:pkg_cpio{x86}/distclean[T]
> Make directory /home/tibs/sw/m3/example/obj/pkg_cpio/x86
> Make directory /home/tibs/sw/m3/example/install/x86
> make  -f Makefile distclean
rm -f /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world
Distclean is just a clean
> Building package:pkg_cpio{x86}/clean[T]
> make  -f Makefile clean
rm -f /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world
> Building package:pkg_cpio{x86}/preconfig
> Make directory /home/tibs/sw/m3/example/.muddle/tags/package/pkg_cpio
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled
$ popd

That is, muddle only rebuilds the packages that depend on this checkout. See muddle help build and its friends for more information on this.

People who like to cd around the filesystem a lot will probably find this intuitive. Those of us who prefer to sit at the top-level and work from there (not necessarily any more sensible, of course) are more likely just to do:

$ m3 distrebuild pkg_cpio

(yes, this has identical effect, and no, by now I'm not going to include the listing yet again).

Note

As an aside, if you want to preserve the source of a build, then (if there aren't any domains) it is sufficient to do:

$ tar -zcvf build.tgz .muddle/Description .muddle/RootRepository .muddle/tags/checkout src

Issue 125 suggests adding a muddle command to make this convenient (including in the with-domains case).

muddle and its directories - part 3 - mechanics

2010-09-29T20:44:00.001+01:00

Contents

The mechanics of the build - the build description
Some naming conventions
Introspection of the dependency tree
Checking out the source code
The mechanics of the build - the makefile

The mechanics of the build - the build description

The build description is one or more Python files, which the muddle command runs before doing anything else.

(A few muddle commands do not need a build description, obviously including init, but these are the exception.)

For various reasons, the main build description file is traditionally called src/builds/01.py. muddle automatically adds the directory containing the main build description (the file named in .muddle/Description) to the PYTHONPATH for the build description. Put more simply, the build description can import other Python files in the same directory, which makes it easy to split build descriptions if necessary.

This example build has a fairly simple build description, in src/builds/01.py:

#!  /usr/bin/env python
#
# An example of how to build a cpio archive as a
# deployment - e.g. for a Linux initrd.

import muddled
import muddled.pkgs.make
import muddled.deployments.cpio
import muddled.checkouts.simple

def describe_to(builder):
    # Checkout ..
    muddled.checkouts.simple.relative(builder, "cpio_co")
    muddled.pkgs.make.simple(builder, "pkg_cpio", "x86", "cpio_co")
    muddled.deployments.cpio.deploy(builder, "my_archive.cpio",
                                    {"x86": "/"},
                                    "cpio_dep", [ "x86" ])

    builder.invocation.add_default_role("x86")
    builder.by_default_deploy("cpio_dep")

# End file.

Briefly, muddle looks for a function called describe_to() in the build description, and runs it to define the build.

This particular example is quite old, and I'm not sure we'd write it quite like that any more. Regardless, it says the following things:

there is a checkout called "cpio_co", which is checked out from the default repository (as defined in .muddle/RootRepository).
the package named pkg_cpio in role x86 is built from that checkout - this will, by default, be done using a Makefile in the checkout directory.
when deploying the final results of the build, a CPIO archive shall be created, with the role "x86" going at the root of the filesystem in that CPIO archive. The name of this deployment is "cpio_dep".
the default package role to build is "x86"
the default thing to deploy is the "cpio_dep" deployment.

Some naming conventions

There are some strong naming conventions associated with muddle build trees:

The build description lives in a checkout called src/builds/
All packages build out-of-tree (that is, in the obj/ directory, which we shall describe later), so that the src/ directory only contains the checkout files as-checked-out.
If there is a checkout containing "helper" makefiles and other associated things, used to build checkouts in other directories, it should be called src/helpers/.

(For instance, if one is building kbus, and downloading it directly from its google code repository each time, then it is convenient to put the muddle-specific makefile in a different checkout, possibly as src/helpers/Makefile.kbus)

There was an earlier convention of using the name src/builders/ for such a directory, but this is visually confusing, and does not work well with tab-completion at the bash prompt.
Checkout names should reflect the name of the external package being built, and generally also the version. So, for instance screen-1.0.3.
Package names should be more general - if a package is built from screen-1.0.3, it would normally be named screen. This allows for a later change in the build to use (for instance) checkout screen-1.0.4 instead.
Deployment names should reflect the purpose of the deployment - for instance, firmware versus kernel, and so on. This is very much dependent on the purpose of the build itself.

Note

This particular build is not really following the naming convention for checkouts, packages and deployments as described above. This is because it is trying to make it very clear which name belongs to which label type (thus co_cpio, pkg_cpio and dep_cpio)

It is also moderately traditional to call the main Python file for a build description 01.py (or 02.py if it is version 2, and so on). This is not, however, a strong convention, it just makes the "main" file easy to spot.

Introspection of the dependency tree

muddle does have some ability to display the dependency tree that is generated from the build description, although it is not as user-friendly as we would like (and, ultimately, there should be graphical tools).

The simplest tool for dumping the dependency rules is the depends command (see muddle help depends for more details on what it does).

The following shows the dependency rules described directly by the build description above:

$ m3 depends user-short
-----
checkout:builds/changes_committed <-VcsCheckoutBuilder-- [ checkout:builds/up_to_date[T] ]
checkout:builds/changes_pushed <-VcsCheckoutBuilder-- [ checkout:builds/changes_committed ]
checkout:builds/pulled <-VcsCheckoutBuilder-- [ checkout:builds/checked_out, checkout:builds/up_to_date[T] ]
checkout:builds/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:builds/checked_out ]
checkout:cpio_co/changes_committed <-VcsCheckoutBuilder-- [ checkout:cpio_co/up_to_date[T] ]
checkout:cpio_co/changes_pushed <-VcsCheckoutBuilder-- [ checkout:cpio_co/changes_committed ]
checkout:cpio_co/pulled <-VcsCheckoutBuilder-- [ checkout:cpio_co/checked_out, checkout:cpio_co/up_to_date[T] ]
checkout:cpio_co/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:cpio_co/checked_out ]
deployment:cpio_dep/deployed <-CpioDeploymentBuilder-- [ package:*{x86}/postinstalled ]
package:pkg_cpio{x86}/built <-MakeBuilder-- [ package:pkg_cpio{x86}/configured ]
package:pkg_cpio{x86}/configured <-MakeBuilder-- [ package:pkg_cpio{x86}/preconfig ]
package:pkg_cpio{x86}/installed <-MakeBuilder-- [ package:pkg_cpio{x86}/built ]
package:pkg_cpio{x86}/postinstalled <-MakeBuilder-- [ package:pkg_cpio{x86}/installed ]
package:pkg_cpio{x86}/preconfig <-MakeBuilder-- [ checkout:cpio_co/checked_out ]
-----

Each line above shows three things:

a target label.
an arrow containing the name of the action to take to "satisfy" or "build" that label (nb: in the current trunk of muddle, this is not present).
a list of the labels that must be "satisfied" or "built" before that action can be performed.

It is sorted by target label, which unfortunately is not too much help, but one can see that, for instance, the deployment:cpio_dep/deployed label depends on package:*{x86}/postinstalled, where the * is a wildcard over all package names in the {x86} role.

A shorter and perhaps more helpful representation of that can be provided using the query deps command:

$ m3 query deps deployment:cpio_dep/deployed
Build order for deployment:cpio_dep/deployed ..
checkout:cpio_co/checked_out
package:pkg_cpio{x86}/preconfig
package:pkg_cpio{x86}/configured
package:pkg_cpio{x86}/built
package:pkg_cpio{x86}/installed
package:pkg_cpio{x86}/postinstalled
deployment:cpio_dep/deployed

This shows all of the labels that must be built before deployment:cpio_dep/deployed, and in what order they will be built.

Checking out the source code

Normally, after the init command, one would just use a "bare" muddle command to perform the rest of the checkouts, build them (as they are checked out), and do any other steps indicated by the build description. However, since I'm interested in the various stages of the build tree, I shall do all of these things one by one.

So we start by checking out the actual source for our build:

$ m3 checkout _all
> Building checkout:cpio_co/checked_out
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/cpio_co cpio_co
A    cpio_co/hello_world.c
A    cpio_co/Makefile
Checked out revision 458.
> Make directory /home/tibs/sw/m3/example/.muddle/tags/checkout/cpio_co

After doing this, we've now gained a new tag in the .muddle/ directory:

`-- .muddle/
    |-- Description
    |-- RootRepository
    `-- tags/
        `-- checkout/
            |-- builds/
            |   `-- checked_out
            `-- cpio_co/
                `-- checked_out

indicating that we have successfully checked out the cpio_co checkout, and the source files for our missing checkout are now present in the src/ directory:

`-- src/
    |-- builds/
    |   |-- 01.py
    |   |-- 01.pyc
    |   `-- .svn/
    `-- cpio_co/
        |-- hello_world.c
        |-- Makefile
        `-- .svn/

(whilst I'm showing the .svn/ directories here, I've removed the listing of their contents).

The mechanics of the build - the makefile

This build only has a single checkout, which produces a single package. As such it also has a single makefile, src/cpio_co/Makefile:

# Makefile for cpio_co
#
# Just dumps some compiled C into the right place.

INSTALL=install

all:
        $(CC) -o $(MUDDLE_OBJ)/hello_world hello_world.c

install:
        if [ ! -d $(MUDDLE_INSTALL)/bin ]; then mkdir $(MUDDLE_INSTALL)/bin; fi
        $(INSTALL) -m 0755 $(MUDDLE_OBJ)/hello_world $(MUDDLE_INSTALL)/bin/hello_world
        $(INSTALL) -m 0644 hello_world.c $(MUDDLE_INSTALL)/hello_world.c

config:
        @echo Nothing to do

clean:
        rm -f $(MUDDLE_OBJ)/hello_world

distclean: clean
        @echo Distclean is just a clean

# end file.

We saw when querying the builds dependencies (in Introspection of the dependency tree) that various dependency rules had MakeBuilder as their action. This action knows what targets in a makefile to call in order to build a particular package: label tag.

The target label tags (the /xxx at the end of a label) correspond to makefile targets as follows:

Target tag Makefile target

/preconfig <none>

/configured config

/built all

/installed install

/postinstalled <none>

Target tag Makefile target

/clean clean

/distclean distclean

Target tag	Makefile target
/preconfig	<none>
/configured	config
/built	all
/installed	install
/postinstalled	<none>

Target tag	Makefile target
/clean	clean
/distclean	distclean

It can be useful to think of a package as "progressing" from /preconfig through to /postinstalled.

preconfig does not correspond to a make target, and there is no default action for it. Things that depend on a package existing (but nothing else) will depend on the /preconfig tagged package label.

There is no action defined by default for /preconfig. That doesn't stop a particular build from defining one - for instance, ExpandingMakeBuilder is a MakeBuilder subclass which expands a .tgz file into source files for the later stages to compile and install (see muddled.pkgs.make).

Although there is no make target associated with /postinstalled, it is at this stage that pkg-config files are rewritten (if requested), and this is the tag that should be used when another package depends on something being "fully" built.

muddle and its directories - part 2 - example build

2010-09-29T20:43:00.001+01:00

Contents

A simple example build
The .muddle directory
The src/builds directory

A simple example build

The muddle repository has a variety of example build descriptions, of varying complexity.

Perhaps inevitably, not all of the examples are maintained as well as they should be. If you do find problems with any of them, please raise an issue about it on the muddle issues page.

We shall use the "cpio" example.

We start with a new, empty directory:

$ mkdir example
$ cd example

and then use the muddle init command:

$ m3 init svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio builds/01.py
> Make directory /home/tibs/sw/m3/example/.muddle
Initialised build tree in /home/tibs/sw/m3/example
Repository: svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
Build description: builds/01.py


Checking out build description ..

> Make directory /home/tibs/sw/m3/example/src
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/builds builds
A    builds/01.py
Checked out revision 458.
> Make directory /home/tibs/sw/m3/example/.muddle/tags/checkout/builds
Done.

This leaves us with two new directories:

$ ls -AF
.muddle/  src/

Or, in more detail (with the .svn/ directory in src/builds/ shown as a "stub"):

.
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       `-- checkout/
|           `-- builds/
|               `-- checked_out
`-- src/
    `-- builds/
        |-- 01.py
        |-- 01.pyc
        `-- .svn/

Whilst the name and content of the src/builds directory might differ, this is the normal state of a muddle build tree after the init command.

The `.muddle` directory

The .muddle/ directory is, in many ways, the heart of the muddle build tree.

In the first place, it identifies a directory as the top-level of a muddle build. Indeed, the muddle command line tool looks in the current directory, and then upwards through the filesystem, to find a .muddle/ directory. If it cannot find one, then it decides that it is not in a muddle build tree, and behaves accordingly (depending on what action it was asked to do).

The .muddle/ directory always contains at least two files and one directory:

The `Description` file

$ cat .muddle/Description
builds/01.py

which should be recognisable from the init command. It tells muddle where its build description was checked out (in the src/ directory, since that is where checkouts go).

You should not normally need to edit this file, but if you do, the next time you run muddle, it will believe that the build description is in the new location.

The `RootRepository` file

$ cat .muddle/RootRepository
svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio

Again, this should be recognisable from the init command. This specifies the default repository, which muddle will assume is being used for checkouts unless the build description says otherwise.

You should not normally need to edit this file either, but again muddle will believe its contents after a change.

The `tags/` directory

The tags/ directory contains the current state of the build - it is essentially a database of "labels satisfied", stored in the filesystem.

In the current state:

.muddle/tags
`-- checkout/
    `-- builds/
        `-- checked_out

we can see that the label checkout:builds/checked_out has been "built" (or has had its target satisfied). This corresponds to having checked out the builds checkout, which is indeed what has been done. We shall see later that it is possible to change the build state by deleting or "touch"ing tag files.

The checked_out file created by muddle does have content - it contains a timestamp:

$ cat .muddle/tags/checkout/builds/checked_out
2010-09-28 11:14:34

but this content is never actually used for anything, and if you do add a tag file, it is quite sufficient to use touch to create an empty file.

If we ask muddle what checkouts it knows about:

$ m3 query checkouts
builds
cpio_co

we see two checkouts. The absence of any tag files for the "cpio_co" checkout tells us that it hasn't been checked out yet.

The `src/builds` directory

The other directory present is the src/ directory, which, so far, contains a single subdirectory for the build description.

`src/
 `-- builds/
     |-- 01.py
     |-- 01.pyc
     `-- .svn/

muddle and its directories - part 1 - introduction

2010-09-29T20:33:00.001+01:00

This is a look at how muddle works, specifically looking at the directory structure of a muddle build.

Since it is quite long, I am splitting it into a series of 7 posts, of varying lengths. The whole thing will also ultimately be integrated into the "proper" muddle documentation.

If you are just going to use muddle to check out and build an existing build tree, then this will probably be much too much information, although the short section An overview of the directory structure (at the end of this introductory post) may be useful.

If you are going to be developing new muddle builds, then this text should enable you to do so more effectively.

If you are going to be reading the muddle source code, or developing muddle itself, then it should explain much of the intent behind muddle's workings.

Contents

Introduction
Notation in this text
An overview of the directory structure

Introduction

muddle is three or four different things, entwined:

It is a command line tool, providing useful tools for managing build trees on Linux.
It is a Python package, muddled
It is a simple dependency tracking and analysis tool, with its state made evident via the file structure.
It is a binding of the above to the common tasks required for managing, building and preparing deployments of Linux software from common sources to multiple architectures and platforms.

In a perfect world (perhaps the eventual "muddle 3") these would be better separated. In particular, it would be nice if it was obvious that muddle can be adapted for things other than Linux build trees.

The best short introduction to muddle is still that written by Richard Watts (its primary author), the README for the project, titled "Welcome to Muddle", which can be found online at:

http://muddle.googlecode.com/svn/trunk/muddle/docs/_build/html/README.html

I recommend it for further study.

This document is meant to be a longer and more discursive introduction to some of the basics of muddle, with a concentration on how the muddle build tree works as a directory tree, and with worked examples.

Notation in this text

Directory names are consistently shown with a trailing /, as by ls -F. Since I'm talking about directories and files a lot, I think this makes it easier to tell which is which.
The "tree" listings of directory structures are produced with the command tree, normally as:
```
$ tree -aF -I .svn --noreport
```
i.e., showing files and directories that start with a dot, suffixing directory names with /, not showing .svn/ directories, and not reporting on the number of files and directories. In a few places the "stub" of a .svn/ directory (that is, just its name) may be shown - this should be obvious from context.
When referrring to the parts of a label (type:(domain)name{role}/tag) it is useful to keep the punctuation that indicates that part (so, type:, (domain), name (no punctuation!), {role} and /tag).

Remember that, in the current scheme of things, only package: labels use {role}, and "normal" builds do not use (domain).
In this document I use the command m3 instead of muddle.

The simple reason for this is that I made the examples using the version of muddle I'm currently working on, the muddle3_labels ("labels all the way down") development branch, and I'm using m3 to distinguish it from the "normal" muddle.

(For the purposes of these examples this should not matter, as the functionality is substantially the same, although the output of some of the commands may be slightly different.)

However, it is also a useful reminder that the muddle command need not be called muddle. This is particularly important to remember when writing makefiles that call back to muddle - they should use $(shell $(MUDDLE) ...) rather than the tempting $(shell muddle ...), for that very reason.

An overview of the directory structure

Normal practice is to keep a muddle build in its own directory. This is not a requirement, but it helps keep it more obvious that it is a build.

The muddle directories are as follows:

.muddle/: contains information about the build state, and signifies that this is a muddle build.
src/: contains the build description and any other checkouts.
obj/: generated by muddle to hold the results of building packages.
install/: generated by muddle to hold the results of "installing" packages (this will be explained later on).
deploy/: generated by muddle to hold the results of "deploying" things in install/.
versions/: generated by muddle to hold the result of muddle stamp version.
domains/: present if there are sub-domains, muddle will create this directory if it is needed.
instructions/: present if there are instruction files for use in deployment. muddle will create this directory if it is needed.
am_subdomain: generated by muddle in subdomains. Ignore it.

Some notes on cross-compiling in a muddled world

2010-09-27T16:17:00.001+01:00

Obviously one of the main aims of developing muddle has been to make it easier to build cross-compiled systems, since that's what I spend a lot of my time doing.

Lately I've been taking a DEB-based build and turning it into a source-based build. This article is some notes on things I've learnt (or had reinforced) in the process.

Makefiles

For the purpose of this post, I'm specifically talking about use of GNU make - other make programs may have different requirements (and portability between make programs is a different issue).

So, some tips for making makefiles that will play well with muddle (whether they are Makefile.muddle files, called directly by muddle itself, or new makefiles for a particular project).

Additive assignment

Always use CFLAGS+= and not CFLAGS= in your makefile (and similarly for other variables that may be inherited from "outside", such as LDFLAGS, CPPFLAGS and CXXFLAGS).

Consider as an example if the makefile does not specify -Wall in its C flags, but the user wishes to do so. If the makefile sets:

CFLAGS = -Iinclude -o2

then this is difficult, but if instead the makefile does:

CFLAGS += -Iinclude -o2

then the caller can safely add their own flag(s).

Don't be overspecific

For somewhate similar reasons, never, ever actually set CC and its ilk in the Makefile.

Too often, the developer of a system may need to do the equivalent of:

CC = "gcc -m32"

or even:

LD=$(CC)

Whilst there are ways of overriding CC=xxx settings in the makefile, these are unobvious, and if the setting was amending the value (i.e., if the makefile itself did CC=gcc -m32) then it is bound to go wrong...

Immediate versus deferred

Remember the difference (we're assuming GNU make) between = and :=, and which one will get evaluated multiple times and thus slow things down.

Specifically, if you're going to do:

PKG_OBJDIR := $(shell $(MUDDLE) query objdir package:pkg{role}/built)

then it is worth knowing that := will evaluate this once, and = will cause it to be evaluated each time $(PKG_OBJDIR) is used. Clearly, we assume that a package's objdir is not going to change, and we would rather not run the muddle command line program multiple times if we don't need to.

(Here I recommend reading the GNU make documentation, which can be found at http://www.gnu.org/software/make/manual/make.html. Look up "immediate" versus "deferred" expansion.)

$(MUDDLE), not muddle

Speaking of which, in a muddle-based world, always use $(MUDDLE) in a makefile, instead of an explicit muddle. muddle itself sets this environment variable to the muddle command-line program, which may not actually be called muddle (in my muddle3 development environment, I use m3), and also may not be on the PATH.

(This is similar to the strong recommendation to always say $(MAKE) instead of make, although this latter carries more implications of what happens to your environment than is the case with muddle.)

The evils of TARGET_ARCH

And finally, don't try to use an environment variable called "TARGET_ARCH".

If $(TARGET_ARCH) has a value, then GNU make will insert it onto the $(CC) command line (and maybe other comamnd lines, as well). This is not documented behaviour (it is not mentioned in the above referenced manual), although one can find out about it by searching the net. It looks as if it is expecting this to be some sort of flags for the compilation (so it should be called TARGET_ARCH_FLAGS), but no-one seems to know or remember why it was introduced.

In fact, GNU make treats a lot of environment variables specially, many of which are not documented. You can use:

make -p

to get a listing of these, as well as the actual implicit rules used (for $(CC) command lines, etc.), allowing one to determine where $(TARGET_ARCH) might actually be used.

Random other notes

libicu

Many packages and libraries are difficult to build with cross-compilation - it's not something best served by the autotools framework, and it's not something that most package builders will have needed to worry about.

There are exceptions, however, and one of the nicest may be libicu (libicu4c, from http://site.icu-project.org/). Its documentation clearly explains how to cross-compile it (basically, build it once on the host, and then use that host build as an enabler for building for the target), and the mechanism to do the building works well.

Assembler, C and underscores

Various mechanisms need to know whether external C symbols need an extra underscore (_) prepending for use from assembler. configure can't decide this, because it wants to do it by trying to build and run a program (which won't work for cross compilation!). In the better packages, it just refuses to guess, and tells the user to find out and define a value to let it know. In the less helpful packages, it silently defaults to one choice or the other.

The former choice is a pain, but clearly the correct thing to do.

The latter choice, found, for instance, in libgcrypt11, is unhelpful. In this particular instance, if it can't tell (because it is cross compiling) it defaults to "yes", which was the wrong choice. This took a while to debug.

gcc as linker

It is an advantage to use gcc or g++ as the linker program ($(LD)), because they implicitly know where their libraries are. Some packages insist on using ld instead, which does not.

One package tried to decide a value by parsing the output of ld's "help" output, which failed in various ways.

muddle and pkg-config

Whilst I may grumble about autotools in general, it should be noted that pkg-config does not seem to be too great a problem, as it is quite possible to amend the text files it uses to describe libraries.

muddle does this quite well, but only if you tell it (via the appropriate arguments to make.twolevel() or whatever) that this should be done. In the current version of muddle, this means specifying the rewriteAutoconf and usesAutoconf arguments as True.

libtool grumbles

libtool itself may or may not be so bad, but when it is used without consideration for future cross-compilation, it can be a real problem.

For the moment:

BAD packages that insist on putting /lib on the library search path explicitly. Bad libtools for not handling this. There may be a fix round the corner, but of course this will never appear in many packages, since they will not be updating their autotools setup...
The whole libtool and can't install to library ending with problem, which is a bug in design, and I've still not figure out how to fix in any clean way...

(if you haven't come across this, it's a particular problem for some gstreamer plugins, and seems in the short term to be a show-stopper)

We shall be at EuroPython 2010

2010-06-08T13:30:00.000+01:00

Both Richard and I are giving talks at EuroPython 2010 in Birmingham. Richard will be talking about muddle, and I shall be talking about Kbus, starting (as currently scheduled) at 17:00 on the afternoon of Monday 19th July.

I'm intending to be at the conference all four days (Monday through Thursday), whilst Richard may only be there on Monday.

Hope to see you there...

Interesting times for OTT video

2010-04-25T15:45:00.006+01:00

So it looks as though we're about to be living in interesting times for OTT video.

Apple and Adobe seem to be having a spat over Flash; specifically, over Adobe's irresistable force of content vs Apple's immovable object of control. A side-effect of this is likely to be that the long-awaited Flash 10.1 for devices will end up on Android first.

This is going to be a bit of a pain for a lot of people; STB manufacturers will need to end up either embedding an X server (for the Linux version) or running Android (for the android version) and neither of those is a good plan - X is big and wastes cycles, and more importantly is terribly buggy for a TV-grade product. Android is small and efficient but entirely convinced that it is a mobile phone and it's very hard to dissuade it from thinking so. I'm thinking that the best way around this at the moment is to run their userlands side-by-side, but it's not a pretty best way around. Hopefully most people will buy their way out via startups like Vidiactive .

In other news, Theora for HTML5 video seems not to be about to go away, and Google are rumoured to be releasing VP8 next month, whilst the MPEG seem to be pressing ahead with their H.265 effort.

Contrary to rumour, Theora and VP8 should not be the performance killers they're generally supposed to be, except on legacy systems. Owners of ST710x s are probably hosed, but both codecs are designed to be fast on general-purpose CPUs so the current generation of chips should be perfectly happy with SD VP8. HD will probably require either a GPU or a programmable video accelerator - On2 used to claim on their website that they could get 720p VP8 decode out of a 1.5GHz ARMv5, so I'm betting an OMAP3730 should be able to get D1 out of its ARM and 720p if you split decode between the cores: Theora 360p decode pretty much works on an N900.

There are, of course, three elephants in the room

The first is adoption: I think in practice we're going to see the world move over to VP8/Vorbis and HTML5 video over the next few years. Flash is just too nasty, big and expensive to survive - particularly since the spec seems to just keep growing. This is not necessarily bad news for Adobe: they've never monetised the Flash client and it is frankly just a drain on their development resources that they can happily live without.

The second is audio: Little-known fact - you don't have to pay royalties for streaming H.264 over the internet but the same is not true of any of the popular audio formats: MP3, AAC and AC3 royalty rates are all much larger than the $0.10/unit H.264 royalty - which in any case only applies if you ship more than 5k units or so of a decoder - and none of them allow you to stream their audio over the network for free, though they're obviously waiting to see how things pan out before asserting their rights. From a royalty regime point of view it's much more important to replace AC3 and MP3 with Vorbis than it is to replace HTML5 with Theora. Particularly since the MP3 licensors, in particular, are apt to charge punitive rates to implementations which take other formats too - this being one reason why IPTV broadcasts generally avoid mp3.

The third is, of course, content access. YouTube in particular has been notable in taking a traditional-media attitude to online video: they'll talk to you, if you have 1m subscribers. If you don't, you don't get their content.

Hopefully, media providers like the BBC and YouTube will take a liberal position and realise that all they need for monetisation is some compliance provisions, such as are already in place with WMDRM. Otherwise, we'll be in for the usual content-access fight as the providers attempt to leverage their content to control the market in decoding devices, hosing consumers in the process. This never works, but it doesn't seem to stop people from trying.

One thing that would help free formats' cause here is an open-source DRM system. The idea isn't a contradiction in terms - there is no reason why you shouldn't 'voluntarily' lock down your STB so as to prevent you from taking digital copies of media without compromising the freedom of the rest of the system, and it is certainly better to have a free, open standard for DRM than to either lock all worthwhile content behind expensive, proprietory technology or to allow unrestricted piracy.

If we're lucky, google checkout/paypal and some smarts from the content providers should see micropayment VoD actually bear fruit. I remember going to a seminar about that in 1993.

Why KBUS?

2010-03-25T20:31:00.001Z

So why are we developing KBUS, rather than using some other messaging system?

(I'm going to assume you've seen the previous blog items on KBUS, and/or read the KBUS documentation on its Google code page.)

Our background

We work primarily in the embedded world, specifically on Set-Top Boxes (STBs). As such, typical software elements include video and audio decoders (or the interfaces to them, if this is done by hardware), user interface via remote control or keyboard, and some form of GUI, typically a web browser.

Clearly, some mechanism is required to provide communication between all of these elements.

We have had experience of bad solutions in the past - their flaws include such things as race conditions (for instance, when a browser crashes, it cannot reliably resume communication with the other processes it needs to liaise with), unreliable implementations (frustrating when one is not allowed to fix them) and poor documentation (well, some of that was fixable).

Aims

We thus set out with the following aims for our solution:

Simple models to "think with", so that we can have a well understood system.
Predictable delivery.
Always get a reply to a request.
Messages (on a particular bus) are in the same order for all recipients.
Small implementation size.
Base code available in C (C++, for instance, is not always available on the platforms we work with).
Good usability from Python (well, that was my requirement)
We'd really prefer an open source package, and we definitely want one that is actively maintained.

and that didn't really seem to leave us with an option other than writing it ourselves.

Simple models

Names for things

We've striven for simple names for things:

KBUS devices are the buses over which communication happens.
Ksocks are the connections to those buses. The name is meant to suggest they are a bit like sockets, but not quite.

(I'm not actually terribly happy with "Ksock", but it's difficult to come up with good names for things, and it's better than the working name of "Elephant" that I was using for a short while in early development.)
The basic messaging entities are Senders, Listeners and Repliers - one should already be able to guess what they do.
The basic types of messages are Announcements, Requests, Replies - again, these should be fairly obvious.
Message names are defined fairly simply, with (we hope) just enough flexibility, and the parts of a message (to and from, in_reply_to, etc.) are hopefully not too hard to understand from their names.
I'm sorry about Limpets.

Interfaces

There are three levels of interface provided:

The "bare Unix" level.

We use a kernel module to provide our devices, which are named as /dev/kbus0 (for bus 0), /dev/kbus1 (for bus 1), and so on.

Ksocks are then implemented with file operations - open, read, write, close and IOCTLs), with which experienced Unix programmers should already be familiar,
The Python API. This was written as the primary testing API, and works with classes that match the main named things. I believe this to be fairly easy to use. I also use it as the main way of illustrating how KBUS works.
The C library. This hides the details of the "bare Unix" level, and also removes the worry about handling such things as ernno when using IOTCLs. It is intended to be the normal means of using KBUS from C, and should also be useful when writing interfaces in other languages (which can typically call C).

The KBUS kernel module

Using a kernel module means that:

We can have a file interface, which makes KBUS easier to use.
We can have a real expectation of our "daemon" not crashing (it is much easier to write a kernel module that is reliable, partly because there are so many constraints on how one does it, partly because one is executing in a different context, and partly because kernel mechanisms mediate the modules interaction with user space).
We get to use relatively sophisticated and proven datastructures. Kernel modules are expected to use the provided mechanisms for handling lists and other datatypes. This avoids a lot of reinventing the wheel, or dependency on other libraries which might not be present.
The kernel hides a lot of the complicated stuff (both at the top and bottom level) from us, so we can't do it wrong (well, it's much harder). For instance, read and write at the user level get filtered down into more predictable calls at the kernel module level.
We stand to gain from the kernel handling such issues as multiple CPUs, threading and so on.

Messages

In order to keep KBUS itself simple, KBUS does not say anything about the message content. It restricts itself to defining the message header and the mechanisms for managing messages.

(We do have a nearly finished ASN.1 library for message data, and are looking at XMPP support, but these will be extras, not core KBUS. And, of course, one can use other mechanisms as one wishes.)

As indicated in the section on naming, the fields in the header aim to be easy to understand, and we try to define just the fields we need.

Predictable delivery

It is acceptable for a Listener to miss messages (although there are ways around that), but a Replier shall never miss the Requests sent to it.

We also want to guarantee that each Request shall produce a Reply (even if it is a Reply indicating failure to deliver the Request).

So:

If a sender attempts to send a Request, but does not have room on its message queue for the (corresponding) Reply, then the message will not be sent, and the send will fail. At the "bare Unix" level, this means that the send IOCTL returns an error - the failure is immediate.
If a replier cannot receive a particular message, because its queue is full, then the message will not be sent, and the send will fail with an error. Again, this failure is immediate.
If a message has the ALL_OR_FAIL flag set, then a send will only succeed if the message could be added to all the (intended) recipient’s message queues (listeners as well). Otherwise, send returns -EBUSY.
If a message has the ALL_OR_WAIT flag set, then a send will only succeed if the message could be added to all the (intended) recipient’s message queues (listeners as well). Otherwise send returns -EAGAIN.

(The sender then needs to discard the message, or play the poll/select game to wait for the send to finish).

Note that we believe these last two mechanisms are primarily of use when debugging systems.

Finally:

"""KBUS guarantees that each Request will (eventually) be matched by a consequent Reply (or Status) message, and only one such."""

If the replier can't give a Reply, KBUS will generate one (e.g., "$.KBUS.Replier.Unbound" or "$.KBUS.ReplierGoneAway").

Message order is the same for all

It is important that if sender A sends message M(a), and sender B sends message M(b), then all recipients of both messages will see them in the same order.

(Imagine sending instructions to a video decoder and a video recorder. Clearly both may need to receive the same instructions, and it is important to receive the instructions in the appropriate order.

Similarly, consider a logging Listener. This too clearly wants to receive messages in the same order as the other Listeners. It especially wants to see Requests and Replies in the appropriate order.)

Since KBUS (the kernel module) has control over both ends of the transactions, this is fairly simple to guarantee.

Why not use?

What else could we have used?

There appear to be two main things we could have considered, DBUS and zeromq.

DBUS

I've not spent an awful lot of time looking at DBUS, but my initial thoughts are that:

I don't understand it. Its documentation seems to show an over complex model, with no quick way in to understanding it, and a wish to split things into as many levels as possible.
It is socket oriented (not a kernel module), which is probably the best approach for a user-space daemon, but
it is in user-space, with the problems that can entail.
I am told it is not all reliable...
...or even all implemented
Perhaps most importantly, it doesn't preserve message ordering.
It is large.

But it is widely used (hmm) and has been around quite a while.

zeromq / 0mq

zeromq (or 0mq) looks rather nice. It has good introductions, and seems to have a clear idea of what its aims are, in particular aiming for speed and scalability.

Its messages are minimalistic in strucure (a name and then content), which is really rather nice. It is also very cross platform, both in the "implemented on" sense, and in the "available for language X" sense.

It doesn't appear to b e aiming for the sort of "predicability" we're after (or so I deduce from a scan of the documentation). And it is written in C++, which rules it out for some prospective platforms.

However, this looks like one it would be fun to play with, and I definitely need to learn more about it.

(It's very tempting to think about it as a potential means of moving KBUS messages over networks, for instance.)

What else?

I must have missed systems that I really should know about.

(Although note I'm ignoring many "enterprise space" systems, which often do seek guarantees of delivery, but at the cost of being an enterprise system.)

Saving and restoring the state of a muddle build

2010-03-25T19:50:00.001Z

So over the last week or so I've been working on adding commands to muddle so that we can store and restore "version" information for a build tree.

The basic idea is to be able to save enough information to a file to allow reconstructing the main build structure, and the content of every checkout.

A build description doesn't necessarily suffice for this, firstly because you don't actually have to specify a revision id for a particular checkout (one can default to HEAD), and secondly because the user of a build might have updated checkouts since the build description was first used to extract it.

So, here is some documentation for the result (so far, at least).

The following applies to muddle revision 317 and after. I do not promise that details will not change (but I don't think they will). I also assume there may be bugs.

If you have comments/suggestions, please let me know.

Muddle version stamping

Build names

First of all, it is now possible to give a (hopefully) meaningful name to a build - for instance, one might call the project "Thing" build on Intel 'ProjectThing_Intel'.

This is done by adding a build_name property to the Builder class. What this means in practice is that in the build description you can now do:

builder.build_name = 'ProjectThing_Intel'

The name may only contain alphanumerics, underscores and hyphens.

If no name is given, the name of the build description file (without the ".py" extension) is used - so, for instance, the ProjectThing build might typically default to being called '01'.

Build names are intended to be more generic than the build description file name - after all, one of our projects in particular has progressed from 01.py to 02.py and may one day change again. It would make sense for the build name to stay the same throughout, however.

(Note that since the build name is set in the build description, it need not necessarily be a constant - so one could conceive of automatically naming 'ProjectThing_ARM' and 'ProjectThing_Intel' builds.)

You can find out the current build name with:

muddle query name

Stamp files

A stamp file is a description of a current build. Specifically, it describes the overall build (root repository and description), the names of any domains in the build, and a full description of each checkout used by the build.

For each checkout, all of the information needed to retrieve exactly the version in use (from its remote repository) is remembered.

If any checkouts contain uncommitted data, or do not appear to match an exact revision in their remote repository (in other words, if it appears that it would not be possible to retrieve them as-is from the remote repository), then the stamping process will warn the user, and will create a "partial" stamp file (extension .partial, instead of .stamp).

Note the file is a simple text file (it's actually a Windows-style .INI file, in fact), which means that it is possible to edit it with a text editor, if wished.

Muddle provides several commands for handling stamp files:

muddle stamp save creates a stamp file.
muddle stamp diff compare two stamp files. (This could do with providing the ability to compare with the current build, and also with a remote stamp file - these should be fairly easy to add.)
muddle unstamp creates a build from a stamp file. It understands how to retrieve a stamp file over the internet in various ways, as well as how to use a local file.
muddle stamp restore is a synonym for muddle unstamp - it's not entirely clear to me if muddle unstamp would be better replaced with this alias, so that all "stamp" related actions are subcommands of muddle stamp.
muddle stamp version is a specialisation of muddle stamp save. and is discussed below.

Version stamp files

A new conventional muddle directory is introduced, versions/.

This lives at the same level as .muddle/, src/ and domains/, and is covnentionally used to store version stamps.

A version stamp is created with muddle stamp version. It writes a stamp file called <build_name>.stamp to the versions/ directory.

The assumption is that one does muddle stamp version to create a new versions/ directory and put the (nicely named) stamp file there. One can then commit this versions/ directory to whatever repository one wishes, just as is done for the src/builds/ directory.

muddle unstamp then understands how to do:

muddle unstamp  bzr+ssh://kynesim.co.uk/repo  versions/ProjectThing.stamp

to clone the directory versions/ from bzr+ssh://kynesim.co.uk/repo/versions, and then unstamp the stamp file indicated. This is deliberately very similar to the way that muddle init works.

A developer may also, of course, make copies of a version stamp file and commit them - for instance:

cd versions/
cp ProjectThing.stamp ProjectThing-v1_0.stamp

The file must, however, still have extension .stamp.

More information

More information is available by doing:

muddle help stamp
muddle help unstamp

What is tested

I've tested stamping and unstamping with local stamp files.

I've tested version stamping, and then unstamping with file+file:// URLs, both direct to the stamp file and also via the "init-like" mechanism.

I've tested version stamping, committing to a local BZR repository, and then unstamping with a file+bzr:// URL, both direct to the stamp file and via the "init-like" mechanism.

Subversion and git are not tested.

Direct file unstamping is not supported for git (there seems to be no way to "extract" a single file without doing a proper clone). I assume if one wishes to do this one would use a URL provided by a webserver.

With thanks to

The Python library, and particularly to @property, collections.MutableMapping, ConfigParser, difflib, hashlib, urllib, urlparse and all the usual suspects.

KBUS Limpets - an introduction with goldfish

2010-02-28T19:35:00.002Z

This is a metaphorical goldfish called A.

This is another metaphorical goldfishcalled B.

A and B live in a metaphorical goldfish bowl.

Each metaphorical goldfish bowl has a KBUS device in it - this bowl has KBUS device 0:

Here are the two fish in their bowl.

Metaphorical goldfish are simple creatures. They can only communicate with each other using KBUS messages.

Here is another metaphorical goldfish bowl. This one contains metaphorical goldfish called R and G. Their bowl has KBUS device 3.

Unfortunately, A and B cannot communicate with R and G. Even if the two metaphorical goldfish bowls are running on the same computer, KBUS does not permit sending KBUS messages between different KBUS devices.

Luckily, KBUS provides Limpets.

A Limpet lives on the side of a metaphorical goldfish bowl, and communicates with another Limpet on another metaphorical bowl.

Here we have Limpet 1 connected to a Ksock on KBUS device 0, and Limpet 2 connected to a Ksock on KBUS device 3:

Limpets always come in pairs. Each Limpet can proxy KBUS messages from the KBUS device in its metaphorical goldfish bowl to and from the other Limpet in its pair.

Think of them as using very low power line-of-sight lasers to send messages between each other.

So a pair of Limpets hide the fact that the two KBUS devices are not the same. This means that fish A and G can talk just as if they were in the same bowl:

To the metaphorical goldfish, it is as if the messages magically pass between the two KBUS devices, and thus A and B can communicate with R and G.

This mechanism even allows such communication if there are intermediate bowls:

A few more technical details.

KBUS Limpets are currently experimental, and have not been extensively tested yet. Limpet daemons are available in Python and C (which intercommunicate happily).

In order to keep Limpets and their implementation as simple as possible, we're willing to put up with some limitations:

We use TCP/IP or named sockets to communicate between Limpets, which means one Limpet in each pair has to be a server, and thus started up first.
All Limpets on a connected network must have unique ids.
Limpets may not be used to form a network with loops in it. This greatly simplifies message management.
We assume a "safe" or trusted network - if it acts like a Limpet, it is a Limpet.
Finally, one cannot connect both ends of a Limpet pair to the same KBUS device (the same KBUS device number on different computers is OK, of course).

A simple introduction to using KBUS

2010-02-14T12:41:00.001Z

This is intended as a very simple introduction to the basics of how to use KBUS. As such, I'm sure I could improve it, but it's been waiting for long enough now that I think I should publish it anyway.

We shall start with a single "actor":

Terminal 1: Rosencrantz

$ python
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from kbus import *

I'm generally against doing an import of *, but it's reasonably safe with the KBUS python module, and it makes the tutorial shorter.

First our actor needs to connect to KBUS itself, by opening a Ksock:

Terminal 1: Rosencrantz
>>> rosencrantz = Ksock(0)

This specifies which KBUS device to connect to. If KBUS is installed, then device 0 will always exist, so it is a safe choice. The default is to open the device for read and write - this makes sense since we will want to write messages to it.

Once we've done that, we can try sending a message:

Terminal 1: Rosencrantz

>>> ahem = Message('$.Actor.Speak', 'Ahem')
>>> rosencrantz.send_msg(ahem)
MessageId(0, 1)

The first line creates a new message named $.Actor.Speak, with the message data "Ahem".

(All message names are composed of ``$`` followed by a series of dot-separated parts.)

The second line sends it. For convenience, the send_msg method also returns the message id assigned to the message by KBUS - this can be used to identify a specific message.

This will succeed, but doesn't do anything very useful, because no-one is listening. So, we shall need a second process, which we shall start in a new terminal.

Terminal 2: Audience

$ python
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from kbus import *
>>> audience = Ksock(0)
>>> audience.bind('$.Actor.Speak')

Here, the audience has opened the same KBUS device (messages cannot be sent between different KBUS devices). We've still opened it for write, since they might, for instance, want to be able to send $.Applause messages later on. They've then 'bound to' the $.Actor.Speak message, which means they will receive any messages that are sent with that name.

(In fact, all messages with that name sent by anyone, not just by rosencrantz.)

Now, if rosencrantz speaks:

Terminal 1: Rosencrantz
>>> rosencrantz.send_msg(ahem)
MessageId(0, 2)

the audience can listen:

Terminal 2: Audience

>>> audience.read_next_msg()
Message('$.Actor.Speak', data='Ahem', from_=1L, id=MessageId(0,2))

A friendlier representation of the message is given if one prints it:

Terminal 2: Audience

 >>> print _
<Announcement '$.Actor.Speak', id=[0:2], from=1, data='Ahem'>

"Plain" messages are also termed "announcements", since they are just being broadcast to whoever might be listening.

Note that this shows that the message received has the same MessageId as the message sent (which is good!).

Of course, if the audience tries to listen again, they're not going to "hear" anything new:

Terminal 2: Audience

>>> message = audience.read_next_msg()
>>> print message
None

and so they really need to set up a loop to wait for messages, something like:

Terminal 2: Audience
>>> import select
>>> while 1:
...    (r,w,x) = select.select([audience], [], [])
...    # At this point, r should contain audience
...    message = audience.read_next_msg()
...    print 'We heard', message.name, message.data
...
(although perhaps with more error checking, and maybe even a timeout, in a real example).

So if rosencrantz speaks again:

Terminal 1: Rosencrantz

>>> rosencrantz.send_msg(Message('$.Actor.Speak', 'Hello there'))
MessageId(0, 3)
>>> rosencrantz.send_msg(Message('$.Actor.Speak', 'Can you hear me?'))
MessageId(0, 4)

the audience should be able to hear him:

Terminal 2: Audience

We heard $.Actor.Speak Hello there
We heard $.Actor.Speak Can you hear me?

So now we'll introduce another participant:

Terminal 3: Guildenstern

$ python
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from kbus import *
>>> guildenstern = Ksock(0)
>>> guildenstern.bind('$.Actor.*')

Here, guildenstern is binding to any message whose name starts with $.Actor.. In retrosepct this, of course, makes sense for the audience, too - let's fix that:

Terminal 2: Audience

<CTRL-C>
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
KeyboardInterrupt
>>> audience.bind('$.Actor.*')
>>> while 1:
...    msg = audience.wait_for_msg()
...    print 'We heard', msg.name, msg.data
...

(as a convenience, the Ksock class provides the wait_for_msg() wrapper around select.select, which is shorter to type...).

And maybe rosencrantz will want to hear his colleague:

Terminal 1: Rosencrantz
>>> rosencrantz.bind('$.Actor.*')

So let guildenstern speak:

Terminal 3: Guildenstern

>>> guildenstern.send_msg(Message('$.Actor.Speak', 'Pssst!'))
MessageId(0, 5)
>>> # Remember guildenstern is also listening to '$.Actor.*'
>>> print guildenstern.read_next_msg()
<Announcement '$.Actor.Speak', id=[0:5], from=3, data='Pssst!'>

and rosencrantz hears:

Terminal 1: Rosencrantz

>>> print rosencrantz.read_next_msg()
<Announcement '$.Actor.Speak', id=[0:5], from=3, data='Pssst!'>

However, when we look to the audience, we see:

Terminal 2: Audience

We heard $.Actor.Speak Pssst!
We heard $.Actor.Speak Pssst!

This is because the audience has bound to the message twice - it is hearing it once because it asked to receive every $.Actor.Speak message, and again because it asked to hear any message matching $.Actor.*.

The solution is simple - ask not to hear the more specific version:

Terminal 2: Audience

<CTRL-C>
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
KeyboardInterrupt
>>> audience.unbind('$.Actor.Speak')
>>> while 1:
...    msg = audience.wait_for_msg()
...    print 'We heard', msg.from_, 'say', msg.name, msg.data
...

Note that we've also amended the printout to say who the message was from. Each Ksock connection has an id associated with it - for instance:

Terminal 1: Rosencrantz
>>> rosencrantz.ksock_id()
1L

and every message indicates who sent it, so:

Terminal 1: Rosencrantz

>>> print 'I heard', message.from_, 'say', message.name, message.data
I heard 3 say $.Actor.Speak Pssst!

We've shown that KBUS allows one to "announce" (or, less politely, "shout") messages, but KBUS also supports asking questions. Thus:

Terminal 3: Guildenstern

>>> guildenstern.bind('$.Actor.Guildenstern.query', True)

allows Guildenstern to bind to this new message name as a Replier.

(Only one person may be bound as Replier for a particular message name at any one time, so that it is unambiguous who is expected to do the replying.

Also, if a Sender tries to send a Request, but no-one has bound to that message name as a Replier, then an error is raised (contrast that with ordinary messages, where if no-one is listening, the message just gets ignored).)

If Rosencrantz then sends a Request of that name:

Terminal 1: Rosencrantz

>>> req = Request('$.Actor.Guildenstern.query', 'Were you speaking to me?')
>>> rosencrantz.send_msg(req)
MessageId(0, 6)

Guildenstern can receive it:

Terminal 3: Guildenstern

>>> msg2 = guildenstern.read_next_msg()
>>> print 'I heard', msg2
I heard <Request '$.Actor.Guildenstern.query', id=[0:6], from=1, flags=0x3 (REQ,YOU), data='Were you speaking to me?'>
>>> msg3 = guildenstern.read_next_msg()
>>> print msg3
<Request '$.Actor.Guildenstern.query', id=[0:6], from=1, flags=0x1 (REQ), data='Were you speaking to me?'>

As we should expect, guildenstern is getting the message twice, once because he has bound as a listener to '$.Actor.*', and once because he is bound as a Replier to this specific message.

(There is, in fact, a way to ask KBUS to only deliver one copy of a given message, and if guildenstern had used that, he would only have received the Request that was marked for him to answer. I'm still a little undecided how often this mechanism should be used, though.)

Looking at the two messages, the first is the Request specifically to guildenstern, which he is meant to answer:

Terminal 3: Guildenstern
>>> print msg2.wants_us_to_reply()
True

(and that is what the YOU in the flags means).

And rosencrantz himself will also have received a copy:

Terminal 1: Rosencrantz

>>> print rosencrantz.read_next_msg()
<Request '$.Actor.Guildenstern.query', id=[0:6], from=1, flags=0x1 (REQ), data='Were you speaking to me?'>

Guildenstern can then reply:

Terminal 3: Guildenstern

>>> reply = reply_to(msg2, 'Yes, I was')
>>> print reply
<Reply '$.Actor.Guildenstern.query', to=1, in_reply_to=[0:6], data='Yes, I was'>
>>> guildenstern.send_msg(reply)
MessageId(0, 7)

The reply_to convenience function crafts a new Reply message, with the various message parts set in an appropriate manner. And thus:

Terminal 1: Rosencrantz

>>> rep = rosencrantz.read_next_msg()
>>> print 'I heard', rep.from_, 'say', rep.name, rep.data
I heard 3 say $.Actor.Guildenstern.query Yes, I was

Note that Rosencrantz didn't need to bind to this message to receive it - he will always get a Reply to any Request he sends (KBUS goes to some lengths to guarantee this, so that even if Guildenstern closes his Ksock, it will generate a "gone away" message for him).

And, of course:

Terminal 2: Audience

We heard 1 say $.Actor.Guildenstern.query Were you speaking to me?
We heard 3 say $.Actor.Guildenstern.query Yes, I was

So, in summary:

To send or receive messages, a process opens a Ksock.
A process can send messages (be a Sender).
A process can bind to receive messages (be a Listener) by message name.
When binding to a message name, wildcards can be used.
When binding to a message name, a process can say it wants to receive Requests with that name (be a Replier)
It is not an error to send an ordinary message if no-one is listening.
It is an error to send a Request if there is no Replier.
There can only be one Replier for a given message name.
There can be any number of Listeners for a given message name.

Note

Running the examples in this introduction requires having the KBUS kernel module installed. If this is not already done, and you have the KBUS sources, then cd to the kernel module directory (i.e., kbus in the sources) and do:

make
make rules
sudo insmod kbus.ko

When you've finished the examples, you can remove the kernel module again with:

sudo rmmod kbus.ko

The message ids shown in the examples are correct if you've just installed the kernel module - the second number in each message id will be different (although always ascending) otherwise.

Small KBUS utility

2010-01-21T11:27:00.000Z

Just committed errno.py to kbus/utils - this is a simple utility to conver Unix error numbers into their corresponding mnemonic and text, or vice versa:

 $ errno.py EPERM
EPERM is error 1 (0x1): Operation not permitted

but also, if the error is one used by KBUS, to give the appropriate text from the KBUS documentation:

$ errno.py 32
Error 32 (0x20) is EPIPE: Broken pipe

KBUS:
On attempting to send *to* a specific replier, the replier with that id is no
longer bound to the given message's name.

because I can never remember which magic number corresponds to what particular usage.

Just before I leave for Christmas ..

2009-12-24T18:40:00.004Z

There should probably have been more of these updates, but we've been a bit busy here and so the blog hasn't been getting the attention it deserves. That'll hopefully be rectified soonish.

Anyway, the news in brief:

Advanced STB: we have a prototype OMAP3530-based STB design for which we've got boards back now, based on the ever-popular Beagleboard - SD only, but if I can fire up the DSP we should at least be able to decode SD versions of all the obscure codecs that no-one's ever heard of. Now all we have to do is write the firmware.. (and work out how the HD version will work).
USB IR widgets - would have been on sale except that someone's gone and bought all my spares. So: if you want one, do comment here or drop me a line and I'll get one sent out as soon as the next batch is ready. They'll hopefully be available via convenient web shop early in the New Year.
I've recently been doing quite a bit of work with GWT and Hadoop; both seem fairly sane and sensible. Something of an odd feeling to be back in a development regime not dominated by the cost of bugs - you start to realise that project planning does work for some people, somewhere.
Site of the day: http://www.h265.net . The idea of H.264 performance / 3 frankly scares me - especially in 3D. Time for those parallel-decode assurance flags people have been muttering about.
tibs has been doing some work on muddle, with the result that it's now faster, better, shinier, etc. Domains now work in a meaningful way and I've now churned out enough build trees that I'm pretty confident it's not going to fall over on anyone immediately.

Anyway, a good Christmas and New Year to everyone (anyone?) reading this; I'm off for a few days' relaxation (see, I told you I would one day..).

maintaining muddle feels a bit like being davros

2009-11-15T19:11:00.004Z

As a result of some recent changes to muddle - mainly tibs's code - you can now merge two build trees (which we call domains) into a single overarching build tree.
I've now added a couple of bits (check out r243) and I have a bit of a port of an ASTB build tree and a client's STB stack running.
With the addition of a couple of simple hooks on each side, I can write 100 lines of python which automatically checks out two entirely unrelated build trees, merges them together, builds them both (on demand, checking out code from various repositories in git, svn, and bzr, and supplying the right kernel includes across the boundary) and then merges the resulting filesystems. On demand.
It's not finished yet, but it's quite impressive to watch it go.

USB widgets are go (part 2)

2009-10-22T12:17:00.005+01:00

Well, we're getting there on the USB IR widgets - we now have Creactive doing some industrial design for us on the casings and the second spin of prototype PCBs is at the shop - back next week, hopefully.

Our core consumer functions are as they always were - will work with any RC5 remote control, programmable via USB serial port, no install on any major platform. As a result of having sourced some cheap remote controls, we'll also be supporting a proprietory NEC remote protocol and selling moderately cheap dedicated remotes to anyone who wants them (to control your model railway, for example).

We've managed to cram on a few new features for the geeks this time:

Boards can be made using 3.3V or 5V parts.
SOT-23 LDO so you can power the boards from somewhat unregulated supplies - eg. to use them without being plugged into a USB port.
UART, I2C and SPI brought out so you can use them as bridges (not all at the same time, though).
More I/Os brought out so you can have ADC inputs and control pins.

And we should be able to open-source both the schematic and the software, though as the software is still encumbered by Microchip's licence for the USB code it sadly can't be GPL.

Oh, we've lopped a couple of millimetres off the size in each direction too :-).

Anyway, watch this space and do get in touch if interested - early product should be available soon and if you ask us nicely we'll sell you some of the pre-production batch.

2009-10-21T20:32:00.006+01:00

Grr. Apologies if the code examples in the previous post are truncated at the right - this is because of the way blogger tries so hard to make the blog a nice column down the middle of your browser. Which is a laudable aim for a plain text article, but doesn't work so well for code examples. If I have time I'll look into it (some more) in the future (I've already tweaked the blog settings to try to make previous posts look sensible).

Update: OK, I think I've fixed that. Now on to trying to make "cut" work in posts...

And in a serious moment of serendipity, I've just discovered that Blogger have now added the ability to do "cut" posts, just as I wanted, by (a) enabling the new editor interface (done) and (b) putting:

<-- more -->

in the appropriate place in the post. So I've gone back and done that to my muddle posts, and they now don't dominate the entire blog. Which is good.

muddle: Cross-compiling V8

2009-10-21T20:30:00.004+01:00

So, I'm transferring a native (Intel) build of various things to a cross-compiled build for the ARM (specifically, for my beagleboard).

Today's tasks were sqlite (which was harder than I expected) and V8, Google's open source JavaScript engine.

V8 is built using scons, which it turns out is not particularly well aimed at the cross compilation fans amongst us.

It turns out that you can set environment variables (such as CC) to tell scons which compiler to use, but of course you need to know all the calls to the toolchain it is going to make, in order to catch them all. The easiest way to do that seems to be to run scons -n (i.e., tell me what commands you'll obey, but don't actually do them), which does work, but means that if you update the scons build environment, and coincidentally the script mutates to use new parts of the toolchain, they won't be covered. Which is rather awful.

Note
I'm rather hoping I've missed something here, and there's actually a better way of doing this. Although any way that includes editing the scons script supplied by the original package won't count as "better".

Anyway, it turns out that the muddle way to build V8 is fairly simple - a small bit of muddle code, and a simple helper Makefile.

First, the muddle code:

# Make sure our host has scons available
aptget.simple(builder, 'platform_host_tools', role, ['scons'])

# Set up the cross compilation environment variables
# - we retrieve the environment that will be applied to all packages
# and add the environment variables to it
label = label_from_string('package:*{*}/*')
env = builder.invocation.get_environment_for(label)
env.set("MUDDLE_CROSS_COMPILE", "/opt/codesourcery/arm-2008q3/bin/arm-none-linux-gnueabi-")
env.set("MUDDLE_ARCH", "arm")

# For reasons to do with the particular system I'm building, the V8
# sources get checked out into 'src/platform/v8'. There's nothing
# special about revision '3108' - it's just today's revision
muddled.checkouts.twolevel.absolute(builder,
        co_dir="platform",
        co_name="v8",
        repo_url="svn+http://v8.googlecode.com/svn/trunk",
        rev="3106")

# I then have an out-of-tree Makefile to build V8 in our context,
# in 'src/helpers/v8'
make.medium(builder,
            name = "v8",          # package name
            roles = [role],
            checkout = 'helpers',
            makefileName = os.path.join("v8","Makefile.muddle"))

# And we can't build the package until we've checked its files out
muddled.pkg.package_depends_on_checkout(builder.invocation.ruleset,
                                        "v8",     # this package
                                        role,     # in this role
                                        "v8",     # depends on this checkout
                                        None)

It looks as if I'm going to standardise on the cross compilation environment variable names as given, and I'm using the technique often enough that the rather clumsy mechanism for setting them will probably get streamlined fairly soon.

The only other thing that is needed, then, is the Makefile, src/helpers/v8/Makefile.muddle.

Note
We started out calling the helpers directory builders (because it has stuff used to build other things), but I find it irritating having to discriminate between src/builds and src/builders, so I'm trying out the new name to see if it works.

The original of this makefile was not cross-platform. I believe this version should work in that situation as well.

# Muddle makefile to control v8 builds

V8_SRC=$(shell $(MUDDLE) query dir "checkout:v8{*}/*")

ifneq ($(MUDDLE_ARCH),)
        FLAGS=arch=$(MUDDLE_ARCH)
        ENV=CXX=$(MUDDLE_CROSS_COMPILE)g++ CC=$(MUDDLE_CROSS_COMPILE)gcc
else
        FLAGS=
        ENV=
endif

SCONS_CMD=scons -C . -f $(V8_SRC)/SConstruct -Y $(V8_SRC)

all:
        (cd $(MUDDLE_OBJ_OBJ); $(ENV) $(SCONS_CMD) $(FLAGS) library=shared mode=release)
        (cd $(MUDDLE_OBJ_OBJ); $(ENV) $(SCONS_CMD) $(FLAGS) library=shared mode=debug)

config:
        -rm -rf $(MUDDLE_OBJ)
        -mkdir -p $(MUDDLE_OBJ_OBJ)

install:
        -mkdir -p $(MUDDLE_OBJ_INCLUDE)/v8
        -mkdir -p $(MUDDLE_OBJ_LIB)
        -mkdir -p $(MUDDLE_INSTALL)/usr/lib

        install -m 0644 $(V8_SRC)/include/v8.h       $(MUDDLE_OBJ_INCLUDE)/v8/v8.h
        install -m 0644 $(V8_SRC)/include/v8-debug.h $(MUDDLE_OBJ_INCLUDE)/v8/v8-debug.h
        install -m 0755 $(MUDDLE_OBJ_OBJ)/libv8.so   $(MUDDLE_OBJ_LIB)/libv8.so
        install -m 0755 $(MUDDLE_OBJ_OBJ)/libv8_g.so $(MUDDLE_OBJ_LIB)/libv8_g.so
        install -m 0755 $(MUDDLE_OBJ_OBJ)/libv8.so   $(MUDDLE_INSTALL)/usr/lib/libv8.so
        install -m 0755 $(MUDDLE_OBJ_OBJ)/libv8_g.so $(MUDDLE_INSTALL)/usr/lib/libv8_g.so
        echo INSTALL=$(MUDDLE_INSTALL)

clean:
        (cd $(MUDDLE_OBJ_OBJ); $(SCONS_CMD) --clean)

distclean:
        -rm -rf $(MUDDLE_OBJ)

The scons switches used are:

-C <directory> -- cd to this directory before building

-f <path> -- the given <path> is the scons build script

-Y <directory> -- look for import files in the given directory (the scons documentation talks about this as specifying the repository path, but that sounds like a use of the same term for a different purpose)

The disposal of the various resultant files within the muddle build tree matches the Intel build.

There are some caveats; notably that I've checked that the build produces ARM files, but haven't yet had a chance to test them, and the Makefile could clearly do with some tidying up -- for a start, it's clearly very clumsy to do cd <somewhere>; scons -C . instead of just doing scons -C <somewhere>.

We're hiring!

2009-10-16T13:46:00.004+01:00

In case you hadn't heard:

Kynesim is urgently in need of developers to expand our team.
We're a small, friendly technology consultancy firm located on Castle Hill, Cambridge with strong links to the University of Cambridge, for which our Managing Director routinely teaches.
We provide top flight hardware and software development services to a variety of clients; mostly doing STB work, embedded Linux (although WinCE expertise welcomed), quite a bit of wireless/embedded micro stuff, but we have been known to do everything from soldering bits onto PICs to writing web applications. We have an outstanding track record of delivering complex and challenging technology on tight deadlines.
Code is mostly in C and C++ with a smattering of assembler, Python and Java and a surprising amount of Javascript and HTML. C# is hovering threateningly on the horizon. A good working knowledge of C and/or C++ is essential.
Starting as soon as possible, this role would suit Computer Science or Maths graduates. We aim to recruit bright, hard working people willing to muck in with challenging tasks they may not have met before.
In return, we provide a decent salary, free food including a pub lunch on Mondays, interesting work, a Playstation 3, proper flexitime and occasional soft toys.
Sound fun? Contact Richard Watts for further details.

The FOSS community finally gets a dose of China ..

2009-09-25T12:20:00.004+01:00

I notice that the FOSS people are finally being bitten by the Chinese problem. The rest of us in CE have, of course, had this problem for years - there are manufacturers out there whose business model is to buy a new product, suck the software image out, reverse-engineer the hardware and then sell their own cut-price version - and more OEMs who do this on the side, selling off-label cut-price versions of the products they make for western consumers to the generic domestic market, from where they naturally turn up here.

There's no effective way to stop this short of tivoisation, which is why it's such a popular tactic and why GPL3 is particularly invidious in CE: none of us want to tivoise our products - it's stupid, time-consuming and difficult (plus we want to hack them ourselves). But if we don't, we get maybe two months' grace before our software gets knocked off and the product driven off the market.

This is a natural side-effect of Asian manufacturing economics: anything you make in the UK will always be more expensive than in the far east, because the far east has much lower regulatory compliance costs - they're allowed to run machines that no-one in the UK would countenance, under lax employment law and with no IP enforcement to speak of.

So perhaps the free software community will be a little more forgiving to those of us who lock them out of our products in future; we're really not that evil (well, some of us aren't) - just trying to get some sort of return on our investment before we're instantly outcompeted.

Marketing

2009-09-16T15:22:00.002+01:00

At a team discussion yesterday, we realised the marketing potential of my cast. This is what we came up with.

IR Widget - the next step

2009-09-10T16:57:00.003+01:00

Hello all,

Just a quick update on the IR Widget.

Trying to control games consoles has proved a little difficult - it appears that not all of the controls for video playback on the PS3 (for example fast-forward, rewind) can come from a keyboard. The only available controls are: bring up a menu, navigate around with the arrow keys and select the appropriate option. The situation is similar on the XBox 360. While this is functional, it's not so nice to use - we want now for the widget to pretend to be a games controller as well as a keyboard, as that has buttons to do what we want. However, this could take a little while to implement due to memory constraints. Richard has offered to solve this with some refactoring and buffer sharing tricks.

I have now added some functionality to allow us to send modifier keys (ctrl, alt etc) which it turns out are essential for some of the media controls in Linux.

I have also now made the programming interface work under OS X, although Windows is a little more fussy - I'm still messing around with INF files.

Work still in progress, so keep checking back!

muddle: Linux on the Beagleboard (part 0)

2009-09-09T20:00:00.003+01:00

I've been playing with a beagleboard at work, as part of the initial investigation towards an advance STB (set top box). It's based around TI OMAP technology (so it's got an ARM with a DSP on the side).

Anyway, my initial investigation put Android on it (in a fairly dumb manner), and I've spent the last couple of days getting the head-of-tree OMAP Linux to boot on it, with busybox. This last I hope, when it's a little more polished, to use as a replacement for the "simple linux" example in the muddle distribution.

Anyway, here as a teaser is today's version of the build description for OMAP Linux on a beagleboard:

#! /usr/bin/env python

"""Muddle build description for a "plain" OMAP build on the Beagleboard.

At least initially, we're trying to build a Linux+busybox system.

The role we're providing is 'omap'.
"""

import os

import muddled.deployments.filedep as filedep
import muddled.pkgs.aptget as aptget
import muddled.checkouts.simple
import muddled.depend
from muddled.depend import label_from_string
import muddled.pkgs.make as make

import wget

def describe_to(builder):

    role = 'omap'
    roles = ['omap']

    builder.invocation.add_default_role(role)

    # filedep.deploy(builder, target_dir, name, roles)
    #
    # Register a file deployment.
    #
    # The deployment will take the roles specified in the role list, and build
    # them into a deployment at deploy/[name].
    #
    # The deployment should eventually be located at target_dir.

    filedep.deploy(builder,
                   '/',
                   'omap',
                   roles)

    # There's a variety of things we need on (this, the host) system
    # in order to build - I hope I've got this right (difficult to tell
    # as I already have some stuff installed on *my* development system)
    aptget.simple(builder, 'development_packages', role,
            ['zlib1g-dev', 'uboot-mkimage'])

    # According to http://omappedia.org/wiki/LinuxOMAP_Kernel_Project, we get
    # our OMAP kernel from:
    muddled.checkouts.simple.absolute(builder, 'omap_kernel',
            'git+git://git.kernel.org/pub/scm/linux/kernel/git/tmlind/linux-omap-2.6.git')

    # We'll aim to make that with an out-of-tree makefile
    # We could make a tailored subclass of muddled.pkgs.linux_kernel, and use
    # that to build our kernel. I may still do that once I've figured out how
    # it is different (one notable change is we're building uImage instead of
    # zImage). For now, it's probably easier to have a Makefile.muddle
    make.medium(builder,
                name = "omap_kernel",    # package name
                roles = roles,
                checkout = "builders",
                deps = [],
                makefileName = os.path.join("omap_kernel","Makefile.muddle"))

    muddled.pkg.package_depends_on_checkout(builder.invocation.ruleset,
                                    "omap_kernel",  # this package
                                    role,           # in this role
                                    "omap_kernel",  # depends on this checkout
                                    None)

    # On top of that, we want to build busybox
    #
    # According to the busybox website, we can retrieve sources via git:
    #
    # To grab a copy of the BusyBox repository using anonymous git access::
    #
    #   git clone git://busybox.net/busybox.git
    #
    # Once you have the repository, stable branches can be checked out by
    # doing::
    #
    #   git checkout remotes/origin/1_NN_stable
    #
    # Once you've checked out a copy of the source tree, you can update your
    # source tree at any time so it is in sync with the latest and greatest by
    # entering your BusyBox directory and running the command::
    #
    #   git pull

    # So, for the moment, at least, let's go with the latest from source
    # control (when we're finalising this, we're maybe better identifying
    # a particular release to stick to)
    muddled.checkouts.simple.absolute(builder, 'busybox',
            'git+git://busybox.net/busybox.git')

    # We'll aim to make that with an out-of-tree makefile
    #
    # 'deps' is a list of package names, which our make depends on.
    #
    # Specifically, for each <name> in 'deps', and for each <role> in 'roles',
    # we will depend on "package:<name>{<role>}/postinstalled"
    make.medium(builder,
                name = "busybox",    # package name
                roles = roles,
                checkout = "builders",
                deps = [ 'omap_kernel' ],
                makefileName = os.path.join("busybox","Makefile.muddle"))

    # And we also depend on having actually checked out busybox
    muddled.pkg.package_depends_on_checkout(builder.invocation.ruleset,
                                    "busybox",      # this package
                                    role,           # in this role
                                    "busybox",      # depends on this checkout
                                    None)

    # Can we use the same bootloader and such that we already had for the
    # android build? I don't see why not.

    # The bootloader and related items, which go into the FAT32 partition on
    # the flash card, are retrieved from the net (eventually, I hope we'll be
    # building u-boot, but for now the binary should do)
    muddled.checkouts.simple.absolute(builder, 'MLO',
            'wget+http://free-electrons.com/pub/demos/beagleboard/android/MLO')
    muddled.checkouts.simple.absolute(builder, 'u-boot',
            'wget+http://free-electrons.com/pub/demos/beagleboard/android/u-boot.bin')

    # We need some way of getting them installed - let's foreshadow the day when
    # we actually want to build u-boot ourselves, and pretend
    make.medium(builder,
                name = "u-boot",    # package name
                roles = roles,
                checkout = "builders",
                deps = [],
                makefileName = os.path.join("u-boot","Makefile.muddle"))
    muddled.pkg.package_depends_on_checkout(builder.invocation.ruleset,
                                    "u-boot",       # this package
                                    role,           # in this role
                                    "u-boot",       # depends on this checkout
                                    None)
    # Oh, and this one as well...
    rule = muddled.depend.depend_one(None,
                              label_from_string('package:u-boot/installed'),
                              label_from_string('checkout:MLO/checked_out'))
    builder.invocation.ruleset.add(rule)

    # And, of course, we need (the rest of) our Linux filesystem
    muddled.checkouts.simple.absolute(builder, 'rootfs',
            'bzr+ssh://bzr@palmera.c.kynesim.co.uk//opt/kynesim/projects/052/rootfs')
    make.simple(builder, 'rootfs', role, 'rootfs', config=False,
            makefileName='Makefile.muddle')

    # But we depend on busybox to have installed the various binaries first
    rule = muddled.depend.depend_one(None,
                              label_from_string('package:rootfs/installed'),
                              label_from_string('package:busybox/installed'))
    builder.invocation.ruleset.add(rule)

    # Deploy all our roles
    builder.by_default_deploy_list(roles)

I'm fairly sure there are some parts of that which could be a lot nicer, and it doesn't make entire sense without some extra infrastructure, but I'm already please with how compact the code I need to build the system is, compare to how I'd have done it before.

I shall try to blog some more at a later date with some explanations of how the whole thing goes together...

Handy hints for debugging optimised assembly, number 2431

2009-09-08T19:40:00.002+01:00

If you're trying to see what effect your optimisations have on assembly code, a useful hint is often to put volatile assembler labels at strategic points in your code.

asm volatile ("foo1:");

in GNU C/C++ source will turn into something like:

#APP
# 667 "src/iatools.cpp" 1
 foo1:
# 0 "" 2
#NO_APP

in the assembler. This may have some slightly adverse effects on the optimiser (though not too many - labels have no side-effects), but it means you can bracket loops you're trying to optimise quite successfully without having to read the runes to work out which bits of assembler correspond to which bits of C.

USB, meet Infra-red

2009-09-03T13:11:00.005+01:00

Been looking at this for a while now, and we've finally got something working.

This little widget will allow us to use an infra-red remote to control media playback (or do pretty much anything else) on pretty much any PC without the need for additional drivers (so long as it can support a USB keyboard).

It is programmable over USB using a simple command line interface allowing the user to configure it on a per-button basis. Currently working on configurations for controlling a PS3 for video playback.

More work being done, so watch this space!

XUL ain't so bad after all ..

2009-08-29T18:28:00.000+01:00

I've been doing quite a bit of XUL hacking lately and I think it's actually beginning to make sense. The main problems seem to be a lack of documentation and Javascript's usual habit of trying to continue in the face of overwhelming odds which means you don't get told about errors at the point of error, but some unspecified time later.

In particular, it took me a while to work out that if you set your browser.chromeURL property to XUL without a browser marked as type=content-primary, your entire XUL gets replaced with the target window content. Any linked Javascript then runs twice - once just before the XUL gets replaced, tricking you into believing that your XUL is fine - and then again afterward for some reason.

There's also a slightly odd interaction of command-line arguments with windows: the primary window gets window.arguments[0], subsequent window.open()'d ones don't and if your content is doing the open, you need to resort to slightly nasty tricks like setting properties in window listeners. Ugh.

In other news, I'll be at ESC UK in October and at IBC on September 14th and 15th so hope to run into people there.

2009-08-28T10:00:00.000+01:00

In case you hadn't guessed, I like muddle. I was formally convinced (i.e., by practice rather than theory) a few weeks ago, when I had to work on a system that already had a muddle description (in fact, the same system into which "george" was being incorporated). With muddle, it was quick and easy to add new software to the system, trivial to update all the packages (muddle update is your friend for updating all the packages in a system from their individual revision control systems - having worked with a "build" system that didn't have this facility, that is so refreshing), and easy to rebuild individual parts. If we hadn't had muddle for this system, we'd still have had to build Makefiles or some other ad-hoc scripting mechanism to make the builds reproducible (been there, done that, ick). With muddle, that step can be avoided (it saves time, in deciding what to do), and the same mechanism can be used for multiple projects (again, it saves time, in learning an individual setup).

(There's a lot to be said for a solution that is ready-made, and avoids the need for deciding what solution to use this time - it's where ASCII, UML and XML have all scored in their time. It's not that the solution need be perfect, just that it be a decent solution that you could have chosen independently, and that everyone is already predisposed to agree on. Such can save months of discussion and reimplementing the wheel.)

Using file-system based tags is useful, too. If you look in the .muddle/tags/packages directories, you'll find the status of each package made visible. Delete one of the tag files, and muddle will believe you that that step is undone (so deleting all of them is one way of making it rebuild a package from scratch). Since checkout and deployment are intrinsically separate from the normal building process, they have their own tags, outside the above. And that's the right thing, too. Perhaps the biggest lesson, though, is to keep going back to the muddle README (this is the first section in the sphinx-ified documentation). It's a good read (it's what made me think muddle2 was really on the right track), but it is also quite information dense - almost everything in there is stuff that you will want to refer back to.

Adding a new program to a muddle build description

2009-08-28T09:59:00.003+01:00

So I have a program (which I shall, for the sake of argument, call "george", because that is not its name) which relies upon ffmpeg and tstools, and which I want to incorporate into an existing muddle build.

This article describes how I did it. The resultant code is that used in the real system (except that the names have been changed).

I shall assume that you've read the muddle documentation, and in particular the Welcome to Muddle readme.

As described in the muddle documentation, the project I'm amending is laid out in the traditional manner:

+- .muddle/
+- src
|  +- builds/
|  |  +- 01.py
|  +- builders/
|  +- package1/
|  +- package2/
|  +- ...
+- obj/
+- install/
+- deploy/

where src/buids/01.py is the build description, and the various package1, etc., are existing software packages that are being built already.

(Naming the main build description 01.py appears already to be a tradition in muddle. It could, of course, be called something else, perhaps named after the project.)

The project I'm building is being deployed on a system similar to that I'm building on, so the same toolchain is being used, and I do not need to specify the compiler and so on. This also means that any configure scripts can configure against the system I'm building on, and get results that make sense on the target. This is unusually simple, but makes the examples a lot easier to read.

ffmpeg

For this program, we need a fairly recent version of ffmpeg, so we need to get it from the ffmpeg site. We are not, however, particularly interested (at this stage) in tracking future changes to the library.

ffmpeg is available as spot releases, via SVN, or via GIT. Since the project I'm adding to uses GIT to store its checkouts, it makes sense to use the GIT version of ffmpeg, and thus I can just git clone ffmpeg (and libswscale) and add them to the project's repository directly. This means we'll be working with a local checkout, rather than a remote one, which will also speed up the process of checking out when doing a build from scratch.

That leaves us with src/ffmpeg.

muddle prefers packages to build out-of-tree, so that it is possible to build for more than one target without needing to "clean" the source tree. ffmpeg supports this, so that's nice.

Since src/ffmpeg is taken directly from the outside world, we'd rather not add muddle specific build files, configuration files, etc., to that directory. The convention for muddle is to have a separate builders directory containing muddle specific infrastructure for an external package - so, in this case:

src/builders/ffmpeg/Makefile.muddle

tells muddle how to build ffmpeg. If we needed (for instance) an init.d script, or other support files, then this would be a natural place to keep them as well.

The code I need to add to the build script (builds/01.py) is then:

# ffmpeg
# - we have a GIT clone of ffmpeg's own repository in our repository already
muddled.checkouts.simple.relative(builder, "ffmpeg")

# - we have our own information on how to build it
make.medium(builder,
            name = "ffmpeg",        # package name
            roles = ["project"],
            checkout = "builders",
            deps = [],
            makefileName = os.path.join("ffmpeg","Makefile.muddle"))

# building ffmpeg requires the Makefile.muddle in builders/ffmpeg
muddled.pkg.package_depends_on_checkout(builder.invocation.ruleset,
                                        "ffmpeg", # This package
                                        "project",# in this role
                                        "ffmpeg", # depends on this checkout
                                        None)

In other words:

We are checking out ffmpeg from our own repository (so it is a "relative" checkout - an "absolute" checkout would need to specify an external repository).
We build package "ffmpeg" for the role "project" using the checkout "builders" (for simplicity I'm checking out all of the builders directory). This doesn't depend on anything else. We're going to use the named makefile.
(Note that we've earlier done:
```
import muddled.pkgs.make as make
```
in order to keep the code easier to read.)
As a shorthand, we tell muddle that that same package ("ffmpeg") in the given role depends on the checkout of ffmpeg - in other words we can't build the ffmpeg library until we've checked it out.
We could have done that last step directly by specifying a dependency in the deps argument to make.medium, but that would involve constructing the appropriate label (not actually that hard, but the "helper" function is probably slightly easier).

Makefile.muddle then looks like:

# Build ffmpeg.
# Ask muddle where the ffmpeg package has been checked out
FFMPEG_SRC=$(shell $(MUDDLE) query dir "checkout:ffmpeg{*}/*")

all: FORCE
        cd $(MUDDLE_OBJ_OBJ); make

# Do our configuration building in the MUDDLE_OBJ_OBJ directory. ffmpeg
# supports remote configuration building, by specifying the path to the
# configure script. It then leaves (a link to) the Makefile in that directory,
# so that we can "cd" there and build "directly".
#
# When configuring, say that our files will end up (when installed) in "/",
# because, on the target machine, they should.
config:
        -rm -rf $(MUDDLE_OBJ)
        -mkdir -p $(MUDDLE_OBJ_OBJ)
        cd $(MUDDLE_OBJ_OBJ); $(FFMPEG_SRC)/configure --prefix=/

# When installing, we're installing within the muddle infrastructrure,
# not onto the target machine. So we need to use DESTDIR to say "although
# you want to install into "/", pretend that "/" is rooted at MUDDLE_OBJ".
install:
        cd $(MUDDLE_OBJ_OBJ); make DESTDIR=$(MUDDLE_OBJ) install

distclean:
        -rm -rf $(MUDDLE_OBJ)

clean: FORCE
        make -C $(FFMPEG_SRC) clean

FORCE:

tsstools

The second package we need to build is tstools.

Unfortunately, tsools does not support building out-of-tree. The best solution to this would have been to fix it (particularly since I am the tstools maintainer), but for speed I didn't bother, and it allows an example of how one might work around this sort of problem.

Again, for simplicity, and because this project does not care about future changes to tstools (particularly if the build mechanism should change!), we take a copy of tstools into our own repository.

The build script code for tstools is very similar to that for ffmpeg:

# tstools
# - we have a copy of tstools' own repository in our repository already
muddled.checkouts.simple.relative(builder, "tstools")

# - we have our own information on how to build it
make.medium(builder,
            name = "tstools",       # package name
            roles = ["project"],
            checkout = "builders",
            deps = [],
            makefileName = os.path.join("tstools","Makefile.muddle"))

muddled.pkg.package_depends_on_checkout(builder.invocation.ruleset,
                                        "tstools",  # This package
                                        "project",  # in this role
                                        "tstools",  # depends on this checkout
                                        None)

So we once again have a Makefile.muddle, predictable in builders/tstools/Makefile.muddle:

# Build tstools.
# tstools doesn't currently support building out-of-tree, so we'll have to cope
# - primarily by cleaning before each build, and then copying the result out of tree

# Ask muddle where the tstools package has been checked out
TSTOOLS_SRC=$(shell $(MUDDLE) query dir "checkout:tstools{*}/*")/trunk

# Force a clean before our build in case someone else has built
# This *is* a pain, but is best solved by making tstools build out-of-tree
# for itself
all: FORCE
        make -C $(TSTOOLS_SRC) clean
        make -C $(TSTOOLS_SRC)

# We install the include files into MUDDLE_OBJ_INCLUDE. At the moment we
# aren't deploying them to the target system, but if we were we'd be
# putting them into one of the main "include" directories, which would
# be rather unfriendly. So that's worth bearing in mind...
install: all
        -mkdir -p $(MUDDLE_OBJ_LIB)
        -mkdir -p $(MUDDLE_OBJ_INCLUDE)
        cp $(TSTOOLS_SRC)/lib/* $(MUDDLE_OBJ_LIB)
        cp $(TSTOOLS_SRC)/*.h   $(MUDDLE_OBJ_INCLUDE)

config:
        # Nothing to do here

distclean:
        make -C $(TSTOOLS_SRC) distclean
        -rm -rf $(MUDDLE_OBJ)

clean: FORCE
        make -C $(TSTOOLS_SRC) clean

FORCE:

george

Finally, we have the program itself. This is the only thing we're actually going to want on the target system, since we're building against static libraries.

Since this is a program that is written for this system, and thus we don't need a separate checkout for the muddle specific Makefile, the muddle description code is very simple:

make.medium(builder,
            "george",
            ["project"],
            "george",
            ["ffmpeg", "tstools"],
            config = False,
            makefileName = "Makefile.muddle")

Specifically, we're building the package "george", for the "project" role, based on the checkout "george". By default this is a simple checkout, and thus we don't need to specify how to get "george". We depend on the two packages named, so they must already have been built. config=False means we don't need to do a configuration step (make configure).

We could have called the Makefile just Makefile, but it seemed clearer to me to make it specific that it is a muddle makefile, since muddle makefiles depend on the muddle infrastructure to work (to set all those special environment variables).

The makefile for "george" (which is in the src/george directory) is then:

# Build george
# MUDDLE_INCLUDE_DIRS is given to us as a list of the include directories
# we said we depended on.
# The incantation below says "For everything in MUDDLE_INC_DIRS, prefix '-I'"
INCS=$(MUDDLE_INCLUDE_DIRS:%=-I%)

# Similarly for MUDDLE_LIB_DIRS
LIBS=$(MUDDLE_LIB_DIRS:%=-L%)

FFMPEG_LIBS=-lavformat -lavcodec -lavutil -lswscale -lz -lbz2 -lm

TSTOOLS_LIBS=-ltstools

# Default to building in the current directory
O ?= $(CURDIR)

$(O)/george:        george.c
        $(CC) $(CCFLAGS) -o $@ george.c $(INCS) $(LIBS) $(FFMPEG_LIBS) $(TSTOOLS_LIBS)

install:
        cp $(O)/george $(MUDDLE_INSTALL)/bin

config:
        # Nothing to do

distclean: clean
        # Nothing to do

clean:
        rm -f $(O)/george

The MUDDLE_INCLUDE_DIRS and MUDDLE_LIB_DIRS magic is explained in the muddle documentation - it gives me a list of the include directories and library directories implied by the dependencies I've stated for this package (so if I get those wrong, that will show up here).

The $(O) business is building george out-of-tree, so that we can have several different builds of the same source at the same time.

Note that the install target is copying the "george" executable to $(MUDDLE_INSTALL)/bin.

The flow of built objects in muddle is, more or less:

src/ --compile--> obj/ --install--> install/ --deploy--> deploy/

Both ffmpeg and tstools were compiled and the results placed into obj/ subdirectories. Since they are not destined for the target system, however, they are not copied to install/.

"george" is to be put on the target system, so it is copied to install/.../bin -- the exact location depends upon the role, etc.

(Copying things from install/ to deploy/ is done by muddle automatically as part of the deployment phase, so we don't need to do anything special for that - again, see the muddle documentation.)

It is perhaps unfortunate that the Makefile target called install is used for both purposes, but that does fit with the normal make usages. Of course, if we were wanting ffmpeg libraries on the final system, we'd have to extend the install target in its Makefile.muddle to copy files from MUDDLE_OBJ to MUDDLE_INSTALL, but that's an example for another time.

Other things

Since muddle is dependency based, it can be useful to look at the dependencies. The main command to do this is:

$ muddle depend user

which shows all the build description dependencies, or (for instance):

$ muddle depend user 'package:ffmpeg{project}/*'

which shows all the dependencies for package "ffmpeg" in role "project", with any tag (nb: package "ffmpeg" is distinct from checkout "ffmpeg", which is separate and distinct - again, see the muddle documentation).

(muddled.depend.Label() can be used to construct a label from its parts, and thus may help to remember what those parts are.)

Meanwhile, as I iterated towards getting builders/ffmpeg/Makefile.muddle correct,

$ muddle distclean ffmpeg

was also my friend.

Finally, all of the information about the current build is kept under the .muddle directory. It is kept in very simple form, and the muddle command reads the information "as given". This means that if I get my system into a strange or broken situation, and want to clear muddle's memory of what has been done, I can simply delete the appropriate tags (which are just text files containing a timestamp) from the .muddle directory structure.

This turns out to be very useful when trying to develop a system, and is much easier to handle than the tangle one tends to get into with traditional (make based) build and deployment systems.

First post..

2009-08-06T23:21:00.000+01:00

Welcome to Kynesim's blog. This will be mostly technology with a side-order of comments on the UK tech business and the odd useful Linux hint. .. or you can read our official website.