muddle and its directories - part 6 - domains

Contents

• Domains
  • What domains are for
  • How domains work
  • A worked example of using domains
  • Version stamps and domains

Domains

Domains are part of advanced muddle use, and are probably not relevant to most builds. Regardless, it is probably worth reading at least What domains are for from this section, as useful background, and so that you can recognise when they might be applicable.

The way domains work follows naturally from the rest of muddle, but the implementation is still being tidied up - hence the muddle3_label branch.

What domains are for

Domains allow one to include one build description inside another, in much the same way as a Python module can import another.

A simple example might be when building a system with WebKit and X11 support. We might have two builds, constructed as follows:

def UI_Build:
   import WebKit_build
       import X11_build

and:

def Login_build:
    import X11_build

Alternatively, we might have some high-level software for handling internet television, and want to build two systems on different architectures. In this case, our contrasting builds might be:

def IPTV_stack:
    import ARM_based_stack

and:

def IPTV_stack:
    import MIPS_based_stack

How domains work

A build includes another build (a subdomain) using the include_domain() function. This takes the same arguments as the muddle init command (that is, a repository specification and a build description therein), plus a name for the domain.

The following sequence of actions is then performed:

1. Create a directory called domains/<name>/, where <name> is the name of the new domain.
2. cd into that new directory, and perform (the equivalent of) a muddle init command therein, using the repository specification and build description given.
3. Read in this new build, giving a stand-alone build tree datastructure.
4. Find all of the labels in the new build tree datastructures, and change them to add the domain name. So, for instance, checkout:fred/checked_out would become checkout:(<name>)fred/checked_out.
5. Incorporate this amended build tree datastructure into the original ("top level") build.
6. Create a .muddle/am_subdomain file in the domain. This allows muddle to distinguish the actual (final) top-level build from any subdomains within it, which is a useful optimisation.

Note that the build description for a domain does not itself know that it is not the top-level of a muddle build. Indeed, this is an important property of domains - it means that any build description can potentially be included in any other.

For most purposes, the subdomain works just as if it were a "normal" top-level build. In particular, it uses its own .muddle/ directory to record its build state, and writes things to its own obj/ and install/ directories.

Also note that, as a consequence of the way domains work (and this is a good thing), it is only possible for a build to refer to labels in subdomains, not to those in sibling or parent build trees.

To allow for more flexibility in how domains are used, there are tools in the muddled package for manipulating the merged label-space. For instance, if one is using subdomains that originated as two different set-top-box builds, both might provide a linux kernel, and the top-level build probably wants to amend the labels such that both subdomains use one of the kernel packages (broadly, saying "this label should be used instead of that label").

A worked example of using domains

So, we shall start in a new empty directory:

$ cd ..
$ mkdir example2
$ cd example2

and start with the same build that we used before:

$ m3 init svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio builds/01.py
> Make directory /home/tibs/sw/m3/example2/.muddle
Initialised build tree in /home/tibs/sw/m3/example2
Repository: svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
Build description: builds/01.py


Checking out build description ..

> Make directory /home/tibs/sw/m3/example2/src
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/builds builds
A    builds/01.py
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example2/.muddle/tags/checkout/builds
Done.

So, just as before, we now have the following directory structure:

.
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       `-- checkout/
|           `-- builds/
|               `-- checked_out
`-- src/
    `-- builds/
        |-- 01.py
        `-- 01.pyc

We can now edit the src/builds/01.py file to include a subdomain as part of the build. As described above, we need to specify how to retrieve the domain, giving the same information as for the "muddle init" command. We shall call our domain b (after the example we're taking it from):

from muddled.mechanics import include_domain
include_domain(builder,
               domain_name = "b",
               domain_repo = "svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b",
               domain_desc = "builds/01.py")

We also have to say how to include the result into our deployment - a simple way to do this is:

import muddled.deployments.collect as collect
collect.deploy(builder, "everything")
collect.copy_from_role_install(builder, "everything",
                               role = "x86",
                               rel = "", dest = "",
                               domain = None)
collect.copy_from_role_install(builder, "everything",
                               role = "x86",
                               rel = "", dest = "usr",
                               domain = "b")

and then deploy the new deployment "everything", instead of the old "cpio_dep". Note that we're not deploying to a CPIO archive this time, but just as normal files.

We should probably also change the introductory comment to:

# An example of building with a subdomain

So the "top level" build description is now:

#!  /usr/bin/env python
#
# An example of building with a subdomain

import muddled
import muddled.pkgs.make
import muddled.deployments.cpio
import muddled.checkouts.simple
import muddled.deployments.collect as collect
from muddled.mechanics import include_domain

def describe_to(builder):
    # Checkout ..
    muddled.checkouts.simple.relative(builder, "cpio_co")
    muddled.pkgs.make.simple(builder, "pkg_cpio", "x86", "cpio_co")

    include_domain(builder,
                   domain_name = "b",
                   domain_repo = "svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b",
                   domain_desc = "builds/01.py")

    collect.deploy(builder, "everything")
    collect.copy_from_role_install(builder, "everything",
                                   role = "x86",
                                   rel = "", dest = "",
                                   domain = None)
    collect.copy_from_role_install(builder, "everything",
                                   role = "x86",
                                   rel = "", dest = "usr",
                                   domain = "b")

    builder.invocation.add_default_role("x86")
    builder.by_default_deploy("everything")

# End file.

If we now do a "checkout _all":

$ m3 checkout _all
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle
Initialised build tree in /home/tibs/sw/m3/example3/domains/b
Repository: svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b
Build description: builds/01.py


Checking out build description ..

> Make directory /home/tibs/sw/m3/example3/domains/b/src
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/b/builds builds
A    builds/01.py
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle/tags/checkout/builds
There is no rule to build label checkout:b_co/checked_out
> Building checkout:cpio_co/checked_out
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio/cpio_co cpio_co
A    cpio_co/hello_world.c
A    cpio_co/Makefile
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/checkout/cpio_co

giving us:

$ ls -AF
domains/  .muddle/  src/

or, in more detail (omitting .svn directories):

.
|-- domains/
|   `-- b/
|       |-- .muddle/
|       |   |-- am_subdomain
|       |   |-- Description
|       |   |-- RootRepository
|       |   `-- tags/
|       |       `-- checkout/
|       |           `-- builds/
|       |               `-- checked_out
|       `-- src/
|           `-- builds/
|               |-- 01.py
|               `-- 01.pyc
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       `-- checkout/
|           |-- builds/
|           |   `-- checked_out
|           `-- cpio_co/
|               `-- checked_out
`-- src/
    |-- builds/
    |   |-- 01.py
    |   `-- 01.pyc
    `-- cpio_co/
        |-- hello_world.c
        `-- Makefile

Apart from the am_subdomain file in its .muddle/ directory, the domains/b/ directory looks like a perfectly normal build.

If we then do:

$ m3 build _all
Building package:(b)pkg_b{x86}/postinstalled package:pkg_cpio{x86}/postinstalled
> Building checkout:(b)b_co/checked_out
> svn checkout  http://muddle.googlecode.com/svn/trunk/muddle/examples/b/b_co b_co
A    b_co/hello_world.c
A    b_co/Makefile
Checked out revision 459.
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle/tags/checkout/b_co
> Building package:(b)pkg_b{x86}/preconfig
> Make directory /home/tibs/sw/m3/example3/domains/b/obj/pkg_b/x86
> Make directory /home/tibs/sw/m3/example3/domains/b/install/x86
> Make directory /home/tibs/sw/m3/example3/domains/b/.muddle/tags/package/pkg_b
> Building package:(b)pkg_b{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:(b)pkg_b{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example3/domains/b/obj/pkg_b/x86/hello_world hello_world.c
> Building package:(b)pkg_b{x86}/installed
> make  -f Makefile install
install -m 0755 /home/tibs/sw/m3/example3/domains/b/obj/pkg_b/x86/hello_world /home/tibs/sw/m3/example3/domains/b/install/x86/hello_world
> Building package:(b)pkg_b{x86}/postinstalled
> Building package:pkg_cpio{x86}/preconfig
> Make directory /home/tibs/sw/m3/example3/obj/pkg_cpio/x86
> Make directory /home/tibs/sw/m3/example3/install/x86
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/package/pkg_cpio
> Building package:pkg_cpio{x86}/configured
> make  -f Makefile config
Nothing to do
> Building package:pkg_cpio{x86}/built
> make  -f Makefile
cc -o /home/tibs/sw/m3/example3/obj/pkg_cpio/x86/hello_world hello_world.c
> Building package:pkg_cpio{x86}/installed
> make  -f Makefile install
if [ ! -d /home/tibs/sw/m3/example3/install/x86/bin ]; then mkdir /home/tibs/sw/m3/example3/install/x86/bin; fi
install -m 0755 /home/tibs/sw/m3/example3/obj/pkg_cpio/x86/hello_world /home/tibs/sw/m3/example3/install/x86/bin/hello_world
install -m 0644 hello_world.c /home/tibs/sw/m3/example3/install/x86/hello_world.c
> Building package:pkg_cpio{x86}/postinstalled

we end up with the subdomain built in its directory structure:

domains/
`-- b/
    |-- install/
    |   `-- x86/
    |       `-- hello_world*
    |-- .muddle/
    |   |-- am_subdomain
    |   |-- Description
    |   |-- RootRepository
    |   `-- tags/
    |       |-- checkout/
    |       |   |-- b_co/
    |       |   |   `-- checked_out
    |       |   `-- builds/
    |       |       `-- checked_out
    |       `-- package/
    |           `-- pkg_b/
    |               |-- x86-built
    |               |-- x86-configured
    |               |-- x86-installed
    |               |-- x86-postinstalled
    |               `-- x86-preconfig
    |-- obj/
    |   `-- pkg_b/
    |       `-- x86/
    |           `-- hello_world*
    `-- src/
        |-- b_co/
        |   |-- hello_world.c
        |   `-- Makefile
        `-- builds/
            |-- 01.py
            `-- 01.pyc

and the top-level build in its:

`-- install/
|   `-- x86/
|       |-- bin/
|       |   `-- hello_world*
|       `-- hello_world.c
|-- .muddle/
|   |-- Description
|   |-- RootRepository
|   `-- tags/
|       |-- checkout/
|       |   |-- builds/
|       |   |   `-- checked_out
|       |   `-- cpio_co/
|       |       `-- checked_out
|       `-- package/
|           `-- pkg_cpio/
|               |-- x86-built
|               |-- x86-configured
|               |-- x86-installed
|               |-- x86-postinstalled
|               `-- x86-preconfig
|-- obj/
|   `-- pkg_cpio/
|       `-- x86/
|           `-- hello_world*
`-- src/
    |-- builds/
    |   |-- 01.py
    |   `-- 01.pyc
    `-- cpio_co/
        |-- hello_world.c
        `-- Makefile

If we ask after packages, we have both sets:

$ m3 query packages
pkg_b
pkg_cpio

(this should arguably report the domain of each package name as well - there should probably be an option to select this).

If we ask what domains we have, we get:

$ m3 query domains

b

(the current printout of domains includes the "empty" or top-level domain in its listing, which is slightly unobvious, and will probably be fixed at some time).

Our dependency rules are now extended to include those from the subdomain as well:

$ m3 depend user-short
-----
checkout:builds/changes_committed <-VcsCheckoutBuilder-- [ checkout:builds/up_to_date[T] ]
checkout:builds/changes_pushed <-VcsCheckoutBuilder-- [ checkout:builds/changes_committed ]
checkout:builds/pulled <-VcsCheckoutBuilder-- [ checkout:builds/checked_out, checkout:builds/up_to_date[T] ]
checkout:builds/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:builds/checked_out ]
checkout:cpio_co/changes_committed <-VcsCheckoutBuilder-- [ checkout:cpio_co/up_to_date[T] ]
checkout:cpio_co/changes_pushed <-VcsCheckoutBuilder-- [ checkout:cpio_co/changes_committed ]
checkout:cpio_co/pulled <-VcsCheckoutBuilder-- [ checkout:cpio_co/checked_out, checkout:cpio_co/up_to_date[T] ]
checkout:cpio_co/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:cpio_co/checked_out ]
checkout:(b)b_co/changes_committed <-VcsCheckoutBuilder-- [ checkout:(b)b_co/up_to_date[T] ]
checkout:(b)b_co/changes_pushed <-VcsCheckoutBuilder-- [ checkout:(b)b_co/changes_committed ]
checkout:(b)b_co/pulled <-VcsCheckoutBuilder-- [ checkout:(b)b_co/checked_out, checkout:(b)b_co/up_to_date[T] ]
checkout:(b)b_co/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:(b)b_co/checked_out ]
checkout:(b)builds/changes_committed <-VcsCheckoutBuilder-- [ checkout:(b)builds/up_to_date[T] ]
checkout:(b)builds/changes_pushed <-VcsCheckoutBuilder-- [ checkout:(b)builds/changes_committed ]
checkout:(b)builds/pulled <-VcsCheckoutBuilder-- [ checkout:(b)builds/checked_out, checkout:(b)builds/up_to_date[T] ]
checkout:(b)builds/up_to_date[T] <-VcsCheckoutBuilder-- [ checkout:(b)builds/checked_out ]
deployment:everything/deployed <-CollectDeploymentBuilder-- [ package:*{x86}/postinstalled, package:(b)*{x86}/postinstalled ]
package:pkg_cpio{x86}/built <-MakeBuilder-- [ package:pkg_cpio{x86}/configured ]
package:pkg_cpio{x86}/configured <-MakeBuilder-- [ package:pkg_cpio{x86}/preconfig ]
package:pkg_cpio{x86}/installed <-MakeBuilder-- [ package:pkg_cpio{x86}/built ]
package:pkg_cpio{x86}/postinstalled <-MakeBuilder-- [ package:pkg_cpio{x86}/installed ]
package:pkg_cpio{x86}/preconfig <-MakeBuilder-- [ checkout:cpio_co/checked_out ]
package:(b)pkg_b{x86}/built <-MakeBuilder-- [ package:(b)pkg_b{x86}/configured ]
package:(b)pkg_b{x86}/configured <-MakeBuilder-- [ package:(b)pkg_b{x86}/preconfig ]
package:(b)pkg_b{x86}/installed <-MakeBuilder-- [ package:(b)pkg_b{x86}/built ]
package:(b)pkg_b{x86}/postinstalled <-MakeBuilder-- [ package:(b)pkg_b{x86}/installed ]
package:(b)pkg_b{x86}/preconfig <-MakeBuilder-- [ checkout:(b)b_co/checked_out ]
-----

and our deployment has the expected build order:

$ m3 query deps deployment:everything/deployed
Build order for deployment:everything/deployed ..
checkout:cpio_co/checked_out
checkout:(b)b_co/checked_out
package:(b)pkg_b{x86}/preconfig
package:pkg_cpio{x86}/preconfig
package:(b)pkg_b{x86}/configured
package:pkg_cpio{x86}/configured
package:(b)pkg_b{x86}/built
package:pkg_cpio{x86}/built
package:(b)pkg_b{x86}/installed
package:pkg_cpio{x86}/installed
package:(b)pkg_b{x86}/postinstalled
package:pkg_cpio{x86}/postinstalled
deployment:everything/deployed

Which means that when we deploy:

$ m3 deploy
Building deployment:everything/deployed
> Building deployment:everything/deployed
> Make directory /home/tibs/sw/m3/example3/deploy/everything
Copying /home/tibs/sw/m3/example3/install/x86/ to /home/tibs/sw/m3/example3/deploy/everything/
Copying /home/tibs/sw/m3/example3/domains/b/install/x86/ to /home/tibs/sw/m3/example3/deploy/everything/usr
> Make directory /home/tibs/sw/m3/example3/.muddle/tags/deployment/everything

we end up with everything deployed in the deploy/ directory at the top-level:

deploy/
`-- everything/
    |-- bin/
    |   `-- hello_world*
    |-- hello_world.c
    `-- usr/
        `-- hello_world*

bin/hello_world comes from the top-level build, and usr/hello_world from the subdomain. The hello_world.c is also from the top-level build (the example Makefile for the cpio package copies the source file into the install/ directory).

Version stamps and domains

Version stamps work with domains as well, although one may only generate them for the top-level in a particular build tree.

Thus we can try:

$ m3 stamp version
Finding all checkouts... found 4
Processing Svn checkout 'builds'
builds: 'svnversion' reports checkout has revision '459M'
Processing Svn checkout 'cpio_co'
Processing Svn checkout '(b)b_co'
Processing Svn checkout '(b)builds'
Found domains: set([DomainTuple(name='b', repository='svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b', description='builds/01.py')])

Unable to work out revision ids for all the checkouts
- although we did work out 3 of 4
Problems were:
* builds: 'svnversion' reports checkout has revision '459M'
Problems prevent writing version stamp file

OK, that makes sense, since we had indeed edited that checkout, and had not commited the changes. Luckily, it is sufficient for our purposes to save the state without that edit:

$ m3 stamp save -force
Forcing original revision ids when necessary
Finding all checkouts... found 4
Processing Svn checkout 'builds'
builds: 'svnversion' reports checkout has revision '459M'
Processing Svn checkout 'cpio_co'
Processing Svn checkout '(b)b_co'
Processing Svn checkout '(b)builds'
Found domains: set([DomainTuple(name='b', repository='svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b', description='builds/01.py')])

Unable to work out revision ids for all the checkouts
- although we did work out 3 of 4
Problems were:
* builds: 'svnversion' reports checkout has revision '459M'
Writing to working.stamp
Wrote revision data to working.stamp
File has SHA1 hash 6425284f002c81c9849f2f23add025a710207509
Renaming working.stamp to 6425284f002c81c9849f2f23add025a710207509.partial

and that at least shows us that the stamp file does record details of any subdomains:

[ROOT]
description = builds/01.py
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio

[DOMAIN b]
description = builds/01.py
name = b
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b

[CHECKOUT (b)b_co]
domain = b
name = b_co
relative = b_co
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b
revision = 459

[CHECKOUT (b)builds]
domain = b
name = builds
relative = builds
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/b
revision = 459

[CHECKOUT cpio_co]
name = cpio_co
relative = cpio_co
repository = svn+http://muddle.googlecode.com/svn/trunk/muddle/examples/cpio
revision = 459

[PROBLEMS]
problem1 = builds: 'svnversion' reports checkout has revision '459M'