Adding a new program to a muddle build description

So I have a program (which I shall, for the sake of argument, call "george", because that is not its name) which relies upon ffmpeg and tstools, and which I want to incorporate into an existing muddle build.

This article describes how I did it. The resultant code is that used in the real system (except that the names have been changed).

I shall assume that you've read the muddle documentation, and in particular the Welcome to Muddle readme.

As described in the muddle documentation, the project I'm amending is laid out in the traditional manner:

+- .muddle/
+- src
|  +- builds/
|  |  +- 01.py
|  +- builders/
|  +- package1/
|  +- package2/
|  +- ...
+- obj/
+- install/
+- deploy/

where src/buids/01.py is the build description, and the various package1, etc., are existing software packages that are being built already.

(Naming the main build description 01.py appears already to be a tradition in muddle. It could, of course, be called something else, perhaps named after the project.)

The project I'm building is being deployed on a system similar to that I'm building on, so the same toolchain is being used, and I do not need to specify the compiler and so on. This also means that any configure scripts can configure against the system I'm building on, and get results that make sense on the target. This is unusually simple, but makes the examples a lot easier to read.

ffmpeg

For this program, we need a fairly recent version of ffmpeg, so we need to get it from the ffmpeg site. We are not, however, particularly interested (at this stage) in tracking future changes to the library.

ffmpeg is available as spot releases, via SVN, or via GIT. Since the project I'm adding to uses GIT to store its checkouts, it makes sense to use the GIT version of ffmpeg, and thus I can just git clone ffmpeg (and libswscale) and add them to the project's repository directly. This means we'll be working with a local checkout, rather than a remote one, which will also speed up the process of checking out when doing a build from scratch.

That leaves us with src/ffmpeg.

muddle prefers packages to build out-of-tree, so that it is possible to build for more than one target without needing to "clean" the source tree. ffmpeg supports this, so that's nice.

Since src/ffmpeg is taken directly from the outside world, we'd rather not add muddle specific build files, configuration files, etc., to that directory. The convention for muddle is to have a separate builders directory containing muddle specific infrastructure for an external package - so, in this case:

src/builders/ffmpeg/Makefile.muddle

tells muddle how to build ffmpeg. If we needed (for instance) an init.d script, or other support files, then this would be a natural place to keep them as well.

The code I need to add to the build script (builds/01.py) is then:

# ffmpeg
# - we have a GIT clone of ffmpeg's own repository in our repository already
muddled.checkouts.simple.relative(builder, "ffmpeg")

# - we have our own information on how to build it
make.medium(builder,
            name = "ffmpeg",        # package name
            roles = ["project"],
            checkout = "builders",
            deps = [],
            makefileName = os.path.join("ffmpeg","Makefile.muddle"))

# building ffmpeg requires the Makefile.muddle in builders/ffmpeg
muddled.pkg.package_depends_on_checkout(builder.invocation.ruleset,
                                        "ffmpeg", # This package
                                        "project",# in this role
                                        "ffmpeg", # depends on this checkout
                                        None)

In other words:

1. We are checking out ffmpeg from our own repository (so it is a "relative" checkout - an "absolute" checkout would need to specify an external repository).

2. We build package "ffmpeg" for the role "project" using the checkout "builders" (for simplicity I'm checking out all of the builders directory). This doesn't depend on anything else. We're going to use the named makefile.

   (Note that we've earlier done:

   import muddled.pkgs.make as make
   

   in order to keep the code easier to read.)

3. As a shorthand, we tell muddle that that same package ("ffmpeg") in the given role depends on the checkout of ffmpeg - in other words we can't build the ffmpeg library until we've checked it out.

   We could have done that last step directly by specifying a dependency in the deps argument to make.medium, but that would involve constructing the appropriate label (not actually that hard, but the "helper" function is probably slightly easier).

Makefile.muddle then looks like:

# Build ffmpeg.
# Ask muddle where the ffmpeg package has been checked out
FFMPEG_SRC=$(shell $(MUDDLE) query dir "checkout:ffmpeg{*}/*")

all: FORCE
        cd $(MUDDLE_OBJ_OBJ); make

# Do our configuration building in the MUDDLE_OBJ_OBJ directory. ffmpeg
# supports remote configuration building, by specifying the path to the
# configure script. It then leaves (a link to) the Makefile in that directory,
# so that we can "cd" there and build "directly".
#
# When configuring, say that our files will end up (when installed) in "/",
# because, on the target machine, they should.
config:
        -rm -rf $(MUDDLE_OBJ)
        -mkdir -p $(MUDDLE_OBJ_OBJ)
        cd $(MUDDLE_OBJ_OBJ); $(FFMPEG_SRC)/configure --prefix=/

# When installing, we're installing within the muddle infrastructrure,
# not onto the target machine. So we need to use DESTDIR to say "although
# you want to install into "/", pretend that "/" is rooted at MUDDLE_OBJ".
install:
        cd $(MUDDLE_OBJ_OBJ); make DESTDIR=$(MUDDLE_OBJ) install

distclean:
        -rm -rf $(MUDDLE_OBJ)

clean: FORCE
        make -C $(FFMPEG_SRC) clean

FORCE:

tsstools

The second package we need to build is tstools.

Unfortunately, tsools does not support building out-of-tree. The best solution to this would have been to fix it (particularly since I am the tstools maintainer), but for speed I didn't bother, and it allows an example of how one might work around this sort of problem.

Again, for simplicity, and because this project does not care about future changes to tstools (particularly if the build mechanism should change!), we take a copy of tstools into our own repository.

The build script code for tstools is very similar to that for ffmpeg:

# tstools
# - we have a copy of tstools' own repository in our repository already
muddled.checkouts.simple.relative(builder, "tstools")

# - we have our own information on how to build it
make.medium(builder,
            name = "tstools",       # package name
            roles = ["project"],
            checkout = "builders",
            deps = [],
            makefileName = os.path.join("tstools","Makefile.muddle"))

muddled.pkg.package_depends_on_checkout(builder.invocation.ruleset,
                                        "tstools",  # This package
                                        "project",  # in this role
                                        "tstools",  # depends on this checkout
                                        None)

So we once again have a Makefile.muddle, predictable in builders/tstools/Makefile.muddle:

# Build tstools.
# tstools doesn't currently support building out-of-tree, so we'll have to cope
# - primarily by cleaning before each build, and then copying the result out of tree

# Ask muddle where the tstools package has been checked out
TSTOOLS_SRC=$(shell $(MUDDLE) query dir "checkout:tstools{*}/*")/trunk

# Force a clean before our build in case someone else has built
# This *is* a pain, but is best solved by making tstools build out-of-tree
# for itself
all: FORCE
        make -C $(TSTOOLS_SRC) clean
        make -C $(TSTOOLS_SRC)

# We install the include files into MUDDLE_OBJ_INCLUDE. At the moment we
# aren't deploying them to the target system, but if we were we'd be
# putting them into one of the main "include" directories, which would
# be rather unfriendly. So that's worth bearing in mind...
install: all
        -mkdir -p $(MUDDLE_OBJ_LIB)
        -mkdir -p $(MUDDLE_OBJ_INCLUDE)
        cp $(TSTOOLS_SRC)/lib/* $(MUDDLE_OBJ_LIB)
        cp $(TSTOOLS_SRC)/*.h   $(MUDDLE_OBJ_INCLUDE)

config:
        # Nothing to do here

distclean:
        make -C $(TSTOOLS_SRC) distclean
        -rm -rf $(MUDDLE_OBJ)

clean: FORCE
        make -C $(TSTOOLS_SRC) clean

FORCE:

george

Finally, we have the program itself. This is the only thing we're actually going to want on the target system, since we're building against static libraries.

Since this is a program that is written for this system, and thus we don't need a separate checkout for the muddle specific Makefile, the muddle description code is very simple:

make.medium(builder,
            "george",
            ["project"],
            "george",
            ["ffmpeg", "tstools"],
            config = False,
            makefileName = "Makefile.muddle")

Specifically, we're building the package "george", for the "project" role, based on the checkout "george". By default this is a simple checkout, and thus we don't need to specify how to get "george". We depend on the two packages named, so they must already have been built. config=False means we don't need to do a configuration step (make configure).

We could have called the Makefile just Makefile, but it seemed clearer to me to make it specific that it is a muddle makefile, since muddle makefiles depend on the muddle infrastructure to work (to set all those special environment variables).

The makefile for "george" (which is in the src/george directory) is then:

# Build george
# MUDDLE_INCLUDE_DIRS is given to us as a list of the include directories
# we said we depended on.
# The incantation below says "For everything in MUDDLE_INC_DIRS, prefix '-I'"
INCS=$(MUDDLE_INCLUDE_DIRS:%=-I%)

# Similarly for MUDDLE_LIB_DIRS
LIBS=$(MUDDLE_LIB_DIRS:%=-L%)

FFMPEG_LIBS=-lavformat -lavcodec -lavutil -lswscale -lz -lbz2 -lm

TSTOOLS_LIBS=-ltstools

# Default to building in the current directory
O ?= $(CURDIR)

$(O)/george:        george.c
        $(CC) $(CCFLAGS) -o $@ george.c $(INCS) $(LIBS) $(FFMPEG_LIBS) $(TSTOOLS_LIBS)

install:
        cp $(O)/george $(MUDDLE_INSTALL)/bin

config:
        # Nothing to do

distclean: clean
        # Nothing to do

clean:
        rm -f $(O)/george

The MUDDLE_INCLUDE_DIRS and MUDDLE_LIB_DIRS magic is explained in the muddle documentation - it gives me a list of the include directories and library directories implied by the dependencies I've stated for this package (so if I get those wrong, that will show up here).

The $(O) business is building george out-of-tree, so that we can have several different builds of the same source at the same time.

Note that the install target is copying the "george" executable to $(MUDDLE_INSTALL)/bin.

The flow of built objects in muddle is, more or less:

src/ --compile--> obj/ --install--> install/ --deploy--> deploy/

Both ffmpeg and tstools were compiled and the results placed into obj/ subdirectories. Since they are not destined for the target system, however, they are not copied to install/.

"george" is to be put on the target system, so it is copied to install/.../bin -- the exact location depends upon the role, etc.

(Copying things from install/ to deploy/ is done by muddle automatically as part of the deployment phase, so we don't need to do anything special for that - again, see the muddle documentation.)

It is perhaps unfortunate that the Makefile target called install is used for both purposes, but that does fit with the normal make usages. Of course, if we were wanting ffmpeg libraries on the final system, we'd have to extend the install target in its Makefile.muddle to copy files from MUDDLE_OBJ to MUDDLE_INSTALL, but that's an example for another time.

Other things

Since muddle is dependency based, it can be useful to look at the dependencies. The main command to do this is:

$ muddle depend user

which shows all the build description dependencies, or (for instance):

$ muddle depend user 'package:ffmpeg{project}/*'

which shows all the dependencies for package "ffmpeg" in role "project", with any tag (nb: package "ffmpeg" is distinct from checkout "ffmpeg", which is separate and distinct - again, see the muddle documentation).

(muddled.depend.Label() can be used to construct a label from its parts, and thus may help to remember what those parts are.)

Meanwhile, as I iterated towards getting builders/ffmpeg/Makefile.muddle correct,

$ muddle distclean ffmpeg

was also my friend.

Finally, all of the information about the current build is kept under the .muddle directory. It is kept in very simple form, and the muddle command reads the information "as given". This means that if I get my system into a strange or broken situation, and want to clear muddle's memory of what has been done, I can simply delete the appropriate tags (which are just text files containing a timestamp) from the .muddle directory structure.

This turns out to be very useful when trying to develop a system, and is much easier to handle than the tangle one tends to get into with traditional (make based) build and deployment systems.