I decided to freeze the current state of project-builder.org to deliver it under the 0.9.9 name.

It’s more to give access externaly to the current state of the art of this project, rather than it represents a final point of features in fact. A certain number of small bugs have been fixed, so it’s useful at least for that. Also I introduced with this version a new tool (and package) called rpmbootstrap, whose role is to create a chroot of a RPM based distro. I’ve been able to create CentOS and Fedora chroot up to now. Still working on it for OpenSUSE and Mandriva. It’s largely based on ideas of rinse, and debootstrap, but tightly coupled with pb and reusing some of its interesting features around distribution detection.

And at the same time, support of debootstrap has also been added so it’s possible to easily build now with chroots for most distro.

And I have passed a lot of time trying to improve documentation. Pod doc was already quite good for functions. Now in addition, we have the Lab for easy setup, and a complete documentation of the pb.conf possibilities, which will allow to work now on the possibility to use tool to manipulate them (Target is to use Config-Model). And a website has been setup. Not as complete as I’d like it to be, but a start

And I’ll be presenting the tool, and re-deliver the Lab during the upcoming HP Technology Summit in Las Vegas in June.

Hope to meet with some of you there, and also next week in Valencia in Spain for the Red Hat EMEA Partner Summit

bcornec

Design document for a patch management system on a DVCS

# patch branches without an explicit dependency depend on the root of the
# patchset tree
# A Root can be given as either a fix commit (seen here), a branch or a tag.
# A fixed commit or tag is useful to maintain a patchset against an older
# upstream version
ROOT: 6a8589de32d490806ab86432a3181370e65953ca
# A tag as a dependency
#ROOT: upstream/0.1.2
# A branch as a dependency
#ROOT: upstream

# A regular patch with it's name and last commit
BRANCH: debian/use-debian-jars-in-build-xml 4bab542c261ff1a1ae87151c3536f19ef02d7937

# two other regular patches
BRANCH: upstream-jira/HDFS-1234 a8e4af13106582ca1bfbbcaeb0537f73faf46d87
BRANCH: upstream-jira/MAP-REDUCE-007 e3426bcbcb2537478f851edcf6eb04b34f6c7106

# This patch depends on the above two patches
# The sha1 below the dependency patches references a merge commit of the two
# dependencies
BRANCH: upstream-jira/HDFS-008 517851aa829d77e09bc5e59985fed1b0aa339cc6
DEPENDENCIES:
  upstream-jira/HDFS-1234
  upstream-jira/MAP-REDUCE-007
    cc294f2e4773c4ff71efb83648a0e16835fca841

# A patch branch that belongs to the patch branch, but won't get exported (yet)
BRANCH: upstream-jira/HDFS-9999 74257905azgsa4689bc5e59985fed1b0aa339cc6
BRANCH-FLAGS: noexport

The upcoming version of project-builder.org will provide a new package called rpmbootstrap. Easy to guess what it does no ?!

Well for those of you Debian/Ubuntu/… fans, who already routinely use debootstrap, it will be no surprise. rpmbootstrap aims to become the debootstrap of rpm world. So it will allow you to create chroot (VE in pb terminology) for supported rpm based distro, in which you’ll be able to do what you have to do. And for pb, what it has to do is building packages.

So the first good news, is that pb in 0.9.9 will support both debootstrap and rpmbootstrap out of the box (on top of rinse and mock). The second good news, is that rpmbootstrap will be delivered jointly with it, trying to follow rpm distros evolution and providing an easy to build VE to build packages in them. Of course, you’ll be able to make a snapshot of that VE to always start over from a known state.

I was initialy tempted to continue to make patches to rinse. But I soon realized that the version I had was a bit far from what rinse provided, without benefiting from the integration I wanted for pb. Also, I couldn’t easily benefit from all the low level libs I now have in perl to help me deal with distro and conf files very efficiently.

I already had the centralized post-install script written for rinse, that I could integrate easily in rpmbootstrap. And writing just that new script took me some 8 hours. Was very easy with the libs ! And now I have a fully integrated solution to cover the whole build lifecycle. I tested it successfully with fedora 12, and I now need to check with other older Fedora distro, and add support for urpmi for mandriva and zipper for opensuse. As well as some tests with RHEL and SLES. But the whole infrastructure is there, just a couple of conf files entries are needed and a bit of code around.

Now, once I solve the package signage on rpm side (using a gpg-agent from perl), I’ll be able to publish version 1.0.

Linux.conf.au 2010 has come to an end and I am looking back at an intense week of conferencing. A big shout out to the organisers for their excellent work. I think LCA (as well as DebConf) just keeps getting better every year. This does not at all discredit previous organisers, because they were the best at their times and then passed on the wisdom and experience to help make it even better in the following year.

The week started off with the DistroSummit, which Fabio and I organised. Slides are forthcoming, as I failed to get them off the speakers right after their talks — it’s interesting how stress levels and adrenaline can cause one to forget the most obvious things. This is where experience comes in. I’ll be there again next year, I hope, to do things better.

The theme of the day was cross-distro collaboration, and we started the day a little bit on the Debian-side with Lucas Nussbaum telling us about quality assurance in Debian, alongside an overview of available resources. We hoped to give people from other distros pointers, and solicit feedback that would enable us to tie quality assurance closer together.

Next up was Bdale Garbee who talked about the status of the Linux Standard Base. While I am really interested in such standardisation efforts, I realised during his talks that I had considerable difficulties paying attention because as organiser of the conference, I had all sorts of other things occupying my thoughts.

I proceeded to tell the audience — the room was mostly filled throughout the day with an estimated 40–50 folks, and I’d say about half of them stayed throughout, while the other half came in and left the room between talks. I could not get the projector to work with my laptop after the upgrade to Kernel Mode Setting, and thus used the whiteboard to give a brief introduction to vcs-pkg.org, talk about the current state of affairs, summarise the trends in discussions around patch management and collaboration, give an outlook of what’s up next, and solicit some discussion.

Sadly, just like during Bdale’s talk, I found myself worrying over the organisation of the day, rather than actually taking in most of the discussion. Fortunately, others have written about the most important points, so I defer to them.

Michael Homer then told us about GoboLinux’s Aliens system, which is a way to integrate domain-specific packages with distro-specific package maintenance — e.g. how to get APT to handle CPAN directly, or how to let YUM manage Python packages. The ensuing discussion was interesting, and we carried it over to the next slot, because Scott, the next speaker, was stuck in traffic. To summarise briefly: scripting languages have a lot of NIH-style solutions because it works for them, but these are a nightmare to distro packagers. One symptom of the status quo is that complex software packages like Zimbra are forced to distribute all required components in their installation packages, which make distro packaging, quality assurance, and security support even harder. I don’t think we found a solution, other than the need for further standardisation (like the LSB), but the road seems to be a long and windy one.

Laszlo Peter introduced the audience to SourceJuicer, a new build system used by OpenSolaris. The idea is that contributors submit packages via a web interface, kicking off a workflow incorporating discussion and vetting, and only after changes have been signed-off are packages forwarded to auto-builders and eventually end up in the package repository. This is very similar to upload ideas I’ve had a while ago, which I’ve started to (finally) implement. Unfortunately, SourceJuicer seems very specific to OpenSolaris, as well as non-modular, so that I probably won’t be able to reuse e.g. the web interface on top of a Debian-specific package builder.

After the break, Dustin Kirkland stepped up to tell us about his user experience of Launchpad. Unfortunately, I found his talk a bit too enthusiastic. Launchpad undoubtedly has some very cool features and ideas, but it’s just one of the available solutions.

The dicussion of Launchpad also dominated the next talk, in which Lucas Nussbaum told us about the Debian-Ubuntu relationship. While his presentation showed that the relationship was improving (Matt Zimmerman made the point that there are rather many relationships, rather than one relationship), I was a bit disturbed by the comments of Launchpad developers in the room, ranging from “Debian is declining anyway” to “Just use Launchpad if you want to collaborate with others and not go down”. There was a slight aura of arrogance in their comments which tainted my experience of the otherwise constructive discussions of the day.

Overall I had a great time. Debian and Ubuntu made up the vast majority of attendants, with only a handful of representatives from other distros present. I wonder why that would be. One reason might be that around 70% of LCA attendants declared themselves Debian or Ubuntu users, and so there weren’t many other distros around. Another might be that I still haven’t spread the word enough. Let’s hope to do better next year!

Thanks to all the speakers. We may have organised the day, but you made it happen and interesting!

Slides and recordings of the talks will be linked from the archived website when they become available (yes, the archive page does not exist yet either).

Update: Jelmer informed me that the people who spoke up against Debian during and after the Launchpad talk were not officially affiliated with Launchpad. It’s a shame that this negatively reflected upon Launchpad for some of the attendees (not just myself).

For the first time this year, I’ll attend the FOSDEM in Brussels. Some FLOSS friends already explained to me it was a must attend event in EMEA, so this I tried to propose some conference material, and my proposal around Project-Builder.org was retained

I’ll only arrive on the Saturday morning, but will leave on Monday, combining a customer visit with it. So if any of the reader is around and want to discuss about Continuous Packaging or Disaster Recovery with Mondo Rescue, or anything else, that would be with great pleasure !

And there will be lots of interesting talks there as well, such as the one from Dominique Dumont on Config-Model, Guillaume Rousse on Youri or Jeff Johnson on RPM, just to name a few in the same room.

I hope to meet you there as

Here’s stuff that I’d like to do this year, more or less by decreasing order of importance:

• translate my Debian book into English and get it published;
• finish the cleanup of the perl API in dpkg-dev in order to create libdpkg-perl;
• create dpkg-buildflags to export default build flags that packages should use (and get rid of the code setting those environment variables in dpkg-buildpackage), needed to properly fix #489771;
• ensure the new source formats continue to gain acceptance by improving whatever is needed;
• design a generic vcs-buildpackage infrastructure to be integrated in dpkg-dev. This design will probably happen through a DEP (Debian Enhancement Proposal) to ensure we have had proper discussion before someone gets to the implementation;
• continue fixing dpkg bugs faster than they are reported;
• enhance our infrastructure to ease interaction between contributors and to have a better view of how each package is maintained (see my last blog entry on this topic);
• update the developers-reference where needed and fix some of the numerous wishlist bugs;
• rewrite in C the last perl scripts provided by the dpkg binary package (update-alternatives/mksplit mainly, for dpkg-divert there’s a preliminary patch available already) so that it’s easier to build a minimal system without perl-base;
• integrate the 3-way merge tool for Debian changelogs in dpkg-dev;

All of this probably doesn’t fit in my free time (being a father since last month does not help increasing my free time ), so if you’re interested in seeing one or more of those projects completed, and if you know some person/company that could sponsor them, get in touch with me!

Bzr build-deb is very nice, but it can be very tricky to get started. I recently did a fresh debianisation of a project that is in bzr upstream, and I thought I’d record the recipe to make it work (at least until the various bugs making it hard re fixed).

Assuming that the upstream uses bzr, it goes like this:

1. Start with a branch that is close to the code you want to Debianise. E.g. if the release was off trunk, 3 commits back: bzr branch trunk -r -3 debian
2. Debianise as normal: put the tarball with the right name in the parent dir,  add a debian directory and fiddle until you build a package you’re happy with. Don’t commit while doing this.
3. Build a source package- debuild -S, or bzr builddeb -S
4. Revert your changes – bzr revert.
5. Import the dsc – bzr import-dsc ../*.dsc
6. Now, you may find that some dot files, such as .bzrignore have been discarded inappropriately (there is a bug open on this). If that happened, keep going. Otherwise, you’re done: you can now use merge-upstream on future upstream releases, and debcommit etc.
7. bzr uncommit
8. bzr revert .bzrignore (and any other files that you want to get back)
9. debcommit
10. All done, see point  6 for details.

Hope-this-helps

Software comes in many shapes and styles. One of the problems the author of software faces is distributing it to their users.

As distributors we should not discourage upstreams that wish to generate binary packages themselves, rather we should cooperate with them, and ideally they will end up maintaining their stable release packages in our distributions. Currently the Debian and Ubuntu communities have a tendancy to actively discourage this by objecting when an upstream software author includes a debian/ directory in their shipped code.  I don’t know if Redhat or Suse have similar concerns, but for the dpkg toolchain, the presence of an upstream debian directory can cause toolchain issues.

In this blog post, I hope to make a case that we should consider the toolchain issues bugs rather than just-the-way-it-is, or even features.

To start at the beginning, consider the difficulty of installing software: the harder it is to install a piece of software, the more important having it has to be for a user to jump through hoops to install it.

Thus projects which care about users will make it easy to install – and there is a spectrum of ease. At one end,

checkout from version control, install various build dependencies like autoconf, gcc and so on

through to

download and run this installer

Now, where some software authors get lucky, is when someone else makes it easy to install their software, they make binary packages, so that users can simply do

apt-get install product

Now some platforms like MacOSX and Microsoft Windows really do need an installer, but in the Unix world we generally have packaging systems that can track interdependencies between libraries, download needed dependencies automatically, perform uninstalls and so on. Binary packaging in a Linux distribution has numerous benefits including better management of security updates (because a binary package can sensibly use shared libraries that are not part of the LSB).

So given the above, its no surprise to me to see the following sort of discussion on #ubuntu-motu:

1. upstream> Hi, I want to package product.
2. developer> Hi, you should start by reading the packaging guide
3. (upstream is understandably daunted – the packaging guide is a substantial amount of information, but only a small fraction is needed to package any one product.)

or (less usefully)

1. upstream> Hi, I want to package product.
2. developer> If you want to contribute, you should start with existing bugs
3. upstream> But I want to package product.

Another conversation, which I think is very closely related is

1. developer> Argh, product has a debian dir, why do they do this to me?!

The reasons for this should be pretty obvious at this point:

• Folk want to make their product easy to install and are not themselves DD’s, DM’s or MOTU’s.
• So they package it privately – such as in a PPA, or their own archive.
• When they package it, they naturally put the packaging rules in their source tree.

Now, why should we encourage this, rather than ask the upstream to delete their debian directory?

Because it lets us, distributors, share the packaging effort with the upstream.

Upstreams that are making packages will likely be doing this for betas, or even daily builds. As such they will find issues related to new binaries, libraries and so on well in advance of their actual release. And if we are building on their efforts, rather than discarding them, we can spend less time repeating what they did and more packaging other things.

We can also encourage the upstream to become a maintainer in the distro and do their own uploads: many upstreams will come to this on their own, but by working with them as they take their early steps we can make this more likely and an easier path.

It took us a bit longer than planned, but we are happy to announce the schedule for the Distro Summit at the upcoming LCA2010 conference. We focused on cross-distro aspects, and we hope that you are as excited as we are about the result.

The schedule is displayed on the Distro Summit homepage and I best refrain from duplicating it here.

The ftpmasters merged my dak branch last week during their meeting and have enabled the support of new source formats “3.0 (quilt)” and “3.0 (native)” in testing, unstable and testing-proposed-updates. I have uploaded 3 packages using the new formats already: logidee-tools using “3.0 (native)”, quilt and ftplib using “3.0 (quilt)”. The latter is arch any and has been successfully built on all architectures even those that still use an old version of sbuild (it looks like the fears that the old version would not cope with the new format were unfounded). For logidee-tools I built it with “-Zbzip2” in order to use bzip2 compression on the native tarball.

I have updated the wiki page and the release goal page with latest information. Feel free to convert some of your packages to give it a try. For ftplib, it led me to discover a Debian specific patch that I completely missed when I took the package over. This is precisely the kind of benefit that I expect from generalizing this format, it will encourage us to have separate documented patches instead of keeping everything hidden inside the usual .diff.gz. Combined with DEP-3 (patch tagging guidelines), we have a better infrastructure to share our patches with the rest of the free software community.

The next step is to fix all bugs listed here and make dpkg-source use the new source formats by default (#553928). Feel free to help by preparing patches (and offering NMUs), it’s a release goal to have all packages buildable with new source formats.

I’ve been considering for some time now to create a merge tool specifically suited for debian/changelog files. My goal was to let Git use it automatically thanks to gitattributes.

I’ve just gone ahead, so let me introduce you git-merge-dch. Grab it with git clone git://git.debian.org/~hertzog/git-merge-dch.git, you can build a package if you wish. Beware, you need to have a dpkg-dev 1.15.5 that is not yet published (so you need to build dpkg from its git repository, git clone git://git.debian.org/dpkg/dpkg.git) as I rely on features that I introduced recently… you will also need the libalgorithm-merge-perl package.

Using it in a git repository requires two changes:

• defining a new merge driver somewhere in the git configuration (in .git/config or ~/.gitconfig for example):

  [merge "git-merge-dch"]
          name = debian/changelog merge driver
          driver = git-merge-dch -m %O %A %B %A
  

• defining the merge attribute for debian/changelog files either in .gitattributes in the repository itself or in .git/info/attributes:

  debian/changelog merge=git-merge-dch
  

Now you can safely maintain two branches of a package with changelog files evolving separately and merge one into the other without creating undue conflicts. Suppose you created an experimental branch for version 2.28 (you use a version 2.28-1~exp1) when 2.26.2 was current stable in the master branch. In the mean time, 2.26.3 got out and was packaged in master. Next time you merge stable into experimental, the changelog entries for 2.28 and 2.26.3 won’t collide despite being at the same place in the changelog file compared to the common ancestor.

Let’s continue with this example, 2.28 is out. Instead of adding a new changelog entry with « New upstream release » without further changes, you keep the current changelog entry and simply change the version into 2.28-1. While preparing this you discover a branch with fixes that was based on 2.28-1~exp1, if you merge it it will reintroduce a 2.28-1~exp1 entry that you don’t want. Fortunately you can use the --merge-prereleases (-m) option of git-merge-dch so that it strip the prerelease part of the version string and considers 2.28-1~exp1 and 2.28-1 to be the same entry really.

The only limitation is that this merge tool will remove any lines which are not parsed by Dpkg::Changelog (and which in theory are not supposed to be there).

Feel free to test, share your comments, report bugs and send patches!

Today, Fabio and I extended the deadline of the call for papers of the upcoming Distro Summit until 18 October 2009. We have already received a number of great proposals, but not enough to complete the schedule. So if you have anything to say on the topic of distributions (such as, but not limited to the various Linux distributions), and especially on cross-distro collaboration, please launch your mailer and compose a message as detailed in the instructions.

Distro Summit will take place as part of LCA 2010 in Wellington, New Zealand. Yes, that is on the Southern Hemisphere, and yes, that is far away from where many of you live, but it’s well worth a visit, if you’ll take my word for it. Unfortunately, we are unable to provide sponsorship for speakers, so talk to your employer or your distro now and get the ball rolling. You have a little more than a week to get your bids in!

What are you waiting for? This is a once-in-a-lifetime chance: Distro Summit, LCA, and Wellington, all-in-one. It doesn’t get much better than that. Let’s hear your proposal!

PS: I loved Wellington (and its people) so much that I picked up my very own pet Wellingtonian!

I’ve seen in the past a lot of small FLOSS projects which were not doing their job up to the end. Work is not done when your code is compiling and begin to work the way you design it. As a project lead, and because you know the project best, it’s your task to take in account the installation procedure. Using Autoconf, is clearly the optimal approach, but for just a couple of script, a simple Makefile, or even an install.sh script could be perfectly sufficient. That way your potential user, has a helper to guide him in the installation of your software on his system, and it’s more easy for him to test it as a consequence, so hopefully to contribute in any way.

But that’s not the end again. Most of the users and system administrators are not using projects out of Version Control Systems (VCS) or Configuration Management Systems (CMS). Neither are they using tar files that you upload, even including a decent installation procedure. What they prefer, and at least that’s how I react, is a package for their distribution. It’s so handy ! All dependencies are taken in account and with tools such s yum, apt or urpmi it’s so easy to install a set of packages and their dependencies that it’s a real pleasure (compare to the boring configure; make; make install). But more over, as it’s easy to install, it’ easy to remove, to test momentarily, to avoid impact on your running system by a well controllled installation process. It’s even easy for newcomers, that have at their disposal graphical interfaces to search and install packages even more easily (at least for them )

So even if of course, this is a technical task (who said burden ) of packaging your project, at the end of the day, you should also perceive it as a marketing activity which helps you promote your project by making it so easy for hundreds of people to just download install and test, even with little system knowledge. More users, more testers, more reports, better software. Here is the virtuous circle in place.

Last point, if you think it’s difficult to package, then have a look at my packages for project-builder.org It’s here to help you !

The other day, Romain shared his concerns about using Git for team-maintained packaging. His comment system is currently broken, so I wrote an e-mail reply, which I would like to share.

I agree with Romain that the design decision to not support subtree checkouts like SVN is not without problems. As opposed to a single SVN repo with components in subdirectories that you can individually check out, you might end up with a hundred Git repos, and the same change to all then requires one to iterate all 100.

I’d like to make the distinction between trivial changes (e.g. s/© 2008/&-2009/g) and those that might not be (e.g. Standards-Version, or something even more elaborate).

In case of the former, there’s no question, it can be painful to operate across a hundred repos. Tools like mr make that a bit easier, but it’s still far from optimal.

The latter, however — updating Standards-Version and adding the appropriate changelog entry — is not really comparable. Neither would be e.g. changing a file location in 100 different repos. In those cases, every single package needs manual intervention, and if only for quality reasons and testing. In this sense, I actually think that a single SVN checkout with all the subtrees and the possibility to easily commit the result of a recursive action is counter-productive.

On the other hand, I don’t say that I am pleased with the workflow Git (or any other DVCS for that matter) imposes. It’s sometimes quite painful, as Romain says. We are missing higher-level tools that allow for easier and more intuitive bulk operations. I think that they should be implemented outside of the VCS-tool though, true to the Unix principles. SVN integrates it all into a monolithic piece of software, and that often isn’t ideal either (think size and slowness, or backup weight, or chance of corruption, or granular access control, or the impossibility to properly track files across subtrees).

mr is a step in the right direction, and we need more tools along those lines. First, however, I think that people need to figure out how exactly to use DVCS for packaging, such that there is any chance of consolidating workflows across a larger number of packages; if everyone does it their own, slightly unique way, then that goal is inifinitely far. This is the reason I started vcs-pkg.org, and even though we’re still far from anywhere, I am quite pleased with what we’ve done so far.

If you’re at Debcamp or DebConf, maybe you could join the discussion.

Romain also mentioned that distributed VCS don’t allow for the same sort of centralisation as SVN does. I disagree: you can use Git in exactly that way, as a centralised repo from which packages are built. The nice advantage over SVN (one which svk tried to close) is the ability for everyone to easily branch/fork, or work offline.

Once you start down that path, it somehow inherently becomes everyone’s own responsibility to ensure that one’s changes end up in the central repository (where commit hooks might verify the build-ability, ensure that the test suite still passes, or run simple format/consistency checks).

This sort of workflow is very different from the one with a self-appointed benevolent dictator at the top, who (like Linus, or Junio for Git) sometimes forget to include patches due to overflooding. The question is really: Given that you need some sort of centralised release coordination, do you want a human or a repo to be the central entity (and single point of failure)?

I really prefer the repo, since that places the sole responsibility on the leafs, on the contributors, who need to see their code through all the way.

It’s a whole lot more rewarding to commit/push, get a rejection, pull, merge, commit/push, and be done, rather than to send a patch to upstream, wait, reping, notice that it’s not in in the new release, ask, ping, change, reping, get angry, ping, hope, wait, ping, wonder why the heck you are still doing this, write angry email but don’t send it, reping, ask, and finally notice that it’s been accepted after all.

NP: Deep Purple: Made in Japan

This is a continuation from before. I am digressing a little in this post. One of the things I want to get out of this exercise is to learn more about Ontologies and Ontology editors, and on the principle that you can never learn something unless you build something with it (aka bone knowledge), so this is gathering my thoughts to get started on creating an Ontology for package building. Perhaps this has been done before, and better, but I’ll probably learn more trying to create my own.

Also, I am playing around with code, an odd melange of my package building porcelain, and gitpkg, and other ideas bruited on IRC, and I don’t want to blog about something that would be embarrassing in the long run if some of the concepts I have milling around turn out to not meet the challenge of first contact with reality.

I want to create a ontology related to packaging software. It should be general enough to cater to the needs any packaging effort in a distribution agnostic and version control agnostic manner. It should enable us to talk about packaging schemes and mechanisms, compare different methods, and perhaps to work towards a common interchange mechanism good enough for people to share the efforts spent in packaging software.

The ontology should be able to describe common practices in packaging, concepts of upstream sources, versioning, commits, package versions, and other meta-data related to packages.

I am doing this ontology primarily for myself, but I hope this might be useful for other folks involved in packaging software.

So, here follow a set of concepts related to packaging software, people who like pretty pictures can click on the thumbnail on the right:

• software is a general term used to describe a collection of computer programs, procedures and documentation that perform some tasks on a computer system.
• software is what we are trying to package
• software has names
• software may exist as
  • source code
  • executable code
  • packaged code
• source code is any collection of statements or declarations written in some human-readable computer programming language.
• source code is usually held in one or more text files (blobs).
• A large collection of source code files may be organized into a directory tree, in which case it may also be known as a source tree.
• The source code may be converted into an executable format by a compiler, or executed on the fly from the human readable form with the aid of an interpreter.
• executable format is the form software must be in in order to be run. Running means to cause a computer “to perform indicated tasks according to encoded instructions.”
• software source code has one or more lines of development. Some Common specific lines of development for the software to be packaged are:
  • upstream line of development
  • feature branch is a line of development related to a new feature under development. Often the goal is to merge the feature branches into the upstream line of development
  • usually, all feature branches are merged into the integration branch, and the package is created from the integration branch.
  • integration branch is the line of development of software that is to be packaged
• some software lines of development have releases
• releases have release dates
• some releases have release versions
• source code may be stored in a version control repository, and maintain history.
• Trees are a collection of blobs and other trees (directories and sub-directories). A tree object describes the state of a directory hierarchy at a particular given time.
• Blobs are simply chunks of binary data - they are the contents of files.
• a tree can be converted into an archive and back
• In git, directories are represented by tree object. They refer to blobs that have the contents of files (file name, access mode, etc is all stored in the tree), and to other trees for sub-directories.
• Commits (or “changesets”) mark points in the history of a line of development, and references to parent commits.
• A commit refers to a tree that represents the state of the files at the time of the commit.
• HEAD is the most recent commit in a line of development or branch.
• A working directory is a directory that corresponds, but might not be identical, to a commit in the version control repository
• Commits from the version control system can be checked out into the working directory
• uncommitted changes are changes in the working directory that make it different from the corresponding commit. Some call the working directory to be in a “dirty” state.
• uncommited changes be checked in into the version control system, creating a new commit
• The working directory may contain a ignore file
• ignore file contains the names of files in the working directory that should be “ignored” by the version control system.
• In git, a commit may also contains references to parent commits.
  • If there is more than one parent commit, then the commit is a merge
  • If there are no parent commits, it is an initial commit
• references, or heads, or branches, are movable references to a commit. On a fresh commit, the head or branch reference is moved to the new commit.
• lines of development are usually stored as a branch in the version control repository.
• A new branch may be created by branching from an existing branch
• a patch is a file that contains difference listings between two trees.
• A patch file can be used to transform (patch) one tree into another (tree).
• A quilt series is a method of representing an integration branch as a collection of a series of patches. These patches can be applied in sequence to the upstream branch to produce the integration branch.
• A tag is a named reference to a specific commit, and is not normally moved to point to a different commit.
• A package is an archive format of software created to be installed by a package management system or a self-sufficient installer, derived by transforming a tree associated with an integration branch.
• packages have package names
• package names are related to upstream software names
• packages have package versions
• package versions may have
  • an upstream version component
  • a distribution or packaging specific component
• package versions are related to upstream software versions
• helper packages provide libraries and other support facilities to help compile an integration branch ultimately yielding a package

This is a continuation from before.

Before I go plunging into writing code for a generic vcs-pkg implementation, I wanted to take a close look at my current, working, non-generic implementation: making sure that the generic implementation can support at least this one concrete work-flow will keep me grounded.

One of the features of my home grown porcelain for building package has been that I use a fixed layout for all the packages I maintain. There is a top level directory for all working trees. Each package gets a sub-directory under this working area. And in each package sub-directory, are the upstream versions, the checked out VCS working directory, and anything else package related. With this layout, knowing the package name is enough to locate the working directory. This enable me to, for example, hack away at a package in Emacs, and when done, go to any open terminal window, and say stage_release kernel-package or tag_releases ucf without needing to know what the current directory is (usually, the packages working directory is several levels deep — /usr/local/git/debian/make-dfsg/make-dfsg-3.91, for instance.

However, this is less palatable for a generic tool – imposing a directory structure layout is pretty heavy. And I guess I can always create a function called cdwd, or something, to take away the tedium of typing out long cd commands.

Anyway, looking at my code, there is the information that the scripts seem to need in order to do their work.

• Staging area. This is where software to be built is exported (and this area is visible from my build virtual machine).
  • User specified (configuration)
• Working Area. This is the location where all my packaging work happens. Each package I work on has a sub-directory in here, and the working directories for each package live in the package sub-directory. Note: Should not be needed.
  • User specified.
• Working directory. This is the checked out tree from the VCS, and this is the place where we get the source tree from which the package can be built.
  • Since we know the location of the working are, if the package name is known, we can just look in the package’s sub-directory in the working area.
    • For rpm based sources, look for the spec file
    • For Debian sources, locate debian/rules
  • If package name is not known, look for spec or debian/rules in the current directory, and parse either the spec file or debian/changelog.
  • If in a VCS directory, look for the base of the tree
    • tla tree-root
    • bzr info
    • git rev-parse --show cdup
    • hg root
    • You have to climb the tree for subversion
  • If you are in a debian directory, and changelog and rules files exist

  Then, look for the spec file or debian/rules in the base directory

• package name
  • User specified, on the command line
  • If in the working directory of the package, can be parsed from the spec or changelog files.
• upstream tar archive
  • Usually located in the parent directory of the working directory (the package specific sub-directory of the working area)
  • If pristine-tar is in use, given two trees (branches, commits. etc), namely:
    • a tree for upstream (default: the branch ~upstream~)
    • a tree for the delta (default: the branch ~pristine-tar~)

    The tree can be generated

  • Given an upstream tree (default: the branch ~upstream~), a tar archive can be generated, but is likely to be not bit-for-bit identical to the original tar archive.

So, if I do away with the whole working area layout convention, this can be reduced to just requiring the user to:

• Specify Staging area
• Call the script in the working directory (dpkg-buildpackage imposes this too).
• Either use pristine-tar or have the upstream tar archive in the parent directory of the working directory

Hmm. One user specified directory, where the results are dumped. I can live with that. However, gitpkg has a different concept: it works purely on the git objects, you feed it upto three tree objects, the first being the tree with sources to build, and the second and third trees being looked at only if the upstream tar archive can not be located, and passes the trees to pristine tar to re-construct the upstram tar. The package name and version are constructed after the source-tar archive is extracted to the staging area. I like the minimality of this.

This is continued here.

I have been involved in vcs-pkg.org since around the time it started, a couple of years ago. The discussion has been interesting, and I learned a lot about the benefits and disadvantages of serializing patches (and collecting integration deltas in the feature branches and the specific ordering of the feature branches) and maintaining integration branches (where the integration deltas are collected purely in the integration branch, but might tend to get lost in the history, and a fresh integration branch having to re-invent the integration deltas afresh).

However, one of the things we have been lax about is getting down to brass tacks and getting around to being able to create generic packaging tools (though for the folks on the serializing patches side of the debate we have the excellent quilt and the topgit packages).

I have recently mostly automated my git based work-flow, and have built fancy porcelain around my git repository setup. During IRC discussion, the gitpkg script came up. This seems almost usable, apart from not having any built-in pristine-tar support, and also not supporting git submodules, which make is less useful an alternative than my current porcelain.

But it seems to me that we are pretty close to being able to create a distribution, layout, and patch handler agnostic script that builds distribution packages directly from version control, as long as we take care not to bind people into distributions or tool specific straitjackets. To these ends, I wanted to see what are the tasks that we want a package building script to perform. Here is what I came up with.

1. Provide a copy of one or more upstream source tar-balls in the staging area where the package will be built. This staging area may or may not be the working directory checked out from the underlying VCS; my experience has been that most tools of the ilk have a temporary staging directory of some kind.
2. Provide a directory tree of the sources from which the package is to be built in the staging area
3. Run one or more commands or shell scripts in the staging area to create the package. These series of commands might be very complex, creating and running virtual machines, chroot jails, satisfying build dependencies, using copy-on-write mechanisms, running unit tests and lintian/puiparts checks on the results. But the building a package script may just punt on these scripts to a user specified hook.

The first and third steps above are pretty straight forward, and fairly uncontroversial.

The upstream sources may be handled by one of these three alternatives:

1. compressed tar archives of the upstream sources are available, and may be copied.
2. There is a pristine-tar VCS branch, which in conjunction with the upstream branch, may be used to reproduce the upstream tr archive
3. Export and create an archive from the upstream branch, which may not have the same checksum as the original branch

The command to run may be supplied by the user in a configuration file or option, and may default based on the native distribution, to dpkg-buildpackage or rpm. There are a number of already mature mechanisms to take a source directory and upstream tar archive and produce packages from that point, and the wheel need not be re-invented.

So the hardest part of the task is to present, in the staging area, for further processing, a directory tree of the source package, ready for the distribution specific build commands. This part of the solution is likely to be VCS specific.

This post is getting long, so I’ll defer presenting my evolving implementation of a generic vcs-pkg tool, git flavour, to the next blog post.

This is continued here.

There are a lot of little git scripts and tools being written by a lot of people. Including a lot of tools written by people I have a lot of respect for. And yet, they are mostly useless for me. Take git-pkg. Can’t use it. Does not work with git submodules. Then there is our nice, new, shiny, incredibly bodacious “3.0 (git)” source format. Again, useless: does not cater to submodules.

I like submodules. They are nice. They allow for projects to take upstream sources, add Debian packaging instructions, and put them into git. They allow you to stitch together disparate projects, with different authors, and different release schedules and goals, into a coherent, integrated, software project.

Yes, I use git submodules for my Debian packaging. I think it is conceptually and practically the correct solution. Why submodules? Well, one of the first things I discovered was that most of the packaging for my packages was very similar – but not identical. Unfortunately, the previous incarnation of my packages with a monolithic rules file in each ./debian/ directory, it was easy for the rules files in packages to get out of sync – and there was no easy way to merge changes in the common portions an any sane automated fashion. The ./debian/ directories for all my packages package that they are instrumental in packaging. So, since I make the ./debian/ directories branches of the same project, it is far easier to package a new package, or to roll out a new feature when policy changes – the same commit can be applied across all the branches, and thus all my source packages, easily. With a separate debian-dir project, I can separate the management of the packaging rules from the package code itself.

Also, I have abstracted out the really common bits across all my packages into a ./debian.common directory, which is yet another project, and included in as a submodule in all the packages – so there is a central place to change the common bits, without having to duplicate my efforts 30-odd times.

Now people are complaining since they have no idea how to clone my package repositories, since apparently no one actually pays attention to a file called .gitmodules, and even when they do, they, and the tools they use, have no clue what to do with it. I am tired of sending emails with one off-cluebats, and I am building my own porcelain around something I hope to present as a generic vcs-pkg implementation soon. The firs step is a wrapper around git-clone, that understands git submodules.

So, here is the browsable code (there is a link in there to the downloadable sources too). Complete with a built in man page. Takes the same arguments as git-clone, but with fewer options. Have fun.

I have been meaning to write this up for a long time now, since I vaguely made a promise to do so last Debconf. I have also been wondering about the inefficiencies in my work-flow, but I kept postponing my analysis since there were still large gaps in my packaging automation since I moved off Arch as my SCM of choice. However, recently I have taken a sabbatical from Debian, so I’ve had time to complete bits and pieces of my package building framework, enough so that I could no longer justify putting off the analysis. I tried writing it up, but the result confused even me; so I instead recorded every shell command during a recent series of packaging tasks, and converted that into a nice, detailed, activity diagram that you see over here. This is as efficient a work-flow as I have been able to come up with.

Along with a git commit hook script, that parses the commit log and adds pending tags to bugs closed in the commit, the figure above represents my complete work-flow – down to the details of every cd command I executed. I think there are too many steps still.

Feedback and commentary would be appreciated, as well as any suggestions to improve efficiency.