Boosting my morale? I wish!

Let’s take a deep breath. You probably remember I’m running a tinderbox which is testing some common base system packages before they are unmasked (and thus unleashed on users); in particular I use it for testing new releases of GCC (4.7) and GLIBC (2.16).

It didn’t take me much after starting GLIBC 2.16 testing to find out that the previously-latest version of Boost (1.49) was not going to work with it. The problem is that there is a new definition that both of them tried to provide, TIME_UTC (probably relates to C++11/C11). Now unfortunately since it’s an API breakage to replace that definition, it means that it can’t be applied to the older versions, and it means that packages need to be fixed. Furthermore, the new 1.50 version has also broken the temporary compatibility introduced in 1.48 for their filesystem module’s API. This boils down to a world of pain for maintainers of packages using Boost (which includes yours truly, luckily none is directly maintained by me, just proxy).

So I had to add one extra package to the list, and ran the reverse dependencies — the positive side is that it didn’t take long to fill the bug although there are still a few issues with older boost versions not being supported yet. This brought up a few issues though…

First problem is the way Boost build itself, and its tests, is obnoxious: it’s totally serial, no parallelisation at all! The result is that to run the whole testsuite it takes over eight hours on Excelsior! The big issue is that for the testing, it takes some 10-20 times longer to build the test than run it (lovely language, C++), so a parallel build of the tests, even if the tests were executed in series, would mean a huge impact, and would also likely mean that the tests would become meaningful. As they sit, the (so-called) maintainer of the package has admitted to not run them when he revbumps, but only on full new versions.

The second problem is how Boost versions are discovered. The main issue is that Boost, instead of using proper sonames to keep binary compatibility, embeds its major/minor version pair in the library name — although most distributions symlinks the preferred version to the unversioned name (in Gentoo this is handled through the eselect boost tool). This is not extremely far from what most distributions do with Berkeley DB — but it causes problem when you have to find which one you should link to, especially when you consider that sometimes the unversioned name is not there at all.

So both CMake and Autotools (actually Autoconf Archive) provide macros to try a few different libraries. The former does it almost properly, starting from the highest version and then going in descending order — but uses a pre-defined list of versions to try! Which mean that most packages with CMake will try 1.49 first, as they don’t know that 1.50 is out yet! If no known version is found, then it will fallback to the unversioned library, which makes it work quite differently whether you have only one or more than one version installed!

For what concerns the macros from Autoconf Archive, they are quite nasty; from one side they really aren’t portable at all, as they use GNU sed syntax, they use uname (which makes them totally useless during cross-compilation), but most worrisome of all, is that they use ls to find which boost libraries are available and then take the first one that is usable. This means that if you have 1.50, 1.49 and 1.44 installed, it’ll use the oldest! Similarly to CMake, it uses the unversioned library last. In this case, though, I was able to improve the macros by reversing the check order, which makes them work correctly for most distributions out there.

What is even funnier about the AX macros (that were created for libtorrent, and are used by Gource, which I proxy-maintain for Enrico), is that due to the way they are implemented, it is feasible that they end up using the old libraries and the new headers (it was the case for me here with 1.491.50, as it didn’t fail to compile, just to link). As long as the interface used have different names and the linker will error out, all is fine. But if you have interfaces that are source-compatible, linker-compatible, but with different vtables, you have a crash waiting to happen.

Oh well…

Autotools Mythbuster: Why autoconf is not getting any friendlier

You know that I’m an “autotools lover” in the sense that i prefer having to deal with autotools-based buildsystem that most custom build-system that are as broken as they can possibly be. Or with cmake.

But I’m definitely not saying that autotools are perfect, far from it. I’m pretty sure that I said before that there are so many non-obvious tricks around that confuse most people and cause lots of confusion out there. The presence of these non-obvious tricks is due partly to the way autoconf tarted in the first place, partly into the fact that it has to generate standard sh scripts for compatibility with older systems, and partly because upstream is not really helping the whole stack to maturate.

One of the most non-obvious, but yet common, macros out there is definitely the AC_ARG_ENABLE (and its cousin AC_ARG_WITH obviously), as I’ve blogged and documented before. The main issue with those macros is that most people expect them to be boolean (--enable and --disable; --with and --without) even though it’s quite freeform (--with-something=pfx or --enable-errors=critical). But also the sheer fact that the second parameters need to be returned by another different macro (AS_HELP_STRING) is quite silly if you think of it. And, if you look at it, the (likely more recent) AC_ARG_VAR macro does not require a further macro call for the help string.

It gets even wilder: a lot of the AC_ARG_WITH calls are, further down, wrappers around calls to PKG_CHECK_MODULES (a pkg-config support macro). How is this worse? Well, you end up repeating the same code between different projects to produce, more or less, the same output. To at least reduce the incidence of this, Luca wrote a set of macros that wrap around and perform these common tasks. The result is that you have to provide a lot less details; you lose some flexibility of course, but it produces a much neater code in your configure.ac.

Now, as I said, autoconf upstream is partly responsible for this: while the new versions have been trying to reduce the possible misuse of macros, they are bound to a vast amount of compatibility. You don’t see the kind of changes that happened with the 2.1x → 2.5x version jump, nowadays. The result of this is that you cannot easily provide new macros, as they wouldn’t be compatible with older versions of autoconf, and that would be taken as mostly bad.

And the reason for that can be found in the reason why libvirt’s configure is still stuck with compatibility toward autoconf 2.59. While autotools are designed to not require their presence on either the host or build system (unlike CMake, imake, qmake, scons, …), developing for, and checking compatibility with, older systems require to have the tools at hand to rebuild configure and company. I solve this myself by just using NFS to pass the code around the various (real and virtual) machines, after regenerating it on my Gentoo box, but I admit it’s definitely not sleek enough. In this case, libvirt is also developed against RHEL-5, and in that system the only autoconf available is 2.59.

One problem to solve this kind of problem would be to have a single, main repository of macros, that everybody could use to perform the same task. Of course this would also mean standardising on some sort of interface for macros, and for the build systems themselves, and this is easier said than done. While we’re definitely not to the point Ruby is we still aren’t really that standard. Many packages handled by more or less the same people tend to re-use the same code over and over, but that only builds a multiple amount of similar, but still half-custom build systems.

There are only two, as far as I know, efforts that ever tried extending autoconf in a “massive” way: KDE (and the results are, well, let’s just say that I can’t blame for trying to forget about KDE3’s build system), and GNOME. The latter is still alive, although it isn’t concerned with giving more generic macro interfaces to common tasks.

There is the autoconf macro archive but there is one catch with it: some of the macros submitted there have GPL or GPL-affine licenses (as long as they are compatible); that means that you might not be able to use them in MIT-licensed systems. While it even says that FSF does suggest using more open licenses for macro files, it does not require the submissions to be. Which can get quite messy on the long term, I’m afraid, for those projects.

At any rate, this post should show that I don’t really think that autotools are error-safe, but at the same time, I don’t think that creating a totally new language to express these things (like CMake does) is the solution. If only Rake was parallel-capable (which is unlikely to happen as long as Ruby cannot seriously multithread), then it would probably be, in my opinion, a better replacement for autotools than what we have now. That is, if the presence of Ruby is not a problem.