“Put it there and wait for users to break” isn’t a valid QA method

So it seems like Jeremy feels we’re asking too much of maintainers by asking them to test their packages. As I said in the comments to his blog, picking up poppler as a case study is a very bad idea because, well, poppler has history.

Historically, new upstream poppler releases in the recent times have been handled by guys working on the KDE frontends, when they suited their need, rather than being coordinated with other developers. This isn’t much of a problem for them because binary distributions have no problem with shipping two version of the same library on different ABI versions, and even building packages against different versions is quite okay for them — it definitely isn’t for us.

In turn, within Gentoo, in the recent times, poppler has been maintained by the KDE team; mostly by the one developer who, not for the first time, wanted to bump to 0.16.0 a couple of days ago, without adding a mask first and ensuring that all the packages in tree that use poppler would work correctly with it. Jeremy states that doing such a test would make ~arch akin to stable; I and I’m sure other of my colleagues, rather see it as “not screwing users up”.

First of all, it’s not like bumping something that you expect not having trouble; bumping grep and finding out that half the packages fail whose tarballs were built with Gentoo and a very very old version of our elibtoolize patches is unexpected. Finding out that, once again, the new minor version of poppler has API breakage is not surprising, it’s definitely expected, which means that when committing the version bump you can have either of two thoughts in mind “I don’t care whether GNOME packages break” or “Ooh shiny!”. In either case, you should learn that such thoughts are not the right state of mind for committing something to the tree. I said that before of other two developers, I maintain my opinion now.

And just to be clear, I’m not expecting that we spend months in overlays before reaching the main tree, I’m just saying that if you know it’ll break something, you should mask it beforehand, and give a heads’ up to the developers maintaining the other packages to due their job and fix it.

Now, in the comments of that post, Jeremy insists that factoring in my tinderbox is not an option for him, because it is “counting on a single developer cpu/time”. Right, because there is no other way to test reverse dependencies, uh? The tinderbox is meant to help the general situation, but it’s definitely not the only way; even though I’d have been glad to run it for poppler to find and report the problems, the task of checking its reverse dependencies is far from impossible to deal with for a single developer. There are a grand total of … thirty-nine (39!) packages with reverse dependencies on poppler! So it’s not a “can’t be done” situation, it’s a “can’t be arsed”. This also brings Jeremy’s ratio of “7/14000” packages with a problem to a more worrisome ⁷⁄₃₉. See the difference?

Simply put, do your own homework: learn about the package you’re maintaining, try hard not to break reverse dependencies; if it happens that you break a package unexpectedly, hey, it’s ~arch, we can deal with it. But do not commit stuff that you know will break a good deal of the packages depending on yours. Especially if you’re not out there to fix them yourself.

Splitting packages: when to bump?

One of the reasons why I think that splitting packages is a much higher overhead than it’s worth, is the version and revision bumping, and in particular, deciding if it has to happen or not. It’s not only limited to patching but also to new version releases, most of the time.

The problem with patching occurs when you have to apply a patch that spans different split packages: if it changes the internal interface between components, you’ve now got to bump the requirements of the internal packages so that they require the new version; if there are circular dependencies in there, you’re definitely messed up. And this also requires you to cut down the patch into multiple parts (and sometimes, apply the same parts at both sides).

The problem with bumping instead is somewhat simpler: when you have a package that is shipped monolithic, but quite separated logically, it’s not rare for upstream to release a new version that only fixes some of the internal “sub-packages”; this is what happens most of the time with alsa-tools and alsa-plugins (the former even more as each tool has its own configure script, instead of a single big one). In this cases, the split packages might not have to be bumped at all, if the main one is bumped. And this is quite a bit of a burden for the (Gentoo) packagers. You cannot just rely on >=${PV} dependencies (as they might not always be satisfied), and you shouldn’t bump it unnecessarily (users don’t like to rebuild the same code over and over again).

In particular, if you end up having the same version bumps for both even if the code hasn’t changed (or you still have to always rebuild them at every revision bump), then you are just making it more complex than it should be: regroup the package in a single ebuild! That is, unless upstream decides to break up the code themselves, like dbus did. If your problem is providing proper dependencies (as it seems it happened with poppler), then your problem is solved by the (now-available) USE dependencies, rather than by splitting to hair-thin packages and increasing the number of “virtual” ebuilds. The same applies to gtk-sharp (and yes, I know both were done by Peter, and yes he knows I don’t like that solution).

Right now, I maintain one split package (by the standard notion of split), and one that comes very near for our purpose: gdbserver and perf. The former is an almost-standalone sub-package of gdb itself (the GNU debugger), which ships with the same tarball, the latter is part of the Linux kernel source tarball (and patches), but is not tied to the rest of the source either.

In the first case, deciding whether to bump or not is quite simple: you extract the gdbserver sources from the old and new tarball, then diff them together. If there is no change, you ignore the bump (which is what I have done with the recent 7.0.1 release). It’s a bit boring but since the gdb sources aren’t excessively huge it’s not too bad.

It’s a different matter for perf: since it’s shipped within the kernel itself, any new release candidate of the kernel is a possible release candidate for perf as well! Luckily I can usually just rely on the release changelog and grep for perf: in front of the git log. It might not be the best choice, as it’s error-prone, but unpacking (and patching, in case of release candidates) the Linux sources that are needed for perf is a non-trivial task by itself, and it takes much more time than it would be worth.

Tree fasting

Since I’ve been working on my tinderbox, I have found lots of packages that are unmaintained both for what concerns ebuilds, and upstream. This actually reminded me a lot of my first months within the multimedia teams (sound and video), during which I have found so many bad implementations that I was, for a while, renamed “the package undertaker” — I think Jeremy (Darkside) has now this title though, thanks to treecleaning.

Up to now I really left it to the actual maintainer (or treecleaners) to remove the broken package from the tree; but since it seems like many developers don’t seem even to care about removing the broken packages when they stop maintaining them, I’ve finally decided to take the matter in my hands, and started looking at the packages to last rite.

This mainly involves old ebuilds, untouched but for batch fixes for the past years, failing with the stable glibc, and with huge QA violations in the ebuild. So that the packages won’t be removed just for fun, I’m giving them 60 days to be fixed and salvaged instead of the standard 30. This hopefully will be enough to save the important useful stuff and get rid of the stuff that is only dragging down the Gentoo quality average.

While I’m absolutely in favour of keeping everything in a single, huge tree (I dislike overlays and I have already said that many times) there is one way we can reduce the amount of problematic software in the tree: removing the stuff that is desperate and that will never be up on par with the rest of the packages, and refusing to add software if it doesn’t look like its QA issues can be solved (even if that means not having software that users might demand in the tree until somebody goes around to fix the issues).

So this goes out as a warning: if you care about a package that is going to be removed for QA reasons, you’re very welcome to step in to fix the problems with it, we’ll be glad to keep it in the tree (unless, of course, it depends on a whole chain of software that should disappear, like for instance aRTs, ESounD and GTK+ 1.2).

Braindumping

I’m writing this entry while I’m waiting for my pasta to get ready, in the lunch break I’m having between finishing some job task. For a series of reasons I’m going slowly at it because life in the past few days have been difficult. Adding to my problem with sleep, my neighbours resumed waking me up “early” in the morning (early by my standards, that is). Yes I know that for most people, 11am is not “early”, but given that I always tended to work during the night (easier not to be disturbed by family, friends, Windows users, …), and that they know that (not adding the fact that my father works shifts so he also works nights quite often), I wouldn’t expect such amount of noise.

With “such amount of noise” I mean that I get woken up, while sleeping with my iPod playing podcasts, the headphones in, with my room’s door closed. And no I don’t have light sleep, once I get to to sleep, unless I know I have to wake up (either to receive a parcel, to go to work remotely, or whatever). This weekend I ended up sleeping just shy of six hours per night, which is not good for my health either. I have now ordered a pair of noise-isolating in-ear phones, they should arrive tomorrow via UPS, they at least tend to be quite on time. On the other hand the Italian postal service, that should deliver me three Amazon packages, takes the usual eternity.

For what most of my readers are concerned, I still have my tinderbox running, after a few tweaks. First of all, Zac provided me a new revision of the script that generates the list of packages that need to be upgraded (or installed) in the system, which takes good care of USE dependencies so that when they are expressed I can see them before the system tries to merge them in. Then I’m wondering about the order of merge. Since I noticed that more than a couple of time I had to suspend the tinderbox run in the midst of it, the lower-end of the list of packages tended to not be merged as often as the upper-end (where quite a few packages I know still fail to merge). This time I executed the tinderbox from the bottom up (more useful since the sys-* packages are lower), but I’m considering next time to just sort the list randomly before feeding it to the merge command so that there are better chances for all the packages to be built in a couple of iterations.

Speaking about tinderbox and packages, I noticed that there are lots of packages that waste time without good reason, doing useless processing. This includes, for instance, compressing the man pages before Portage does so. While one can understand that upstream would like to provide the complete features to the users, it’s also usually a task that distributions do pick up, and would make sense to provide an option “I’m a packager” to not execute them.

Now, you could argue that those are very small tasks, and even for packages installing ten or twenty man pages it doesn’t feel like too much time wasted. But please think of two things first of all: the compression often enough is done with a for loop in sh rather than with multiple Make rules (which would be ran in parallel), serialising the build, and taking more time on multi-core systems. Secondarily, the man pages will have to be decompressed and compressed again by Portage, so it’s about three time the work strictly needed.

Another problem is with packages that, not knowing where to find an interpreter, be it Perl, Ruby, Python or whatever else, check for it in configure or with other methods and then replace it in all their scripts. And once again quite often using for in sh rather than Make rules. Instead of doing that they should just use /usr/bin/env $interpreter to let the system find the right one, and not hardcode it in the files at all (unless you need a specific version, but that’s another point altogether).

Well, I’ve eaten my pasta now (bigoli with pesto, for those interested), so I’ll get a coffee and be back to work. I’ll try to write a more useful blog later on today after I’m done with work. I have lots of things to write about, included something about XBMC (from an user standpoint, I don’t have time to approach it with the packager’s eye).

Why foreign language bindings shouldn’t live with their parents

Gentoo users probably know better than other distribution users the pain that is with foreign language bindings (C++, Python, Ruby, …) for various libraries. But these pains often are caused by what I think is a mistake in packages design: letting the foreign language bindings to “live with their parents”, that is with the original library that the bindings are bound to.

The first issue here is now we implement these cases in Gentoo: this is usually done through USE flags with the name of the language, like ruby, python, cxx, which enable building and installing the bindings; this also means that if a software is written using those bindings it has to use EAPI 2 USE dependencies or built_with_use checks. This is certainly not optimal, but I cannot blame upstream to the way we have to handle it, can I?

What I think the problem is, is that by having bindings and original library in the same packaging you’re forcing the same release cycle on two products that have lives that should be mostly independent. If you have to fix a nasty security bug in the underlying library, you don’t really need to release or rebuild the actual bindings. If you need to release new bindings to go with a new language version for things like Python and Ruby, you most likely not have to release a new version of the underlying library.

In general, you can expect that as long as the interface of the underlying library is stable, the bindings can remain the same even after ten bugfix releases of the libraries; or if the library is stable by itself, and the bindings aren’t, they can be released ten times for a single library release. This goes well in the name of reducing the amount of code that a single release affects.

The reason why reducing the amount of code a single release affect is not just that source-based distribution like Gentoo want to reduce the build time for their users, but it also goes to make it much easier for the various distribution to test the code, and ensures that unrelated changes get merged in with important fixes. This is especially useful for distributions with stable branches like Debian and Gentoo.

So please, when you write a library and want to add further bindings, create further packages, pretty please!

Delegating

No I’m not working on C# again, although I admit i have Mono installed. Actually to be honest I’m not working on anything at the moment.

I have scheduled a visit in Verona next Tuesday, then they’ll probably tell me when I’ll have surgery, and which kind of surgery. Up to then I’m not sure how to organise myself with jobs and other things so I’ll probably keep a very low profile.

So, to make sure I don’t end up leaving packages unmaintained and bugs unseen, I’m trying now to delegate as much as I can of my responsibilities to other developers. Hopefully this way there won’t be anything left untouched while I’m convalescent.

The only things I’ll be working on this week, and even those only briefly, are short documentation updates, and maybe, but just maybe, version bumps. I already updated the --as-needed fixing guide trying to make it a bit more clear what to do with some errors, and how to properly filter the flag, and when to actually filter it. I’ll probably write something more about that, and then start seeing if I can write something about autotools update problems.

I’m doing this because multiple people (both devs and non) contacted me while I was in the hospital asking how to fix stuff, for --as-needed, new libtool, new autoconf, and so on. As I cannot really answer them while I’m in the hospital, I’d rather see to write a proper guide before I enter again, and then leave that to them as a reference.

I really can’t afford another pancreatitis, so I’ll have to reduce my stress levels a lot. Help with stuff I’ve been working on is very very welcome. So if you wish to spare some time and help me with, for instance, identifying packages that need to be fixed to work with OpenPAM, so that Seraphim’s work could be actually made good use of after it’s done.

Now I’ll cut this entry short, as I want to take care of other things today, like being able to play a bit (I haven’t played seriously in weeks!), plus I have to download from the camera some more photos I made this morning.

Again, donations to me, gifts, but even more importantly, day after day, to the research on pancreatic diseases are welcome.

Proposal: reduce the size of system packages set

This is not an official proposal. I repeat, this is not an official proposal. It’s just an idea I had and which I should propose officially one day on gentoo-dev, but as I’m not subscribed right now and I didn’t want to yet, I’ll wait for another day to do that.

I already ranted about the fact that the dependency tree of our ebuilds is vastly incomplete, as many lack dependency on zlib; trying to get this fixed was impossible, as Donnie and other insisted that as zlib was in system, we shouldn’t depend on it at all. I disagree, and I would like to know why we can’t depend on a system package, but whatever.

Anyway, as having a complete dependency tree is almost impossible because of that, I have an alternative proposal: reducing the size of the system package set.

Right now system contains stuff like ncurses, readline, zlib, autoconf, automake and m4, perl, gnuconfig, and so on. Those are packages that certainly would be part of any base Gentoo system, but are those actual part of the system set of packages? I sincerely doubt it.

The reason of the existence of the system package set is related first and foremost to breaking circular dependencies: for instance if any package that used the C compiler would depend on gcc, then the packages that gcc depends upon would create a circular dependency between gcc and itself. Also, specifying libc in almost any ebuild would be quite pointless, as well as coreutils (or freebsd-bin/ubin) for cp, mv, install, …

But why autoconf and automake? Well the easy answer is that those are often used without making sure they are depended upon explicitly… or at least this was the case till me and Martin added autotools.eclass to the tree; nowadays all the ebuilds using autotools should have proper autoconf/automake dependency already, and if they don’t they are broken anyway. So why leaving them in system? And what about m4? m4 is not part of a common Unix system, it’s just an autoconf dependency, why isn’t it just an autoconf dependency?

For what concern the three main libraries, there aren’t that many packages using zlib directly nowadays, this is especially easy to spot on a system built with --as-needed, as without that you actually do see zlib used in every other binary, for indirect dependencies. Nor there aren’t tons and tons of packages using readline, or ncurses. Actually in my own vserver’s chroot I only found four packages using readline, none of them part of system: ruby with the readline extension (uhm I wonder if I should ask for this to become an USE flag, I certainly don’t need it and I’d rather get rid of it), psql from postgresql (which maybe it’s still good to have with readline compiled in, but I could easily get rid of), bc (which is just an e2fsprogs build-dep, and I could build without readline just fine), and mysql.

A little bit different the status of ncurses, which is used by screen, gettext (only a build-dep, not needed for runtime on Linux anyway), procps, psmisc and util-linux (and I wonder why we don’t have a switch to turn it off), texinfo (wonder why we can’t remove it entirely actually) and yet again ruby. Still, I wonder why ncurses is in system rather than being properly on the dependencies list of those packages.

As for perl, that’s probably a bit more justified, there are tons of packages using perl directly or indirectly, especially in build systems. But I would like those to depend on perl properly rather than having perl in system, as there are cases where perl is not really needed at runtime at least.

And the only users of gnuconfig are the packages making use of the old and deprecated gnuconfig.eclass, or portage’s econf. Why can’t it be a dependency of portage then?

There are probably other packages that should, in my opinion, be removed from system, but these are certainly some of the most common. Unfortunately there’s a recursive problem here: to remove the packages from system without breaking stuff you’d need a proper deptree, and to get a proper deptree you need to remove the packages from system. This is what actually stops me from proposing this right away…