Testsuites are important, once again.

I start sincerely to get tired about this, but here it comes: testsuites are important to know whether a package is good or not; so the package you’re developing should have a testsuite. And the ebuild you’re writing should execute it. And you should run it!

This is something I noticed first in the tinderbox, but some people even complained about that directly to me: a lot of ebuilds make a mess with testsuites. The problems with them range from not running them at all and restricting without really good reasons, to testsuites that are blatantly broken because they were never tested before.

I guess the first problem here is the fact that while the test feature that executes the actual testuites is disabled by default, Portage provides a default src_test. Why is this a problem, you say? After all, it really does add some value when the testsuite is present, even if the maintainer in the ebuild didn’t spend some extra minutes writing it down. Unfortunately, while it adds test phases to lots of ebuilds where they are correctly executed, it also adds them to packages that don’t have testsuites at all (but, if they use automake, it’ll still run a long make chain, recursive if the build system wasn’t built properly!), to packages that have different meanings for the check or test targets (like all the qmail-related packages, for which make check checks the installation paths and not the just-built software), and to packages whose testsuite is not only going to fail, but also to hog a computer down for a pretty long time (did somebody say qemu?).

Now, the problems with tests does not stop here with the default src_test, otherwise it would also be pretty easy to fix; the problem is that we don’t really have a clear policy on how to deal with the testsuites, especially those that fails. And I have to say that I’m as bad as the rest of the group when it comes to deal with the testsuites. I can, first thing, bring up two packages I deal with that have problems with their testsuites.

PulseAudio, which is a pretty important package, you’d say, has a complex testsuite; for quite a long time in the test releases (that in Gentoo become RCs even though they really are not, but that’s another issue here) one of the tests (mix_test) failed, because the test itself wasn’t being updated to support the new sample format, this was only fixed recently (there were other tests failure, but those I fixed myself at the first chance); on the other hand, the tests for the translations, that are also part of the package’s testsuite, are still not executed: the current version of intltool (0.40) does not interpret correctly the configure.ac file (it parses it like it was a text file, rather than accepting that it’s a macro file), and causes the test to fail in a bad way; the solution for this part is to package and add a dependency over intltool 0.41, but seems like nobody is sure whether that’s an official release or a development release. For now, only the software tests are executed;
speaking of docbook the XSL stylesheet for Docbook used to have a complex testsuite that checked that the output was what it was supposed to be; now they weren’t really comprehensive and indeed at least one bug was missed by the testsuite in the whole 1.74 series. Starting from 1.75 the new testsuite should probably be tighter and support more than just one XSLT engine… the problem is that upstream doesn’t seem to have described the testing procedure anywhere, and I haven’t figured out how it works yet, with the result that the testsuite is now restricted in the ebuilds (with a test USE flag that is not masked, I already filed an enhancement request for Portage to handle this case).

At this point what I’m brought to wonder is: how harsh should we be on the packages with flawed, broken, or incomplete testsuites? Should they be kept in package.mask? Should they not reach stable? The stable-stopper for this kind of problems used to be Ferris, and it’s one reason I’m really going to miss him badly. On the other hand it seems like a few other arch team members started applying the same strictness, which I don’t dislike at all (although it’s now keeping libtool-2.2 from going stable, and with that PulseAudio as well). But what about the packages that already fail in stable? What about the packages failing because of mistakes in the testsuites?

There are also arch-specific issues, for instance I remember some time ago Linux-PAM requiring a newer glibc than it was available on some arches for its testsuite to proceed correctly… the running logic of PAM, though, seemed to work fine beside the test. What should have been the correct approach? Make the whole of Linux-PAM depend on the new glibc, making it unusable by some arches, or just the tests? I decided for the tests, because the new version was really needed, but on a pure policy point of view I’m not sure if it was the right step.

I guess the only thing I can add here is, once again, if you need to restrict or skip tests, keep the bug open, so that people will know that the problem has only been worked around and not properly fixed. And maintainers, always remember to run the testsuites of your packages when bumping, patching or otherwise changing your packages. Please!

Bundling libraries for despair and insecurity

I’ve been told quite a few times that my posts tend to be too long, and boring for most basic users, so for a time I’ll try to use the “extended content” support from Typo, and see how people react. What this means is that the blog post is just summarised on feeds, and on aggregators like Planet, while the complete text can be read by accessing the article directly on my blog.

When I started my work reporting bundled libraries almost an year ago, my idea had a lot to do with sharing code and just to the side to do with the security issues related to bundled libraries. I had of course first hand experience with the problem, since xine-lib has (and still in part had) bundled a lot of libraries. When I took over maintainership of it in Gentoo, it was largely breaching policy, and the number of issues I had with that was huge. With time, and coordination with upstream (to the point of me becoming upstream), the issues were addressed, and nowadays most of xine-lib bundled libraries are ignored in favour of the system copies (where possible; some were largely modified to the point of not being usable, but that’s still something we’re fighting with). Nowadays, the 1.2 branch of xine-lib already doesn’t have a FFmpeg copy at all, always using the system copy (or an eventual static copy built properly).

But nowadays I started to see that what is obvious to me about the problems with bundled copies of libraries is not obvious to all developers, and even less obvious to “power users” who proxy-maintain ebuilds and just want them to work for them, rather than complying with Gentoo policies and standards. Which is why I think that sunrise and other overlays should always be scrutinised carefully before being added to a system.

At any rate, for this reason I’m going to explain in this post why you should not use bundled internal copies of libraries for packages added to Gentoo, and why in particular these packages should not be deemed stable at all.

The first issue to discuss is why do upstreams bundle libraries, since knowing the reasoning behind that is often helpful to identify whether it makes sense at all to keep them or not. The first most obvious answer is: to reduce dependencies. For a long time this was the major reason behind xine-lib usage of internally bundled libraries. As it turns out, with time this reason became moot: distributions started packaging xine-lib directly reducing the number of users wishing to build it from sources; even those wanting to build xine-lib from sources would find all the needed libraries in the distributions, most of the times. When this is the sole reason for libraries building, upstream should be very well open to add a configure option (or anything similar to that) to use the system copy, optionally or by default, with fallback to the bundled copy.

A second reason might be that the library is being unstable when it comes to API; this is probably the first reason why FFmpeg is often bundled in software rather than using the copy on the system; while this is a concern that makes more sense then the one before, it’s still mostly a moot point since it really just requires to fix the issue at the source: get the original project to maintain API compatibility, to provide an API compatibility layer, or to finalise its API. Even when it cannot be helped, because the API is in flux, maintained software fears not the API break; it might be a bit of an hassle but in general it’s feasible to keep the use and the library in sync.

Thirdly, more worrisome, is when the library is modified, slightly or heavily, after bundling; in this case using the system copy might be quite a burden because it will lack the specific changes as made by the project. In this case there is a lot of work involved, sometimes more work that it can be taken care by distributions, and requires coordination of the project’s upstream together with the higher level upstream. This is what happened with xine-lib and FFmpeg: the copy in the xine-lib sources was heavily modified to suit both the build system and the interface requirements of xine, which made it also very difficult to update the internal snapshot. All the interface changes needed have then been pushed upstream to FFmpeg, and the buildsystem changes were made moot by using the default buildsystem (with needed changes pushed upstream) embedded in autotools; and then FFmpeg was entirely removed from xine-lib’s sources.

Now, on the other hand, the disadvantages of using bundled libraries are probably worse: code duplication means that there is more data to process (both at build time to compile and at load time to store in memory), there is more space used by the binaries, and there are duplicated bugs that need to be fixed twice. A lot of time in xine-lib the problems with decoding something with FFmpeg were solved by just using a newer FFmpeg; why keeping one then?

The most important issue though is about security: when a vulnerability is found in a library like zlib, fixing the library alone is not enough: while that fixes the majority of the software in a system, it’s not going to fix those who bundle it, both closed-source and open source. For instance, take dzip; it uses an ancient internal version of zlib; if somebody knows the format well enough, it’s far from impossible to craft a dzip file that contains a deflated stream that can executed malicious code.

For this latter issue alone, I’d say that any software bundling source code is not good enough to go stable on its own. Of course sometimes one has to bend the rules because of past mistakes, for instance even though Ruby bundles stuff, we cannot stop newer versions to go stable; this problem is not a regression. But should stop other broken software from entering portage or at least the stable tree.

But it’s not just security, subtle bugs might actually be quite a problem. For instance, you might remember all Java applications failing when libX11 was built with XCB support some time ago. The problem was due to some stricter checks in libxcb compared to what libX11 have been checking before, but the source of the problem was Xinerama. The problem with that was that Sun bundled an internal copies of libXinerama sources in the JRE sources, and even though libXinerama was since then fixed regarding that particular issue (the crash with XCB), it was never updated in the JRE before the issue became a nuisance for users.

A very similar issue, also involving X11 (just by chance, it’s not that all the issues involve X11) is this particular bug in Xorg that is triggered when launching SDL-based applications, because libSDL bundles ancient versions of X11 libraries.

As I said earlier, unbundling is rarely easy; there are subtle issues to be checked out, for instance one has to check if there are changes at all beside eventual build-system related things (for instance to avoid using a full-fledged ./configure), but altogether it’s usually not tremendously impossible. Of course one has to stop thinking “Oh my, what if a library changes and the software breaks?”, otherwise the task gets impossible. Software changes, software bitrots. It’s not by bundling internal copies of libraries that you can stop that. When the compiler gets upgraded, you’re going to have your software break, and you should fix your software; if the C library cleans up the includes, your software might not compile or might misbehave, deal with it. Sometimes the bundled libraries implement protocols and formats that need to work together with some other piece of software; if that changes, the bundled libraries are just going to break further.

Your software is rarely special enough that you can be exempted from following the rules. Even OpenOffice is using lots of system libraries nowadays!

Bundling and modifying libraries is just like forking a project, and forking might not always be the best approach ; sometimes with dead upstream for a project, forking is your only hope; but even in those cases there are nice ways to bundle libraries. If you look at the way nmap bundles libdnet, you can see that they not only document all the changes they made to the library, but also provides splitted down and commented patches for the various changes, making it possible to adapt it to their need.

For proprietary software packages, of course, the matter is different, since you cannot usually unbundle the libraries yourself; but it’s a good idea to ask upstream nicely if they can use system copies instead of internal ones. Mind you, some might be happy to fix their packages not to be vulnerable any longer. Although I guess lots of them might actually prefer to keep them as they are since it’s a cost to them. One more reason not to trust them, to me.

So bottom line, if you’re working on an ebuild for a new piece of software to submit for Portage addition, please look well to see if the software is bundling libraries, and if it is, don’t let it enter portage that way. And if you’re a developer who wants to push some ebuild to the tree, also remember to ensure that it complies with our policies and doesn’t bundle in libraries.

I’m running Gnome

As it turns out, I start to dislike the way the KDE project is proceeding, and I don’t refer to the Gentoo KDE project, but to the whole of KDE project.

I dislike the way KDE 4 is being developed, with a focus on eyecandy rather than on features. This is easily shown by the Oxygen style; not only it is taking up a amount of screen real estate for widgets that remind me of Keramik (and if you remember, one thing that made happy a huge amount of users was the switch from Keramik to Plastik as default style in KDE 3.3), but it’s also tremendously slow. And I’m sure of this, it’s not just an impression: as soon as I switch Qt to use Oxygen, it takes five seconds for Quassel to draw the list of buffers; once I use QtCurve, it takes just one second. I don’t know if this is because Enterprise is using XAA and not EXA, but it certainly doesn’t look like something that the default theme should do.

And no, I’m not expected to use a computer that has less than an year, with a hyper-strength gaming videocard to be able to use KDE.

But this is just one of the issues I have with KDE recently. There are some policies I really, really, dislike in KDE. The first is one I already mentioned quite often and it’s the move to CMake. The only “good” reason to move to CMake is to be able to build under Windows using Microsoft’s Visual C++ compiler; yet instead of just saying “we needed cmake because it’s necessary to build for Windows” I see so many devs saying “cmake is just better than everything else out there”. Bullshit.

The other policy that I dislike regards the way KDE is developed and released as a single, huge, monolithic thing. One of the things that made KDE difficult to package in Gentoo (and other source-based distributions) was the fact that by default the source has to be built in those huge amorphous packages. And if the autotools-based build system of KDE sucked so much, it was also because of that.

But even if we leave alone the way the releases are made, it’s just not possible for everything to fall into a single release cycle kind of thing. There are projects that are more mature and projects that are less. Forcing all of them in a single release cycle makes it difficult to provide timely bugfixes for the mature projects, and makes it impossible for the not-so-mature projects to be tested incrementally. The last straw I could bear to see because of this stupid way of releasing, was knowing that Konversation in KDE 4 will probably lose the IRC-over-SSL support because KSSL was removed from the base libraries.

And now KDE 4.1 is on the verge of release, and Kopete still segfaults once you connect to Jabber. Yet when I tried (multiple times) to gather information about the possible cause in #kopete (so I could at least try to debug it myself), I had no feedback at all; maybe it’s because I run Gentoo, although the same happens on (K)Ubuntu. Yeah not the kind of people I like to deal with.

I’m not saying that I think Gnome is perfect for policies and other things. I dislike the fact that it’s always more Linux-/Solaris-centric than cross-platform centric; but I think KDE4 was a set back for that too, for what I read. And their release method does look a lot more sane.

I started using Linux with KDE 2. I moved to Gnome once KDE 3 was being taken care of. I came back to KDE just a bit before 3.3 release. Now I’m going to try Gnome for a while, and if I like it, I’ll think more than twice before going back to KDE. Yeah sure I liked KDE 3 better than I liked Gnome before that, but it’s just not feasible that I have to switch DE every time they want to make a new release.

Besides, since I last used it, Gnome seems much more mature and nicer to deal with.