Testing stable; stable testing

It might not be obvious but my tinderbox is testing the “unstable” ~x86 keyword of Gentoo. This choice was originally due to the kind of testing (the latest and greatest version of packages for which we don’t know the averse effects of), and I think it has helped tremendously, up to now, to catch things that could have otherwise bit people months, if not years later, especially for the least-used packages. Unfortunately that decision also meant ignoring for the most part the other, now more common architecture (amd64 — although I wonder if the new, experimental amd32 architecture would take its place), as well as ignoring the stable branch of Gentoo, which is luckily tracked by other people from time to time.

Lacking continuous testing though, what is actually considered stable, sometimes it’s not very much so. This problem is further increased by the fact that sometimes the stable requests aren’t proper by themselves: it can be an user asking to stable a package, and the arch teams being called before they should have, it might be an overseen issue that the maintainer didn’t think of, or it might simply be that multiple maintainers had different feelings about stabilisation, which happens pretty often in team-maintained packages.

Whatever the case, once a stable request has been filed, it is quite rare that issues are brought up that are severe enough the stable is denied: it might be that the current stable is in a much worse shape, or maybe it’s a security stable request, or a required dependency of something following one of those destinies. I find this a bit unfortunate, I’m always happy when issues are brought up that delay a stable, even if that sometimes means having to skip a whole generation of a package, just as long as they mean having a nicer stable package.

Testing a package is obviously a pretty taxing task: we have numbers of combination of USE flags, compiler and linker flags, features, and so on so forth. For some packages, such as PHP, testing every and each combination of USE flags is not only infeasible, but just impossible within this universe’s time. to make the work of the arch team developers more bearable, years ago, starting with the AMD64 team, the concept of Arch Testers was invented: so-called “power users” whose task is to test the package and give green light for stable-marking.

At first, all of the ATs were as technically capable as a full-fledged developer, to the point that most of the original ones (me included) “graduated” to full devship. With time this requirement seems to have relaxed, probably because AMD64 became usable to the average user, and no longer just to those with enough knowledge to workaround the landmines originally present when trying to use a 64-bit system as a desktop — I still remember seeing Firefox fail to build because of integer and pointer variable swaps, once that became an error-worthy mistake in GCC 3.4. Unfortunately this also meant that a number of non-apparent issues became even less apparent.

Most of these issues often end up being caused by one simple fault: lack of direction in testing. For most packages, and that includes, unfortunately, a lot of my packages, the ebuild’s selftests are nowhere near comprehensive enough to tell whether the package works or not, and unless the tester actually uses the package, there is little chance that he knows really how to test it. Sure that covers most of the common desktop packages and a huge number of server packages, but that’s far from the perfect solution since it’s the less-common packages that require more eyes on the issues.

What the problem is, with most other software, is that it might require specific hardware, software or configuration to actually be tested. And most of the time, it requires a few procedures to be applied to ensure that it is actually tested properly. At the same time, I only know of Java and Emacs team publishing proper testing procedures for at least a subset of their packages. Most packages could use such a documentation, but it’s not something that maintainers, me included, are used to work on. I think that one of the most important tasks here, is to remind developers when asking stable to come up with a testing plan. Maybe after asking a few times, we’ll get to work and write up that documentation, once and for all.

4 thoughts on “Testing stable; stable testing

  1. The saddest part of this story is that all the job done (providing ebuilds, providing patches to remove bundled libs, providing feedback upstream and providing extensive TESTS on real SCADA systems with a couple of millions per minute of messages to route) for the package that started the userrel discussion was done by me without the help of nobody, than a new co-maintainer wanted to help, I welcomed him, but after a couple of emails he stopped to collaborate; he never contacted me by IRC or other media but he simply asked me feedback by bugzilla which I never had the time to reply because after less than seven days he acted on his own and closed the reports (testing ZeroMQ requires a lot of time however, don’t except “quick feedback” from me). I’m fine with it, he has done the job properly, but it was done by completely ignoring me and, most importantly, wasting all my efforts (why duplicate the efforts? If you want to do something I’ll be glad to delegate you some of my duties, but if you don’t speak and don’t collaborate I cannot suppose things).The same thing happened for the stabilization, nobody asked me if it was OK to do it and suddenly I had to spent time to teach to an AT how to do his job, how to test the package despite the fact I have repeatedly told him that the package was impossible to test just by the compilation per se – as nothing in Portage is using that library. The problem isn’t the stabilization of that package that pissed me off, but the lack of communication, cooperation and SUPPORT from other Gentoo devels (except from you Diego, obviously); I’m donating my time to Gentoo for free, I can be fine if nobody thanked me (sigh!) for the contributions done and it’s also OK to get, from time to time, some random insults from Gentoo developers, but almost don’t waste my time because it’s sad and really unfair, and it’s even more sad and unfair if then people call me a slacker as an excuse to cover their responsibilities. It’s REALLY SAD.It’s a general Gentoo problem and a really big issue at this point; bugzilla and mailing-lists are NOT the proper tools to manage a project and to share feedback between people, especially bugzilla is sub-optimal for developers coordination and is the source of quite all the troubles that Gentoo has faced in the last years (and I’m following Gentoo since the beginning, since 2004).At the moment Gentoo is more like a biggest-dick-competition rather than a collaborative effort; sorry guys, I had enough of all these bullshits and I give up, good luck for the future.

    Like

  2. I’ve given this some consideration but after lurking via email the stabilization requests the sheer volume of packages is extremely daunting to say the least.I like the latest ‘proxy maintainership’ idea. The whole collaborative and bug fixing approach needs thorough examination and some protocol. There just never will be enough devs and Gentoo needs to leverage their users better somehowI’ve been trying tools to test static source. cppcheck seems the easiest to use so far and actually gives informative results with minimal effort. Consider adding it to your toolbag if you haven’t But you have to pull in a ‘git’ and test the latest source as the versions ebuilds are using usually far behind and bug reports on those aren’t usually welcomed.

    Like

  3. despite all problems and bullshit that happens, thanks for you work on gentoo!hope a new generation of developers will join gentoo that is more open minded to get over the biggest dick competition. Please don’t stop to name the problems, someone maybe come up with a solution some day.What were better tools to manage a project like gentoo when bugzilla and the mailing list does not fit our needs?

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s