I’ve already written something about automation for Gentoo bug search, but I think sometimes it’s not easy to understand that just using a huge tinderbox, even distributed, is not going to help much to make sure that software works. The problem is that sometimes, even if software builds fine, it’s just going to break at runtime, and even though tests help, when they are handled properly, they are far from complete solutions.
Problems like the one described in the post above for hfsplusutils are just impossible to gather at build time without running the tests on sample data (which reminds me of the uif2iso problem); but this can get even more subtle: while it’s true that most --as-needed failure happen at buildtime, there are quite a few that will not hit until runtime; one such cases happened to me with libcompizconfig, compizconfig-python and fusion-icon: this last one software wouldn’t start because Python failed to load the second, because the first one wasn’t linking to libX11.
Now of course this could have been found if either libcompizconfig or compizconfig-python had a testsuite, but since I already said that running a tinderbox run with testsuite is probably not something that I would like to do on a daily basis.
Especially for old software, there are problems like endianness issues, 64-bit arches and PIC code that are almost impossible to figure out at buildtime, and that need to be checked during software use. But it’s not just that. For binary packages, especially those of proprietary software, there usually isn’t a testsuite, for this reason, their executables are rarely checked at runtime for consistency. This becomes a problem because sometimes you have software that links against old versions of libraries. This is the same reason why adding VirtualBox 2.1.0 binary package in the tree is going to take a while for Alessio: it uses the old ABI of libcap, which will require resurrecting, and maintaining, an old verison of libcap just for that. (I have a few more issues to talk about regarding the new VirtualBox release but I’ll get to that in another moment).
And yet, the reasons why neither my nor Patrick’s tinderbox can be a replacement for a more throughout approach to packages testing are not finished here. But before proceeding to more, I have to make a distinction between the different approaches me and Patrick took. Patrick’s tinderbox removes all the superfluous packages from the system when installing a new one, which is very good to test for missing dependencies; my method instead iterates over each of the packages in the tree and installs them one by one in the system, filling up the space, which can easily ignore missing dependencies but provides more interesting results regarding iteration of particular ebuilds.
So while my method glosses over broken runtime and buildtime dependencies, like pkg-config not depended upon and similar, Patrick’s method is not going to hit problems like dev-scheme/chicken breaking most of the Mono packages (that would pick up /usr/bin/csc as the C# compiler rather than mcsc), or collisions between unrelated packages.
This means that either one of the two tinderboxes is just not enough to find all the issues, and even the two of them together won’t be enough. Even adding AutoTua to that, it’s just not going to cut it. As Jeremy said on a blog post of mine, we need humans (developers and users) to report issues. I start to feel we also have a need for some real numbers of how many users use packages. Yes I know that’s going to be a popularity contest, and it’s likely that there will be people that would just go on to submit fake results, but even for tree cleaning, it’s important to know whether packages don’t have bugs failed against them because they are good, or just because nobody has used them in so much time.
Oh and so that you know, I currently have little less than 1500 bugs open that I reported (and over 3000 bugs that I reported since I started contributing to Gentoo), and all of them are reported by hand, there are still issues that force me not to use scripts like pybugz. I’ll see to write about them, maybe Zac can see to find a solution to those, like he has been doing quite a while lately for me. Thanks Zac!