For the tinderbox I’ve been adding quite a few tests that are not currently handled by Portage; some of these tests as I said before are taxing, and all of them are linear, serial, which means that even with the huge amount of power Yamato they take quite some time (here the only thing that does help is the memory that allows for more files to be kept in cache).
But even more difficult than implementing the tests is to come up with them. As I said, they are not in Portage for one reason or another: too specific, too taxing, possibly disruptive and so on. Not all of them are my ideas, some are actually suggestions by other developers for particular cases, or by users.
For instance, while I added the tests for /usr/man, /usr/doc and /usr/local directories, the one for the misplaced Perl modules (site dir rather than vendor dir) is something that tove came up with (and I’m filing bugs about that to help him transition site_dir away); and the make “cheat” to call parallel build anyway is something Kevin Pyle came up with.
I came up with the “ELF file in /usr/share” because I had seen Portage stripping some in those path which I knew shouldn’t be there; and from that, I started wondering about the binchecks restriction, so I added the (very slow!) test that ensures that bincheck-restricted packages don’t install ELF files at all (actually it seems like all Linux kernel sources do install some ELF files, I need to track down if it’s a Gentoo problem or if upstream packages them up like that).
The test for bundled libs is very shallow (and leads to false positive on the packages that do provide those libraries of course) but it tries to identify the most common offenders, which is something quite important given that a nasty weakness was found on libtool’s libltdl which is among those most often bundled with software.
The notice on setXid files is something that Robert (rbu) asked me to add so that it would be possible to get a list of setXid executables in Gentoo (just getting them off the livefs of the tinderbox is not possible because you never ever cannot get all packages merged in the same system).
Recently I’ve added one check for libtool .la files so that I can find at least some certainly unhelpful ones (installed with Ruby, Python and Perl extensions: none of those use ltdl to load them so none of those will need the extra file beside the shared object).
If you got any idea for QA tests that you think the tinderbox should be running, you can either mail them to me or leave them as a comment here, just remember that they need to have a fair balance so that I don’t get one out of three to be a false positive, and they need to be efficiently executed over all the packages in the tree.