For the tinderbox I’ve been adding quite a few tests that are not currently handled by Portage; some of these tests as I said before are taxing, and all of them are linear, serial, which means that even with the huge amount of power Yamato they take quite some time (here the only thing that does help is the memory that allows for more files to be kept in cache).
But even more difficult than implementing the tests is to come up with them. As I said, they are not in Portage for one reason or another: too specific, too taxing, possibly disruptive and so on. Not all of them are my ideas, some are actually suggestions by other developers for particular cases, or by users.
For instance, while I added the tests for /usr/man, /usr/doc and /usr/local directories, the one for the misplaced Perl modules (site dir rather than vendor dir) is something that tove came up with (and I’m filing bugs about that to help him transition site_dir away); and the make “cheat” to call parallel build anyway is something Kevin Pyle came up with.
I came up with the “ELF file in /usr/share” because I had seen Portage stripping some in those path which I knew shouldn’t be there; and from that, I started wondering about the binchecks restriction, so I added the (very slow!) test that ensures that bincheck-restricted packages don’t install ELF files at all (actually it seems like all Linux kernel sources do install some ELF files, I need to track down if it’s a Gentoo problem or if upstream packages them up like that).
The test for bundled libs is very shallow (and leads to false positive on the packages that do provide those libraries of course) but it tries to identify the most common offenders, which is something quite important given that a nasty weakness was found on libtool’s libltdl which is among those most often bundled with software.
The notice on setXid files is something that Robert (rbu) asked me to add so that it would be possible to get a list of setXid executables in Gentoo (just getting them off the livefs of the tinderbox is not possible because you never ever cannot get all packages merged in the same system).
Recently I’ve added one check for libtool .la files so that I can find at least some certainly unhelpful ones (installed with Ruby, Python and Perl extensions: none of those use ltdl to load them so none of those will need the extra file beside the shared object).
If you got any idea for QA tests that you think the tinderbox should be running, you can either mail them to me or leave them as a comment here, just remember that they need to have a fair balance so that I don’t get one out of three to be a false positive, and they need to be efficiently executed over all the packages in the tree.
On the subject of new tests: I would like to see an automated reporting of packages which fail to honor CFLAGS/LDFLAGS. Detecting it is relatively easy with some specfile tricks, but I have not yet put much thought into how to avoid reporting bugs for ebuilds that intentionally trash user CFLAGS, unless you want to consider such disrespect to be a bug, but at least for those there is evidence that the Gentoo maintainer intended to disrespect the user flags. My first interest is in hitting the packages where upstream simply trashed the flags for no good reason. I will dust off those specfile tricks and mail them to you. I am not sure how well a specfile will survive blog comment formatting. ;)A check that reports packages which disable _FORTIFY_SOURCE would be nice. There may be legitimate reasons for some packages to disable it, but I expect there are some that just turned it off as the path of least resistance when it broke badly written constructs, rather than fixing the bad construct. Again, specfile tricks should catch this.As a general tinderbox enhancement, it might be worth setting the environment variables that make CMake and friends chatty. Some of your build logs have been less than fully helpful because the upstream build system was too terse to see what it was doing when it failed.Your reporting of packages calling bare gcc/g++ seems to have fallen off. Is this because you gave up on that check, or because you have stopped hitting affected packages?
Both my previous CFLAGS-not-respected and CC-ignored series of bugs have been dropped in the current run of the tinderbox for a few reasons: * the amount of false positive made it very difficult to properly address the issues (should we always export CC? or just respect it if it’s set explicitly?); * the “CFLAGS test”:https://blog.flameeyes.eu/2… in particular tended to be disruptive because packages don’t really seem to understand all flags that well; * it also brought in some very strange results of null-compiled object files, especially in relation to @AC_REPLACE_FUNCS@ calls; * the gcc/g++ testing was vastly ignored by a number of developers, so the result wasn’t that great in general.