One big problem with a huge tree like we have (which is not, in my opinion, solved by splitting it in multiple overlays!) is that it’s very difficult to clean up the small things that pile up. Knowing that my bug count in Bugzilla is already nearing 1200 again, to me it feels obvious enough.
For instance, tonight I filed a bug for a package that failed to build with a GCC 4.2 failure. Yes GCC 4.2; while it never went stable up to now, it certainly is not the latest version available, and one would expect that most of the software was ready to work with that at least.
Even more interesting, there are a few packages, with mirror restrictions turned on, that fails fetching; while packages failing to fetch with mirrors are easily found by the mirror admin, the same is not true, obviously, for those that don’t even pass through our mirrors, and there are a few, especially when it comes to games (but not limited to).
But there is more to that: the collisions between packages start to get pretty nasty; I had to mask quite a few packages to avoid blockers and removals; but a lot still fail to merge because of file collisions, for instance there’s sci-biology/emboss that is driving me crazy because it’s rebuilt many times a run and is never merged because of file collisions (this is where ccache does come useful).
But all these little things do pile up as I said, and it starts to become clear that we should find some way to remove the cruft as it starts to rot. Donnie suggested continuous tinderboxing of packages, but it becomes pretty difficult, to be honest; even for Yamato the whole rebuild takes weeks, and it’s building all day long on an eight-core system! (Which, I’d like like to remind, I’m not paid to do, and it’s actually start to feel like a money drain, so thank yous and kudos are welcome, as usual).
What would probably be nice is a way to have a central service where data could be collected so that different systems can be working together with testing different situations maybe even prioritising packages that don’t get tested often, or that have a known broken clean report.
Unfortunately even just the logs can easily be huge and can fill up an hard disk quite soon (between chroots, logs, virtual machines and the like, Yamato’s 2.1TB are almost full again already, I’ll go fetch two eSATA disks in the next days).
At any rate, I’ve been filing Portage feature request to make my current tinderbox setup easier to reproduce on other systems… hopefully one day we might have a working tinderbox script that is actually part of the development set. For now, I hope to get all the QA tests I’m using generic enough that we can have them run on normal merges too, so that developers see them, and users report them, right away without waiting for the tinderbox to reach that point of the tree.
One step at a time.