Gentoo binhosts notes

I’ve been meaning to write about this before, and since I was asked of something related by the Gechi I thought this was as good as a moment as it could be, for what concerns this topic. You probably all know that I run a tinderbox that is trying to ensure that the packages in Gentoo build, and work at least as intended. But as the description page says, the same name has been used by other projects, such as the official Infra one which provides logs, reverse dependency information, and binary packages for various architectures.

Given that, some people asked me before if I can provide binaries of the packages built by my tinderbox; the answer is “not a chance”, for a few reasons that are more complex than the average first glimpse would tell you. And it’s mostly beside the basic problem that Gentoo has very shabby support for binary packages, as both Fabio and Constanze could easily tell you.

In the set of problems, the issue at hands is licenses (and here the expert would probably be Matija as he’s quite interested in getting them right). Not only the tinderbox accept almost all the possible licenses that are available in Portage so that it can actually merge as many packages as are available, but it also does not turn the bindist USE flag on (which is used to make sure that the result is actually distributable; it disables feature that would link GPL-incompatible libraries into GPL software; it disables feature that are known patented and shouldn’t be used; it make sure that trademarks are respected — like Firefox). Both these issue make a good deal of generated binaries non-redistributable by themselves.

But that’s not all; even when the license would let me redistribute the binaries; copyleft licenses require me to redistribute the sources as well; just pointing at the Gentoo mirrors is not a good option there; I would have to provide all the downloaded and used distfiles, and all the ebuilds as used for build that binary. You might guess it’s not an easy task nor one that requires just a bit of online space.

Now, after you actually tackle all these issues, enabling the bindist USE flag, only accepting true opensource licenses that allow redistribution, and providing a way to publish all the sources together with the binaries, you’re still not set. The rest of the problems are actually technological ones, tied to the way my tinderboxing scripts are designed to work. Even without counting the fact that the flag combinations are so many that you have to limit to some sanity, actually building every package in the tree gives me a number of headaches, starting with a number of automagic dependencies which would make the binary packages unusable.

On a side note, I’ve been thinking for a while of getting a way to set up dependency verification to ensure that no automagic dependencies enter the ELF files at least; unfortunately this is not as straightforward as I’d like it to be. New-style virtuals mean that the dependency is hidden under a second layer of packages, which in turn would make it difficult to actually pinpoint the error conditions. Think of the PyCURL trouble that I pointed at a few weeks ago.

What would actually work to produce decently-working binary packages is the method that Patrick has been using for tinderboxing: build one by one each package with only its own detailed dependencies merged in. After all, this is the method that the Debian- and RPM-based distributions follow. This has the opposite side effect of maybe failing if an indirect (transitive) dependency is missing, which often happens with pkg-config, but at least you wouldn’t find automagic dependencies on the binary packages themselves.

And as I noted in passing above, there are a number of problems related directly with the way Portage manages binary packages by itself: it does not have a properly clean way to track actually needed dependencies (ABI-wise that is) – and let’s not even go about the tracking of python dependencies – and the format itself is not really flexible enough, which causes headaches when trying to deal with special features like file-based capabilities.

So, good luck if you want to provide public binhosts, myself I don’t have time nor will to even think about them, for the most part.