Enabling –as-needed, whose task is it?

A “fellow” Gentoo developer today told me that we shouldn’t try to get --as-needed working because it’s a task for upstream (he actually used the word “distributors” to intend that neither Gentoo nor any other vendor should do that, silly him)… this developer will go unnamed because I’ll also complain right away that he suggested I give up on Gentoo when I gave one particular reason (that I’ll repeat in a moment) for us to do that instead. Just so you know, if it was up to me, that particular developer right now would have his access revoked. Luckily for him, I have no such powers.

Anyway, let me try to put in proper writing why it should fall to Gentoo to enable that by default to protect our users.

The reason I gave above (it was a twitter exchange so I couldn’t articulate it completely), is that “Fedora does it already”. To be clear, both Fedora and Suse/Novell do it already. I’m quite sure Debian doesn’t do it, and I guess Ubuntu doesn’t do that either to keep in line with Debian. And the original hints we took to start with --as-needed came from AltLinux. This alone means that there is quite a bit of consensus out there that it should be a task for distributors to look at. And it should say a lot that the problems solved by --as-needed are marginal for binary distributions like the ones I named here; they all do that simply to reduce the time needed for their binary packages to rebuild, rather than to avoid breaking users’ systems!

But I’m the first person to say that the phrase “$OtherDistribution does it, why don’t you?” is bogus and actually can cause more headaches than it solves. Although most of the time, this is meant when $Otherdistribution is Ubuntu or a relative of theirs. I seriously think that we should take a few more hints from Fedora; not clone them, but they have very strong system-level developers working on their distributions. But that’s a different point altogether by now.

So what other reasons are there for us to provide --as-needed rather than upstream? Well, actually this is the wrong question; the one you should formulate is “Why is it not upstream’s task to use --as-needed?”. While I have been pushing --as-needed support in a few of my upstream packages before, I think by now that it’s not the correct solution. It all boils down to who knows better whether it’s safe to enable --as-needed or not. There are a few things you should assess before enabling --as-needed:

• does the linker support --as-needed? it’s easier said than done; the linker might understand the flag, but supporting is it another story; there are older versions of ld still in active use that will crash when using it; other with a too greedy --as-needed that will drop libraries that are needed, and only recently the softer implementation was added; while upstream could check for a particualr ld version, what about backports?
• do the libraries you link to, link to all their needed dependencies? one of the original problems with --as-needed when introduced to the tree was that you’d have to rebuild one of the dependencies because it relied on transitive linking, which --as-needed disallowed (especially in its original, un-softened form); how can a given package make sure that its dependencies are all fine before enabling --as-needed?
• do the operating system at all support --as-needed? while Gentoo/FreeBSD uses modern binutils, and (most of) the libraries are built so that all the dependencies are linked in for --as-needed support, using it is simply not possible (or wasn’t possible at least…), because gcc will not allow for linking the threading libraries in for compatibility with pre-libmap.conf FreeBSD versions; this has changed recently for Gentoo since we only support more recent versions of the loader that don’t have that limitation; even more so, how can upstream know whether the compiler will have the fix already or not?

Of course you can “solve” most of the doubts by running runtime tests; but is that what upstreams should do? Running multiple tests from multiple packages require sharing the knowledge and risks for the tests to get out-of-sync one with the other; you have redundancy of work.. when instead the distributor can simply decide on whether using --as-needed is part of their priorities or not. It definitely is for Fedora, Suse, AltLinux… it should be for Gentoo as well, especially as a source-based distribution!

Of course, you can find some case-by-case where --as-needed will not work properly; PulseAudio is unfortunately one of those, and I haven’t had the time to debug binutils to see why the softer rules don’t work well in that case. But after years working on this idea, I’m very sure that it’s a very low percentage of stuff that fails to work properly with this, and we should not be taken hostage by a handful of packages out of over ten thousands!

But, when you don’t care about the users’ experience, when you’re just lazy or you think your packages are “special” and deserve to break any possible rule, you can afford yourself to ignore --as-needed. Is that the kind of developers Gentoo should have, though? I don’t think so!