Quite some weeks ago, Markos stated asking for a more thorough testing of ebuilds to check whether they support LDFLAGS; he then described why that is important which can be summarised in “if LDFLAGS are ignored, setting --as-needed by default is pointless”. He’s right on this of course, but there are a few more tricks to consider.
You might remember that I described an alternative approach to --as-needed that I call “forced --as-needed”, by editing the GCC specifications. The idea is that by forcing the flag in through this method, packages ignoring LDFLAGS and packages using unpatched libtool won’t simply ignore it. This was one of the things I had in mind when I suggested that approach, but alas, as it happens, I wasn’t listened to.
Right now there are 768 bugs reported blocking the tracker of which 404 are open and, in the total, 635 were reported by me (mostly through the tinderbox but not limited to). And I’m still opening bugs on a daily basis for those.
But let’s say that these are all the problems there are, and that in two weeks time no package is reporting ignored LDFLAGS at all. Will that mean that all packages will properly build with --as-needed at that point? Not really.
Unfortunately, LDFLAGS-ignored warnings have both false positives (prebuild binary packages, packages linking with non-GNU linkers) and false partially-negatives. To understand that you have to understand how the current warning works. A few binutils versions ago, a new option was introduced, called --hash-style; this option is used to change the way the symbol table is hashed for faster loading by the runtime linker/loader (ld.so); the classic hashing algorithm (SysV) is not excessively good for very long, similar symbols that are way too common when using C++, so there has been some work to replace that with a better GNU-specific algorithm. I’ve followed most of the related development closely at the time, since Michael Meeks (of OpenOffice.org fame) actually came asking Gentoo for some help testing things out; it was that work that actually got me interested in linkers and the ELF format.
At any rate, while the original hash table was created in the .hash section, the new hash table, being incompatible, is in .gnu.hash. The original patch only replaced one with the other, but what actually landed in binutils was slightly different (it allows to choose between SysV, GNU, or both hash tables), and the default has been to enable both hash tables, so that older systems can load .hash and the new ones can load .gnu.hash; win win. On the other hand, on Gentoo (and many other distributions) where excessively old GLIBC versions are not supported at all, there is enough of a reason to use --hash-style=gnu to disable the generation of the SysV table entirely.
Now, the Portage warning is derived by this situation: if you add -Wl,--hash-style=gnu to your LDFLAGS, it will be checking the generated ELF files and warn if it finds the SysV .hash section. Unfortunately this does not work for non-Linux profiles (as far as I know, FreeBSD does not support the GNU hashing style yet; uClibc does), and will obviously report all the prebuilt binaries coming from proprietary packages. In those cases, you don’t want to strip .hash because it might be the only hash table preventing ld.so from doing a linear search.
So, what is the problem with this test? Well, let’s note one big difference between --as-needed and --hash-style flags: the former is positional, the second is not. That means that --as-needed, to work as intended, needs to be added before the actual libraries are listed, while --hash-style can come at any point in the command line. Unfortunately this means that if any package has a linking line such as
$(CC) $(LIBS) $(OBJECTS) $(LDFLAGS)
It won’t be hitting the LDFLAGS warning from Portage, but (basic) --as-needed would fail — OTOH, my forced --as-needed method would work just fine. So there is definitely enough work to do here for the next… years?
Well, there are actually two solutions.1. Your spec file approach gains enough support among Gentoo devs.2. Binutils flips the default state of the flag as enough distributions want to use it and software that actually requires –no-as-needed can easily be fixed.Actually, I’d like the second one more. 😉
Second one would make it even easier, but I don’t think that’s going to happen too soon unfortunately…
I thought we were supposed to be striving to minimize deviations from upstream and release packages as close to vanilla as possible. How does patching the spec files fly with that? :PI’ve been slowly making progress getting our GCC to resemble something closer to upstream. So, you know, they don’t immediately close bugs as invalid as soon as they see Gentoo in the version string. So any patching of the spec files would have to be for a very good reason, and frankly, in the grand scheme of things, LDFLAGS is really not that big of a deal.I would like to see the second one as well. You could always try floating a patch that makes the default a compile time option (they just added one for -fomit-frame-pointer to GCC).
Beside the fact that I’d probably smack someone’s head upstream if they were to close a bug as INVALID for a spec file change that does something that LDFLAGS already do.Beside the fact that I wouldn’t be surprised if either RedHat or SUSE already do something like that.Beside the fact that hardened already has modified spec files.It’s purely a squarely a matter of effort/results: * ensuring that all packages support LDFLAGS properly is a _very_ huge effort, and a good result; * changing binutils to enable it by default is going to be probably more effort than that to coordinate, especially with the new gold stuff; * changing the GCC spec file is relatively low effort and has the total result at once.Now, this doesn’t mean that we have to make the patch Gentoo specific; my solution above is a “nice dirty trick” to enable it; a proper patch would make GCC detect a “new enough Linux distro” or something (or pass it to configure obviously) and enable it on its own specs.What does it do right now for @_FORTIFY_SOURCE@? It shouldn’t be much different.
I think _FORTIFY_SOURCE is a much better reason though. But to each his own.I guess my viewpoint is this: you want to force –as-needed through spec files to find issues in packages that don’t respect LDFLAGS. I say fix the package to respect LDFLAGS and these issues will be immediately apparent. Plus you then have a patch to send upstream which actually has a benefit to anyone not hacking spec files/etc.But yeah, that’s still a huge undertaking.
Ryan, you’re showing by your own opinion why a spec file / binutils approach is best. Many people don’t care strongly about LDFLAGS or as-needed.Actually, most programs don’t really need to. It only makes a really major difference for widely used libs pulling in something special which is then changed. Great joy, 300 packages break for a change in a lib they don’t use at all.On the other hand, what’s the percentage of packages in the tree which will not work with –as-needed after a small patch as they really need to disable it? 0.3%? Less? If packages are simply forced to use the flag, quite a few will start to use as-needed where they ignored LDFLAGS before and nothing breaks as they did everything right already.
Few thoughts:- editing spec files for emerging stuff seems fine, but for user using gcc for his own stuff, wouldn’t this be unexpected change? Is it possible to have separate spec files active simultaneously for emerge and users?- is –as-needed in LDFLAGS really so pointless until EVERYTHING in tree supports it? Aren’t just relatively few system-wide often used libraries what’s most important to prevent mass rebuilds on ABI breakages, and some less used stuff has relatively negligible impact? BTW the hwoarang’s post is called “Why should a package respect LDFLAGS anyway” but I see no explanation of that there :)- could one create better test for the positional parameter problem by replacing ld with a wrapper that would check it and die/warn when there’s a problem? Would also remove most of the false positives you talk about.
a peek at bu#331531 will show that gcc-4.4.4-r2doesn’t support your LDFLAGS