Since my blog post about forced --as-needed yesterday, I started building the whole portage in a chroot to see how many packages break with the forced --as-needed at build time. While build-time failure are not the only problem here (for stuff like Python, Perl and Ruby packages, the failure may well be at runtime), build-time failure are probably the most common problem with --as-needed; if we also used --no-undefined, like Robert Wohlrab is trying (maybe a little too enthusiastically), most of the failures would be at build time by the way.
As usual, testing one case also adds tests for other side cases, this time I’ve added further checks to tell me if packages install files in /usr/man, /usr/info, /usr/locale, /usr/doc, /usr/X11R6, …and filed quite a few bugs about that already. But even without counting these problems, the run started telling me some interesting thing that I might add to the --as-needed fixing guide when I get back to work on it (maybe even this very evening).
I already knew that most of the failures I’d be receiving would be related to packages that lack a proper buildsystem and thus ignored LDFLAGS up to now (included --as-needed), but there are a few notes that really gets interested here: custom ./configure scripts seem to almost always ignore LDFLAGS and yet fail to properly link packages; a few glib-based packages fail to link to libgthread the main executable, failing to find g_thread_init(); and a lot of packages link the wrong OpenSSL library (they link libssl when they should link libcrypto).
This last note, about OpenSSL libraries, is also a very nice and useful example to show how --as-needed helps users in two main ways. Let’s go over a scenario where a package links in libssl instead of libcrypto (since libssl requires libcrypto, the symbols are satisfied, if the link is done without --as-needed).
First point: ABI changes: if libssl changed its ABI (happens sometimes, you know…), but libcrypto kept the same, the program would require an useless rebuild: it’s not affected by libssl ABI, but by libcrypto’s.
Second point, maybe even more useful at runtime: when executing the program, the libraries in NEEDED are loaded, recursively. While libssl is not too big, it would still require loading one further library that is unneeded to the program since libcrypto is the one that is actually needed. I sincerely don’t know if libssl has any constructor functions, but when this extra load happens with libraries that have many more dependencies, or constructor functions, it’s going to be a quite huge hit for no good reason.
At any rate, I wish to thank again all the people who contributed to pay for Yamato, as you can see the horsepower in it is being put to good use (although I’m still just at app-portage); and just so I don’t have to stress it to do the same work over and over again, I can tell you that some of the checks I add to my chroots for building are being added to portage, thanks to Zac, who’s always blazing fast to add deeper QA and repoman warnings, so that further mistakes don’t creep into the new code.. one hopes, at least.
Oh and before people start expecting cmake to be perfect with --as-needed, since no package using it has been reported as failing with --as-needed … well the truth is that I can’t build any package using cmake in that chroot since it fails to build because of xmlrpc-c. And don’t get me started again on why a build system has to use XML-RPC without a chance for the user to tell “not even in your dreams”.
So, what is the issue with xmlrpc? I suppose I could add a switch to turn it off. It will disable the ability for the testing system to send data to Dart2, but at this point CDash is the only thing folks are using any more. However, I am still confused as to why the use of this package is seen as such a sin… 🙂 So, if I add an option to CMake to disable xmlrpc testing submissions then cmake will handle –as-needed? They seem like separate issues.
xmlrpc-c has a pretty complex build system which is not @–as-needed@ safe. Since xmlrpc-c with @–as-needed@ lacks a libxml2 link, cmake will fail its own linking phase (because adding -lxml2 there is going to be ignored, cmake does not need libxml2, xmlrpc-c does). Get rid of xmlrpc-c in cmake and it could finally build on a system with forced @–as-needed@.And why I do find nasty that CMake requires xmlrpc-c is that on my systems xmlrpc-c is _never_ used… CMake is bringing it in just for itself (and no, using the internally bundled version is worse), and for a feature (Dart2 as you said) that is almost never used and certainly never used in my systems. The same most likely holds true for most users.
The problem is the “most likely” part of that, we work hard to maintain backwards compatibility for the CMake user base. I agree in this particular case xmlrpc seems little used, and I will work on deprecating it. However, if by some chance you were using xmlrpc-c in some other project on your system, then you would not find the use of it by CMake less objectionable? What if you did not have curl installed on your machine? Then would the use of curl be bad? If we allowed CMake to be built without curl, the testing sub-system would not work, but you could still build stuff with CMake.It really is not CMake’s fault that xmlrpc is not as-needed safe. Are all projects that use xmlrpc nasty? For example php seems to be using xmlrpc, is php bad? Seems like the complaint should be directed at the maintainers of xmlrpc instead of CMake. Is there a blacklist of non-as-needed projects to be avoided in the future?
It’s really not that easy on this side either. First of all, yeah I would debate about the usefulness of requiring curl too. Why does the testing subsystem _require_ curl? I can understand it wouldn’t be able to submit results without it, but I sincerely find that a desirable feature (to be able _not_ to ever submit tests results).I sincerely can’t think out of the blue of any software in mainline that is requiring xmlrpc-c, in mandatory way. Sure PHP _can_ use it, but it’s not mandatory, you can have PHP without using xmlrpc-c (and I sincerely would prefer it that way remembering how many bad things happened with PHP and XML-PC in the past), but that’s not even the point.I would object to GNU make requiring any network-related library too. Why? Because it’s a _build system_. Okay, the test submission is a feature that you actually are interested in, but does everybody who uses CMake have it? And why should user submit their test results by default?I don’t like the fact that CMake decides by default that the results submission is a needed feature.But in general I find nasty any mandatory dependency for things that bring in extra requirements (like network connection for instance) for packages that would otherwise work without that.