Some details about our old friends the .la files

Today, I’m going to write about the so-called “libtool archives” that you might have read about in posts like What about those .la files? or Again about .la files (or why should they be killed off sooner rather than later) (anybody picking up the faint and vague citation here is enough of a TV geek).

Before starting I’m going to say that I’m not reading any public Gentoo mailing list lately which means that if I’m going to bring out points that have been brought up already, I don’t care. I’m just writing this for the sake of it and because Jorge asked me some clarification about what I did write some longish time ago. Indeed the first post is almost exactly one year ago. I do know that there has been more discussion about the need or not need of these libraries for ebuild-provided stuff, so I’m going to try to be as clear as possible.

First problem is to identify what these files are: they are simple text files, and they provide metadata about a library, or a pair of static and shared libraries; this metadata includes some obvious and non-obvious stuff like the names of the two type of libraries, the formal name (soname) of the shared library that can be used with dlopen() and a few more things included the directory the library is to be found in. The one piece of data that creates a problem for us is, though, the dependency list of libraries that needs to be linked against when linking against this library, but I’ll go back to that later. Just please note that it’s there.

Let’s go on with what these files are used for: libtool generates them, and libtool consumes them; they are used when linking with libtool to set the proper runpath if needed (by checking the library’s directory), to choose between the static or shared library version, and to bring in further dependency libraries. This latter thing is controversial and our main issue here: older operating systems had no way to define dependencies between libraries of any kind, and even nowadays on Linux we have no way to define the dependencies of static libraries (archives). So the dependency information is needed when using static link; unfortunately libtool does not ignore the dependencies with shared objects, which can manage their dependencies just fine (through the DT_NEEDED tag present in the .dynamic section), and pollutes the linking line causing problems that can be solved by --as-needed.

Another use for libtool archive files is to know how to load or preload modules (plugins). The idea is that when the operating system provides no way to dynamic load and link further shared objects (that is, a dlopen()-like interface), libtool can simulate that facility by linking together the static modules, using the libltdl library instead. Please note, though, that you don’t need the libtool archives to use libltdl if the operating systems does provide dynamic loading and linking.

This is all fine and dandy for theory, what about practicality? Do we need the .la files installed by ebuilds? And slightly related (as we’ll see later) do we need the static libraries? Unsurprisingly, the answer comes from the developer’s mantra: There is no Answer; it Depends.

To simplify my discussion here, I’m going to reduce the scope to the two official (or semi-official) operating systems supported by Gentoo: Linux and FreeBSD. The situation is likely to be different for some of the operating systems supported by Gentoo/Prefix, and while I don’t want to reduce their impact, the situation is likely to become much more complicated to explain by adding them to the picture.

Both Linux and FreeBSD use as primary executable and linkable format ELF (which stands exactly for “executable and linkable format”), they both support shared objects (libraries), the dlopen() interface, the DT_NEEDED tag, and they both use ar flat archives for static libraries. Most importantly, they use (for now, since FreeBSD is working toward changing this) the same toolchain for compile and link, GCC and GNU binutils.

In this situation, the libltdl “fake dlopen()” is sincerely a huge waste of time, and almost nobody use it; which means that most people wouldn’t want to use the .la files to open plugins (that is, with the exception of KDE 3), which makes installing libtool archives of, say, PulseAudio’s plugins, pointless. Since most software is likely not to use libltdl in the first place, like xine or PAM to cite two that I maintain somehow, their plugins also don’t need the libtool archive files to be installed. I already have reported some rogue pam modules that install pointless .la files (even worse, in the root fs). The rule of thumb here is that if the application is using plugins with standard dlopen() instead of libltdl (or an hacked libltdl like it’s the case for KDE 3), the libtool archives for these plugins are futile; this, as far as I know, includes glib’s GModule support (and you can see by using ls /usr/lib*/gnome-*/*.la that there are some installed for probably no good reason).

But this only provides a description of what to do with the libtool archive files for plugins (modules) and not with libraries; with libraries the situation is a bit more complicated, but not too much, since the rule is even simpler: you can drop all the libtool archives for libraries that are only ever shared (no static archive provided), and for those that have no dependency other than the C library itself (after making sure it does not simply forget to link the dependencies in). In those cases, the static library is enough to be listed by itself, and you don’t need any extra file to tell you what else is needed to be linked in. This already takes care of quite a bit of libtool files: grepping for the string “dependency_libs=''” (which is present in libraries that don’t have any further dependencies than the C library), provides 62 files in my system.

There is another issue that was brought up last year: libraries whose official discovery method is a -config script, or pkg-config; these libraries can ignore the need for providing dependencies for the static variant since they provide it themselves. Unfortunately this has two nasty issues: the first is that most likely someone is not using the correct scripts to find the stuff; I’ve read one blog post a week or two ago about a developer disgruntled because pkg-config was used for a library that didn’t provide it and suggested not to use pkg-config at all (which is quite silly actually). The other problem is that while pkg-config does provide a --static parameter to use different dependency lists between shared and static linking of a library (to avoid polluting the link line), I know of no way to tell autoconf to use that option during discovery at all. But there are also a few things that can be said, since there is enough space for binutils to just implement an extension to the static archives that can actually provide the needed dependency data, but that’s beside the point now I guess.

So let’s sidestep this issue for now and return to the three known cases when we can assert with a relative certainty that the libtool archives are unneeded: non-ltdl-fakeloaded plugins (xine, pam, GModule, …), libraries with no other dependencies than the C library and libraries that only install shared objects. While the first two are pretty obvious, there is something else to say about that last one.

By Gentoo policy we’re supposed to always install both the static and shared object version of a library; unless, that is, upstream doesn’t think so. The reason for this policy is that static linking is preferred for some mission-critical software that might not allow the system to boot up if the library is somehow broken (think bash and libreadline), and because sometimes, well, you just have to leave the user with the option of static linking stuff. There have been proposals of adding an USE flag to allow enabling/disabling build of static libraries, but that’s nowhere to use, yet; one of the problems was to link up the static USE flag with the static-libs USE flag of its dependencies; EAPI 2 USE dependencies can solve that just fine. There are, though, a few cases where you might be better off not providing a static library at all, even if upstream doesn’t say something outright about it, since most likely they never cared.

This is the case for instance of libraries that use, in turn, the dlopen() interface to load their plugins (using static linking with those can produce nasty results); for instance you won’t find a static library for Linux-PAM. There are a few more cases where having static libraries is not suggested at all and we might actually decide to take it out entirely, with the due caution. In those cases you can remove the libtool archive file as well since shared objects do take care of themselves.

Now, case in point, Peter took lots of flames for changing libpcre; there are mixed flames related to, from one side, him removing the libtool archive, and from the other to him removing the static library. I haven’t been part of the flames, in any way, because I’m still minding my own health first of all (is there any point in having a sick me not working on Gentoo?), yet here is my opinion: Peter did one mistake, and that was to unconditionally remove the static library. Funnily enough, what probably most people shouted him at for is the removal of the libtool archive, which is just nothing useful since, you can guess, the library has no further dependency beside the C library (it’s different for what regards libpcreposix, though).

My suggestion at this point is for someone to actually finish cleaning up the script that I think I posted to the mailing lists some time ago, and almost surely can be recreated quite quickly, that takes care of fixing the libtool archive files in the system without requiring a full rebuild of everything. Or even better get a post-processing task for Portage that replaces the direct references to libtool archives in the new libtool archives with generic library references (so that /usr/lib/libpcre.la would become -lpcre straight away and so on for all libraries; the result would be that breakage for future libtool archives removal wouldn’t exist at all).