Library SONAME bumps and .la files: some visual clues

Before going on with the post, I’ll give users who’re confused by the post’s title some pointers on how to decipher it: I discussed .la files extensively before, and you can find a description of SONAMEs in another post of mine.

Long- and medium-time Gentoo users most likely remember what happened last time libpng was bumped last year, and will probably worry now that I’m telling them that libpng 1.5 is almost ready to be unmasked (I’m building the reverse dependencies in the tinderbox as we speak to see what breaks). Since I’ve seen through it with the tinderbox, I’m already going to tell you that it’s going to hurt, as a revdep-rebuild call will ask you to rebuild oh-so-many packages due to .la files that, myself, I’ll probably take the chance to move to the hardened compiler and run an emerge -e world just for the kicks.

But why is it this bad? Well, mostly it is the “viral propagation” of dependencies in .la files, which by itself is the reason why .la files are so bad. Since libgtk links to libcairo, and libcairo to libpng, any other library linking with libgtk will be provided with a -lpng entry to link to libpng, no matter whether it uses it or not. Unfortunately, --as-needed does not apply to libtool archives, so they end up overlinking, and only the link editor can drop the unused libraries.

For the sake of example, Evolution does not use libpng directly (the graphic files are managed through GTK’s pixbuf interface), but all of its plugins’ .la files will refer to libpng, which in turn means that revdep-rebuild will pick it up to rebuild it. D’oh!

So what about the visual clue? Well, I’ve decided to use the data from the gold based tinderbox to provide a graph of how many ELF objects actually link to the most common libraries, and how many libtool archives reference them. The data wasn’t easy to extract, mostly because at a first glance, the .la files seemed to be dwarfed by the actually linked objects.. until I remembered that ELF executable can’t have a corresponding .la file.

Library linking histogram

I’m sorry of some browsers might fail to display the image properly; please upgrade to a decent, modern browser as it’s a simple SVG file. The gnuplot script and the raw data file are also available if you wish to look at them.

The graph corroborates what I’ve been saying before, that the bump of libraries such as libexpat and libpng only is a problem because of overlinking and .la files. Indeed you can see that there are about 500 .la files listing either of the two libraries, when there are fewer than a hundred shared objects referencing them. And for zlib it’s even worse: while there are definitely more shared objects using it (348), there are four times as many .la files listing it as one of the dependencies, for no good reason at all.

A different story applies to GLib and GTK+ themselves: the number of shared objects using them is higher than the number of .la files that list them among their dependencies. I guess the reason here is that a number of their users are built with non-libtool-based build systems, and another good amount of .la files are removed by the less lazy Gentoo packagers (XFCE should be entirely .la free nowadays, and yes, it links to GTK+).

Now it is true that the amount of .la files and ELF files is not proportional to the number of packages installing them (for instance Evolution installs 24 .la files and 69 ELF objects), so you can’t really say much about the number of packages you’d have to rebuild when one of the three “virulent” libraries (libpng, libexpat, libz) is installed, but it should still be clear that marking five hundreds files as broken simply because they list a library that is gone, without their respective binary actually having anything to do with said library, is not the best approach we can have.

Dropping the .la file for libcairo (which is where libgtk picks it up) should probably make it much more resilient to the libpng bumps, which have proven to be the nastiest ones. I hope somebody will step up to do so, sooner or later.

Surviving without libtool archives

The sheer amount of articles regarding libtool archives in my blog’s archive is probably enough to tell that I have a pretty bad relationship with them. But even though I have been quiet, lately, I haven’t stopped working on them.

In particular, my current “daily job”, or at least one of them, has among its task the one of getting more reliable cross-compilation support with Gentoo’s ebuilds, and dropping the cursed .la files is part of the plan. With them around, often enough, -L/usr/lib64 is added to the linking line, which causes all sorts of troubles once it’s resolved to a fully qualified path such as /usr/lib64/libfoo.so which cannot obviously be linked against.

After a bit of doubts on how to proceed (waiting for Gentoo proper to remove all of them, given the kind of obstruction I find coming from developers, was not really an option), it was decided to try building the whole chroot used for cross-compilation, and the cross-compiled roots by removing all the .la files with the exclusion of the one that is known to be required (libltdl.la).

For those wondering, it was achieved with this “simple” setting in make.conf:

INSTALL_MASK="
/usr/lib/lib[0-9]*.la
/usr/lib/lib[a-k]*.la
/usr/lib/lib[m-z]*.la
/usr/lib/libl[0-9]*.la
/usr/lib/libl[a-s]*.la
/usr/lib/libl[u-z]*.la
/usr/lib/liblt[0-9]*.la
/usr/lib/liblt[a-c]*.la
/usr/lib/liblt[e-z]*.la
/usr/lib/libltd[0-9]*.la
/usr/lib/libltd[a-k]*.la
/usr/lib/libltd[m-z]*.la
/usr/lib/libltdl[0-9]*.la
/usr/lib/libltdl[a-z]*.la
"

At this point, my opinion has always been that there would be no problem with building further packages even though the libtool archives were gone… turns out I was mostly, but not entirely, right. Indeed if the system were built using the standard GNU ld linker, there would have been no package failure; on the other hand, all of this is built using gold ­– which is much stricter about underlinking – and that makes a huge difference.

Not only the use of libtool by itself would make it mostly pointless to use --as-needed, but it also makes --no-add-needed (the feature that makes the linker strict in terms of underlinking) much less effective: if you only link to libfoo.la, which in turn lists -lbar, and you use symbols from both, the libtool archive would provide both to the linker, hiding the fact that you didn’t express your dependencies properly.

But out of a whole operating system built without .la files, how many packages did require fixes? The answer is two: libsoup (which was actually already fixed in the second-to-last release, so the fix was simply updating the version used) and tpm-tools (that, similarly to trousers and opencryptoki has a quite bogus build system).

I’m not saying that they would be the only packages suffering from these issues, and in particular, with the fact that this system is not building anything statically, it is likely to encounter much fewer complications, but it is more than likely that with minimal effort we’d suffer fewer problems with linking, rebuilds, and dependencies if we were to drop those files entirely, and switch to fixing the few packages failing.

Libtool archives and their pointless points

Since the discussion resumed about libtool files, and we’re back at deciding whether to kill them now or “plan” for the next five to ten years, I guess I better summarise again all the information regarding them, maybe trying to extend what I already wrote about a number of times and provide all the reasoning to abandon them as soon as possible.

I’m writing it here since this is what I use as main reference; I’ll see to send this to gentoo-dev tomorrow if I have time and I feel motivated enough, but if somebody wants to beat me to this, I’ll be just happy since it’ll mean less work for me.

About the chance of just removing all of them unconditionally. This is one thing I sincerely don’t think it’s possible, even though some trials have been done toward that target, for instance with the Portage-Multilib branch. The reasons are multiple; the most obvious one is that calling them *.la is not just enough; there is at least one package in the tree (in the subset of packages the tinderbox is able to merge, more to the point) that names files with .la suffix but simply are not libtool archives at all. So simply removing all of the files based on the file name and path is a bad idea.

About fixing this at the core, libtool. That’s a feasible, albeit way long term solution, if it wasn’t that I find it unlikely that upstream will accept that. Beside the fact that rather than coupling autotools more tightly to reduce duplication issues with version compatibility, they seem to be splitting it further and requiring users to use more code that might not be useful at all for them (for instance by deprecating the mktime() checks in favour of using gnulib, as they did with autoconf 2.68), libtool developers seems to have indicated they don’t intend conceding to modern system use cases if they might hinder support of much older systems.

While it is feasible to patch libtool to not emit the .la files, and quite a number of projects will rejoice of that, it cannot be done unconditionally, as I’ll explain further along the post. So this will require either doing it under conditional of -shared flag, or adding a new flag parameter to use. But even assuming that upstream were to merge the flag, fixing all of the packages upstream not to emit the libtool archive files, well, it’s a plan that takes way too many years, with user inconvenience not stopping until all of it is done. So as it is, it’s something that should be sought out, but won’t help on short to mid term.

About usefulness of libtool archives with plugins, which is something I wrote in detail over an year ago. I’m not sure if I can say “each and every software building plugins with automake uses libtool”, as I don’t know all of them (yet), but I can safely assume that most of the automake-based build systems out there can only make use of libtool to build plugins, shared objects that are dynamically loaded into processes to provide expanded features. This means that most of them will emit and install, by default, a number of libtool archive files.

But even if you build the plugins with libtool, it does not mean you’re loading them with it as well; you could be using the libltdl library that libtool provides, but sincerely, most of the time just learning about it is a waste of time. The most important feature it provides you with is a common interface for various dynamic linker interfaces; indeed there are different dynamic linker interfaces on Linux, Windows and Mac OS X (if I recall correctly, at least), but in this usage mode, there is no use of the archive files either, they are not really needed; the interface allows you to load any object file, as long as you use the full basename, including the operating system-specific extension (.so, .dll, .dylib…).

Where the libtool archive files are actually used is dynamic linker emulation, which is another work mode of libltdl; instead of accessing the standard dynamic linker (loader) in the system, the host software only relies on static linking, and the libtool archives are used for knowing what to look for. In this mode, the archives are used even when not using the emulation itself, for they describe the plugin is used from a shared object anyway. In this case you cannot remove the libtool archive without changing the underlying host software architecture considerably.

The result is that for most usage of libltdl, you can remove the .la files without thinking twice once you know that the software uses dlopen() to load the plugins (such is the case of xine-lib, which also uses a nasty Makefile.am hack to remove them altogether). You might require a bit more analysis for the software that does use libltdl.

About using libtool archive files for standard libraries is definitely a different, complex topic that I’ll have to write probably a lot about.

The first problem: static archives (and thus static linking) for libraries hosting plugins (like xine-lib, for instance). Most of these host programs do not support static linking at all, so for instance xine-lib never provided a static archive to begin with. This does not mean that there is no way to deal with static-linked plugins (the so-called built ins that I talked a lot about; heck Apache pulls it off pretty nicely as well). But it means that if the plugin links to the host library (and most do, to make sure proper linking is applied), then you cannot have a statically-linked copy of it.

Anything that uses dlopen(), anything that uses features such as PAM or NSS, will then not have a chance to work correctly with static linking (I admit, I’m oversimplifying here a bit, but please bear with me, getting into the proper details of that will require a different blog post altogether and I’m too tired to start drawing diagrams). After pushing this to an accepted state, we can now look at what libtool archive files provide for the standard libraries.

The libtool archive files were mainly created to overcome the limitations of the original library implementations for GNU and other operating systems, as they did not provide ABI version information, or dependency data. On some operating systems (but please don’t ask me to track down which), neither shared objects nor static archives provide these information; on most modern operating system, Unix, Unix-like, or not at all, at least shared objects are advanced enough to not require support files to provide metadata; ELF (used by Linux, BSD and Solaris) provides them in form of sonames and needed entries; Mach-O (used by OSX) and PE (used by Windows and .NET) have their own ways as well. Static libraries are a different matter.

For most Unix and Unix-like system, static libraries are rather called “static archives” because that’s what they are: archive files created with ar where a number of object files are added. For these, the extra information about static linking is somewhat valuable. It is not, though, as valuable as you might think it is. While it does provide dependency information, there are no warranty that the dependencies used to create the shared object and those to link the static archives are the same; transitive dependencies cover part of the issue, but it doesn’t let you know which builtins you’d have to link statically against. Also, they can be only used by the software that in turns use libtool to link.

With the current tendency at abandoning autotools (for good or bad, we have to accept that this is the current trend), this means that the .la files are getting even more useless, especially because the projects that do build libraries with libtool cannot simply rely on their users to use libtool on their own, and that means they have to provide them with options to link statically (if it’s at all supported) without using libtool. This usually boils down to using pkg-config to do the deed; this also have the positive effect of working for non-autotools and/or non-libtool based projects.

Sincerely, the only relatively big subsystem that relied heavily on libtool archive files was KDE 3; since they switched away from autotools (and thus, libtool) altogether, the rest of the software stack I know of, only consider libtool archives side effects, most of the time not thinking twice about their presence. A few projects are actively trying to avoid the presence of such files, for instance by removing them through an install hook (which is what xine-lib has been doing for years), but also has a drawback: make uninstall does not work if you do that, because it relies on the presence of the .la files (we don’t need those in ebuilds since we don’t use the uninstall target at all).

Taking all of this in consideration, I reiterate my suggestion that we start removing .la files on an ebuild basis (or on subsystem basis like it’s the case for X11 and gnome, as they can vouch for their packages not to idiotic things); doing so in one big swoop will cause more trouble (as a number of packages that do need them will require the change to be reverted, and a switch to be added); and simply adding a f..ine load of USE flags to enable/disable their installation is just going to be distracting, adding to the maintenance burden, and in general not helpful even for the packages that are not in tree at all.

Repeat after me: nobody sane would rely on libtool so much nowadays.

Let’s call a spade a spade

Some people thought that my previous blog about the libpng debacle was meant as an attack, or a derision, of the work and effort that Samuli put into getting us out of the libpng-1.2 mess we were. Let me be clear: it wasn’t. I’m glad that Samuli is there, without him I would probably have left Gentoo a long time ago, frustrated by nothing happening.

But I think that we shouldn’t hide our head under the sand and keep repeating “it’s all good, it’s all good”. It isn’t.

Samuli did the best he could to get us out of the trouble, which is much bigger than a single person, two, three or even a dozen could properly tackle with all the possible bases covered, at this point, unless there is consensus among the whole developer body, which isn’t there.

But first, I have to say that one thing I’m going to maintain was done wrong, in the rush of the moment: stabling libpng-1.4 as part of a security fix. That was simply reckless. But the fault does not lie in a single person, but rather in the general spirit of avoiding doing extra work… still, reckless or not, it’s done and we have to live with it, and learn from it.

And learning seems like we are; finally there is enough traction for --as-needed to become default as I wrote recently and I got to thank Samuli and Kacper without whom we wouldn’t be able to reach that point at all. I unfortunately still don’t see the same traction behind the hidea of dropping .la files. Removing them from gtk+ which doesn’t install any static library and thus does not need the .la files at all, would have solved if not all, most of the problems people had with the upgrade…

Oh and by the way, the update script for libpng is a hack and it will leave behind .la files when packages will start dropping them, as it changes their checksum and timestamp without updating the package database. The same is true for the (generic) lafilefixer which is why I’ll recommend again to apply the incremental one, as declared in the two posts linked at the beginning.

Finally, libpng-1.5 is going to be released sometime soon… either we make a plan now, or we’re going to suffer through another identical pain soon. And libpng is known for having security issues quite often… I already sent Samuli the plan I was thinking on this morning; I’ll write more details about that as I find the time.

Stable users’ libpng update

Seems like my previous post didn’t make enough of a fuss to get other developers to work on feasible solutions to avoid the problem to hit stable users… and now we’re back to square one for stable users.

Since I also stumbled across two problems today while updating my stable chroots and containers, that represent the local install of remote vservers, and a couple of testing environments for my work, I guess it’s worth writing of a couple of tricks you might want to know before proceeding.

Supposedly, you should be able to properly complete the update without running the libpng-1.4.x-update.sh hack! (and this is important because that hack will create a number of problems on the longer run, so please try to avoid it!). If you have been using --as-needed for a decent amount of time, the update should come almost painless. Almost.

I maintained that revdep-rebuild should be enough to take care of the update for you, but it comes with a few tricks here that make it slightly more complex. First of all, the libpng-1.4 package will try to “preserve” the old library by copying it inside itself, avoiding dynamic link breakage. This supposedly makes for a better user experience as you won’t hit packages that fail to start up for missing libraries, but has two effects; one is that you may be running a program with both libpng objects around, which is not safe; the second is that revdep-rebuild will not pick up the broken binaries at all, this way.

Additionally, there is a slot of the package that will bring in only the library itself, so that binary-only packages linked to the old libpng can be used still; if you have packages such as Opera installed, you might have this package brought in on your system; this will further complicate matters because it will then collide with libpng-1.4… bad thing.

These are my suggested instructions:

  • get a console login, make sure that GNOME, KDE, any other graphical interface is not running; this is particularly important because you might otherwise experience applications that crash mid-runtime;
  • emerge -C =libpng-1.2* make sure that you don’t have the old library around; this works for both the old complete package and for the new library-only binary compatibility package;
  • rm -f /usr/lib/libpng12.so* (replace lib/ with lib64/ on x86-64 hosts; this way you won’t have the old libraries around at all; actually this should be a no-op since you removed it, but this way you ensure you don’t have them around if you had already updated;
  • emerge -1 =libpng-1.4* installs libpng-1.4 without preserving the libraries above; if you had already updated, please do this anyway, this way you’ll make sure it registers the lack of the preserved libraries;
  • revdep-rebuild -- --keep-going it shouldn’t stop anywhere now, but since it might, it’s still a good idea to let it build as much as it can.

Make also sure you follow my suggestion of running lafilefixer incrementally after every merge, that way you won’t risk too much breakage when the .la files gets dropped (which I hope we’ll start doing systematically soon), by adding this snipped to your /etc/portage/bashrc:

post_src_install() {
    lafilefixer "${D}"
}

Important, if you’re using binary packages! Make sure to re-build =libpng-1.4* after you deleted the file, if you had updated it before; otherwise the package will have preserved the files, and will pack it up in the tbz2 file, reinstalling it every time you merged the binary.

This post brought to you by the guy who has been working the past four years to make sure that this problem is reduced to manageable size, and that has been attacked, defamed, insulted and so on so forth for expecting other developers to spend more time testing their packages. If you find this useful, you might want to consider thanking him somehow…

Gentoo Failed Us Again

Kudos to Markos who basically gave me the title for this blog post!

I’ve spent the past week or so away from computers, I’m having some personal trouble, tied with bad migraines that would have burnt the hell out of me. I came back to updating my systems today, and I received a nasty surprise. Unmasked libpng 1.4 is wrecking havoc on so many systems that it’s not even funny.

I’m not complaining about the fact that we’ve finally unmasked the new libpng, it was needed and we should probably proceed on getting it stable soonish as well. What I complain about is that we’re hitting the same obstacles we hit with libexpat:

  • we still don’t have enabled --as-needed by default, which would reduce considerably the amount of packages that actually need to be rebuilt after such an update (and, by the way, not using --as-needed also increases tremendously the chance that some program will be loading both 1.2 and 1.4 versions, with the usual trouble of symbols’ collisions);
  • we still haven’t solved the problem with libtool archives , requiring the rebuild (or nasty hack) of a number of packages for no good reasons.

The worst part is that I have been preaching about both things for a while, a few years I’d say, and yet we have not gotten our heads out of the sand, so we hit users in the face after this kind of updates. Still.

Using --as-needed, only a fraction of the packages installed in the system will actually link against the libtool file, and only those would need to be rebuilt; without it, it’s very likely almost all the libtool-using packages, as well as most pkg-config using packages will be linking in libpng as a dependency of other libraries, such as GTK+ or Qt. And since you will start updating from those libraries, the newly-started packages will have problems because both libpng versions will be loaded at the same time: once from the library and once from the application.

For what concerns the .la files, the problem is mostly at buildtime and it makes it very very difficult to get out of the mess caused by the update, as a number of packages will start lamenting of missing targets for -lpng12. The solution for this would be to, obviously, carefully remove .la files within the ebuilds; this way we reduce the chances that the dependencies end up polluting packages that would, otherwise, have no involvement whatsoever with libpng.

Unfortunately, removing all the libpng file indiscriminately is a Bad Idea™ (and yes, I know some people experimented with that, I still maintain it’s a bad idea!). What you want to do is to reduce their impact as much as possible, but to do so you have to do some extra work, and that requires developers to understand the problem and accept working on a solution, even a temporary, imperfect one, to avoid staying in the problem area.

Do you remember when I checked eog (Eyes of Gnome) .la files resulting in stating clearly that they are totally useless? Well, eog still installs them; sure enough they are not excessively important in this situation, as they are not linked against, but they will create false positives in revdep-rebuild for instance. Even my flowchart has gone mostly unused by developers.

And we still don’t have any way to sanitise those files within Portage; lafilefixer does solve some stuff, but it’s not part of Portage proper, nor it’s integrated with it. If you want, in the future, to reduce your system’s pollution, do something like this:

# /etc/portage/bashrc
post_src_install() {
    lafilefixer "${D}"
}

This way the files will be sanitised before being merged in the system, and you won’t have to fix them manually.

Will we ever learn?

Avoid building (some) static libraries

Warning: the following trick is the kind of material that allows you to shoot yourself in the foot. On the other hand, since there isn’t enough push in Gentoo to solve the situation for good, I’ve decided to use this extensively myself.

Servers and embedded have two completely opposite conception of utility for libraries; in the latter case you most likely prefer static archives – when a single package use them at least – while in the former you prefer shared objects, even if it’s a single package, even better if it’s shared by multiple packages. This is mostly because of the difference in update policies: you don’t update an embedded system that is working unless you really have to; you update a server as soon as a security issue is found.

So, if shared objects are preferred for my own server, why should I have static archives laying around? Well I shouldn’t; especially on the server itself. Now, to remove static archives and include files on the server side, when using binary packages to upgrade the server, it’s actually quite easy: you just have to INSTALL_MASK them in the server’s configuration files (but not on the build chroot system you use to generate them). It solves the problem of using up space for no good reason but it still packages them in the the .tbz2 files that are sent to the server.

Now, for include files there is little you can do, since you do need on the build system, and you cannot really just INSTALL_MASK the static archives away entirely, because things like flex or libatomic_ops don’t install anything else but a static archive. What you can do is use the per-package environment support to remove the static archives (and, most of the time, the libtool files that are unneeded):

# /etc/portage/env/dev-libs/libxml2
export INSTALL_MASK="${INSTALL_MASK} /usr/lib*/*.a /usr/lib*/*.la"

This will be parsed at each phase, and will avoid merging in the static and libtool archives in the system and the tarballs. You just have to create a file like this for each of the packages that install unwanted static archives. But this will still compile two copies of every source file at build time, one with PIC and one without, to properly create static and shared objects. While solving this in a totally generic fashion is basically not possible, you can make use of some features of standard autoconf-based packages, starting from EXTRA_ECONF:

# /etc/portage/env/dev-libs/libxml2

# don't build static archive
export EXTRA_ECONF="${EXTRA_ECONF} --disable-static"

# get rid of the (useless) libtool archive
export INSTALL_MASK="${INSTALL_MASK} /usr/lib*/*.la"

This will pass the --disable-static option to econf, like most of the static-libs USE flags do. It only works for standard autoconf packages so it fails for stuff like OpenSSL and ncurses, for which you don’t have a proper escape route. On the other hand, it does its job for the most common packages. This trick gets also pretty useful when you have to deal with EC2, as the double-build is one heck of a killer.

Try not to abuse of EXTRA_ECONF, though. It actually lets you change ebuilds without having to hack eclasses or ebuilds altogether, but as I said, it’s one very good way to shoot yourself in the foot.

More pointless .la files

The long-time followers of my blog will almost surely know of my fight against the pollution from libtool archive (.la) files in Gentoo packages. I already said that they are often useless, or even dangerous when dealing with cross-compilation, and that they create problems that can be solved only with --as-needed (over-linking).

But in particular, I’ve started making the tinderbox warn me when libtool archive files are found in three particular sub-trees where they are absolutely not useful: Python modules, Ruby extensions and PAM plugins. Neither of these use libtool for loading the extensions (for quite obvious reasons), so the .la file is not going to be used for loading; you also shouldn’t link against dynamically-loaded modules (and libtool warns you about that if you try to do it), so the .la file is not used for build-time linking. It’s simply unused.

But it goes even worse than that: the warning has also made me notice that quite a few packages not only install the libtool archive, but build and install a static archive as well! This is not only an extra file installed (and in the search path of those implementations) but also force libtool to build all the source code twice: with and without Position-Independent Code (the static archives are usually non-PIC but shared objects are PIC).

In the bugs I reported for this kind of problems I also linked my own documentation on building plugins with libtool, so maybe the Gentoo developers reaching the bugs would be able to fix both up- and down-stream the issue of double build of both PIC and non-PIC objects. Not that I count on it but it might be possible.

A shared library by any other name

One interesting but little known fact among users and developers alike is the reason why shared libraries are installed on systems with multiple file names. This ties a bit into the problem of .la files, but that’s not my topic here.

Shared libraries, especially when built and installed through libtool, usually get installed with one file, and two symlinks: libfoo.so, libfoo.so.X and libfoo.so.X.Y.Z. The reasoning for this is not always clear, so I’ll try to shed some light on the matter, for future reference, maybe it can help solving trouble for others in the future:

  • first of all, the real file is the one with the full version: libfoo.so.X.Y.Z: this because libtool uses some crazy-minded versioning scheme that should make it consistent to add or remove interfaces… in truth it usually just drives developers crazy when they start wondering which value they have to increase (hint: no, the three values you set into libtool flags are not the same three you get in the filename);
  • the presence of the other two names are due to the presence of two linker programs: the build-time linker (or link editor) and the runtime (or dynamic) linker: ld and ld.so; each one uses a different name for the library;
  • the link editor (ld) when linking a library by short name (-lfoo) isn’t in the known about which version you’re speaking of, so it tries its best to find the library transforming it to libfoo.so, without any version specification; so that’s why the link with the unversioned name is there;
  • the dynamic linker, when looking up the libraries to load, uses the NEEDED entries in the .dynamic section of the ELF file; those entries are created based on the SONAME entry (in the same section) of the linked library; since the link editor found the library as libfoo.so it wouldn’t be able to use the filename properly; the SONAME also serves to indicate the ABI compatibility, so it is usually versioned (with one or more version components depending on the operating system’s convention — in Gentoo systems, both Linux and FreeBSD, the convention is one component, but exceptions exist); in this case, it’d be libfoo.so.X; so this is what the dynamic linker looks up, it’s also not in the known about the full version specification.

Now there are a few things to say about this whole situation with file names: while libtool takes care of creating the symlinks by itself, not all build systems do so; one very common reason for that is that they have no experience of non-GNU systems (here the idea of “GNU system” is that of any system that uses the GNU C library). The thing is, ldconfig on GNU systems does not limit itself at regenerating the ld.so cache, but it also ensures that all the libraries are well symlinked (I sincerely forgot whether it takes care of .so symlinks as well or just SONAME-based symlinks, but I’d expect only the latter). A few packages have been found before explicitly relied on ldconfig to do that using a GNU-specific parameter (a GNU lock-in — might sound strange but there are those as well) that takes care of fixing the links without changing the ld.so cache.

And there our beloved .la files come back in the scene. One of the things that .la files do is provide an alternative to the -lfoolibfoo.so translation for the linkers that don’t do that by themselves (mostly very old linkers, or non-ELF based linkers). And once again this is not useful to us, at least in main tree, since all our supported OSes (Linux, FreeBSD, with all three the supported C libraries) are new enough to properly take care of that by themselves.

Plugins and static libraries don’t mix well

There is one interesting thing that, I’m afraid, most devleopers happen to ignore, either spontaneously, or because they really don’t know about them. One of this is the fact that static libraries and plugins usually don’t mix well. Although I have to warn you, that’s not an absolute and they can easily designed to work fine.

The main problem though is to ponder whether it is useful to use static libraries and plugins together, and then it’s to find out if it’s safe to or not. Let’s start from the basis. What are static libraries used for? Mostly for two reasons: performances, and not having to depend on the dynamic version of the same library in the system. If performance of the library is a problem, it’s much more likely that the culprit is the plugins system rather than the dynamic nature of the library; I have said something about it in the past, although I didn’t go much in details and I haven’t had time to continue the discussion yet.

For what concerns dependencies, the plugins usually need a way to access the functions provided by the main library; this means there is an ABI dependency between the two; now the plugins might not link against the library directly, to support static libraries usage, but it also means that if the internal ABI changes in any way between versions, you’re screwed.

What does this mean? That in most cases when you have plugins, you don’t want to have static libraries around; this means that you also don’t need the .la files and so you have quite a bit of cleanup.

More to the point, if you’re building a plugin, you don’t want to build the static version at all, since the plugin will be opened with the dlopen() interface, from the dynamic version of the library (the module). Since not always upstream remember to disable the static archive building in their original build system, ebuild authors should take care of disabling them, either with --disable-static or by patching the build system (if you don’t want to stop all static lib building). And this is not my idea but a proper development procedure (and no, it does not mean any discussion: if it’s a plugin — and it’s not possible to make it a builtin — you shouldn’t install the archive file! Full stop!).

Now, you can see where this brings us again: more .la files that are often currently installed and are not useful at all. Like .la files for PAM modules (libpam does not load them through the .la so they are not useful — and this is definitely the word of the PAM maintainer! And for PAM-related packages, that word is The Word). Let’s try to continue this way, shall we? From the leaves.