Linkers and names

I received a question by mail the other day from Randy, about sonames and in particular their versioning which I already discussed about a couple of times. On the other hand, his concerns are good enough to warrant for another blog post on the matter, hoping not to bore my readers too much with it.

First of all, let me repeat that there are a number of different names that a shared library “owns”, on Unix and Unix-like systems based on ELF:

  • the actual name of the filename;
  • the name the link editor (or build-time linker if you prefer, ld) uses to link to it;
  • the name the loader (or dynamic linker if you prefer, ld.so) uses to find it.

The obvious way to correlate all of them together is to use symbolic links; and that’s why your /usr/lib directory is full of those. But how that happen?

When linking, the link editor (ld) can take a library as parameter in two different ways: either with full path to the library (such as /usr/lib/libcrypto.so.0.9.8) or via a link-library parameter (-lcrypto); in the first case, the linker know exactly which files it has to link against; in the latter, though, it has to go look for it.

The paths where the linker search for the library is worth discussing separately, and all in all the library search path list is not something that is important to the aim of this article. What we do care about is the basename the linker is going to look out for; well, the answer is easy, it applies the same substitution you’d have with sed and s/^-l(.+)$/lib1.so/ — it prepends lib and adds .so to the end of the library name. This is why all the libraries you can link to have a .so suffix-only variant, beside the full name and the version-suffix symlink.

Obviously, when the link-editor adds the dependencies on the libraries to the output, it has to use some kind of name; the full path is not a viable option (you might be linking to a non-installed library that will be installed later), so you can either use the library base name (PE – the format used by Windows – did this, but Microsoft seems to have changed idea when .NET came around), or find some other value to link (no pun intended) it to, which is what we’re concerned with here, since that’s what Unix (and Linux) do.

Each library declares its canonical name (the soname, standing for Shared Object Name); the linker will copy that value from the tag called DT_SONAME to a DT_NEEDED entry tag of the output file. Now, these two tags are really what’s important in this discussion; as you saw, the two are respectively consumed and produced by the link editor. The loader, instead, only cares about DT_NEEDED: it uses that tag to know the filename of the libraries to look for in its library search path (that does not necessary have to be the same as the one for the link editor).

Actually, it’s not all true; there are a number of libraries that don’t declare a soname at all, in some cases that’s correct, for instance plugins don’t need a soname at all since they are never linked against, but a good many times, it’s actually a bug in the build system itself. But that’s a topic for another day.

The loader, for good or bad, does not check that the value in DT_SONAME of the loaded library corresponds to the DT_NEEDED tag it encountered — this is why symlinking libraries with incompatible ABIs seem to work at first: the loader does not fail to load them. This means that it really doesn’t matter what a DT_NEEDED tag points to, as long as, within the library path, there is a file (or symlink) with that name. This lets you do some cool but braindead thinks, which I’d rather not write of myself.

To ensure that the loader is able to find the library, then, there is need to have the symlinks in place that correspond to the name it’s going to look for. Now this can be done by two pieces of software, at two different times. The first is, well, libtool, which takes care of the task both at build-time and install-time, and makes both the link used by the loader, and that used by the link editor to find the library. A number of other build systems imitate that behaviour as well.

The second component that does so is ldconfig, but only on GNU systems; by default it recreates all the symlinks to sonames in the library search path (of the loader); with various options, it processes particular libraries or paths. The result is pretty nifty, but not portable. I discovered this myself a long time ago when working on Gentoo/FreeBSD, as the preplib command from Portage failed on that architecture.

With hindsight, I wish I didn’t ask for preplib to die, but rather rewrite it to use scanelf to do the job rather than ldconfig.

What probably confused Randy, is that many documents out there assumes the libtool/Linux semantic of soname and versioning. I call it libtool/Linux because libtool changes its soname semantic depending on the operating system it’s working on, as again I found out while working on Gentoo/FreeBSD. In Linux, it produces a file with a three-components name, and sets DT_SONAME to just the first component; on FreeBSD (by default) it creates a file with a two-components name, and sets DT_SONAME to the full name.

In Gentoo/FreeBSD, we hack libtool to use the same naming scheme as Linux, since that is what library developers tend to think about when setting the version components. Without this hack, each minor version of glib would have a new soname, requiring all the linked software to be re-built, even if the ABI didn’t change, or changed in a compatible way.

I admit I haven’t even started looking up the reasoning for either the version scheme; they are probably mostly historical; in the case of FreeBSD, having the soname and the full name to match is probably needed to avoid the need for symlinks (as ldconfig does not create them as I said above). In the case of Linux, I’m not really sure why so much complexity. Without digressing too much, the components passed to -version-info don’t match directly the actual components used in either the Linux or FreeBSD semantics; they are both related by way of not-too-simple arithmetic functions.

At any rate, neither the link editor nor the loader have a real concept of “ABI version”, which is why they don’t enforce it at all. So you can have libraries that don’t even update their ABI version at all, or others that use the full version as ABI, and bump it even if there aren’t incompatible changes.

So to answer Randy’s final question, this happens because each project is free to decide what its soname is, and thus its soversion. In the case of OpenSSL, they use a three-components version that is the same of the file name, and of the version of the released package, as can be seen by looking at the following scanelf -S output from the tinderbox :

tinderbox ~ # scanelf -S /usr/lib/libcrypto.so.*
 TYPE   SONAME FILE 
ET_DYN libcrypto.so.0.9.8 /usr/lib/libcrypto.so.0.9.8 
ET_DYN libcrypto.so.1.0.0 /usr/lib/libcrypto.so.1.0.0 

Any document that implies explicitly that the soname has to consist of the library name and its ABI version, is simply wrong. That is a practice that definitely should be followed, but there are no technical restrictions in there, they are only practical restrictions.

Yes, it is a sorry state of the affairs.

A shared library by any other name

One interesting but little known fact among users and developers alike is the reason why shared libraries are installed on systems with multiple file names. This ties a bit into the problem of .la files, but that’s not my topic here.

Shared libraries, especially when built and installed through libtool, usually get installed with one file, and two symlinks: libfoo.so, libfoo.so.X and libfoo.so.X.Y.Z. The reasoning for this is not always clear, so I’ll try to shed some light on the matter, for future reference, maybe it can help solving trouble for others in the future:

  • first of all, the real file is the one with the full version: libfoo.so.X.Y.Z: this because libtool uses some crazy-minded versioning scheme that should make it consistent to add or remove interfaces… in truth it usually just drives developers crazy when they start wondering which value they have to increase (hint: no, the three values you set into libtool flags are not the same three you get in the filename);
  • the presence of the other two names are due to the presence of two linker programs: the build-time linker (or link editor) and the runtime (or dynamic) linker: ld and ld.so; each one uses a different name for the library;
  • the link editor (ld) when linking a library by short name (-lfoo) isn’t in the known about which version you’re speaking of, so it tries its best to find the library transforming it to libfoo.so, without any version specification; so that’s why the link with the unversioned name is there;
  • the dynamic linker, when looking up the libraries to load, uses the NEEDED entries in the .dynamic section of the ELF file; those entries are created based on the SONAME entry (in the same section) of the linked library; since the link editor found the library as libfoo.so it wouldn’t be able to use the filename properly; the SONAME also serves to indicate the ABI compatibility, so it is usually versioned (with one or more version components depending on the operating system’s convention — in Gentoo systems, both Linux and FreeBSD, the convention is one component, but exceptions exist); in this case, it’d be libfoo.so.X; so this is what the dynamic linker looks up, it’s also not in the known about the full version specification.

Now there are a few things to say about this whole situation with file names: while libtool takes care of creating the symlinks by itself, not all build systems do so; one very common reason for that is that they have no experience of non-GNU systems (here the idea of “GNU system” is that of any system that uses the GNU C library). The thing is, ldconfig on GNU systems does not limit itself at regenerating the ld.so cache, but it also ensures that all the libraries are well symlinked (I sincerely forgot whether it takes care of .so symlinks as well or just SONAME-based symlinks, but I’d expect only the latter). A few packages have been found before explicitly relied on ldconfig to do that using a GNU-specific parameter (a GNU lock-in — might sound strange but there are those as well) that takes care of fixing the links without changing the ld.so cache.

And there our beloved .la files come back in the scene. One of the things that .la files do is provide an alternative to the -lfoolibfoo.so translation for the linkers that don’t do that by themselves (mostly very old linkers, or non-ELF based linkers). And once again this is not useful to us, at least in main tree, since all our supported OSes (Linux, FreeBSD, with all three the supported C libraries) are new enough to properly take care of that by themselves.