Since the discussion resumed about libtool files, and we’re back at deciding whether to kill them now or “plan” for the next five to ten years, I guess I better summarise again all the information regarding them, maybe trying to extend what I already wrote about a number of times and provide all the reasoning to abandon them as soon as possible.
I’m writing it here since this is what I use as main reference; I’ll see to send this to gentoo-dev tomorrow if I have time and I feel motivated enough, but if somebody wants to beat me to this, I’ll be just happy since it’ll mean less work for me.
About the chance of just removing all of them unconditionally. This is one thing I sincerely don’t think it’s possible, even though some trials have been done toward that target, for instance with the Portage-Multilib branch. The reasons are multiple; the most obvious one is that calling them *.la is not just enough; there is at least one package in the tree (in the subset of packages the tinderbox is able to merge, more to the point) that names files with .la suffix but simply are not libtool archives at all. So simply removing all of the files based on the file name and path is a bad idea.
About fixing this at the core, libtool. That’s a feasible, albeit way long term solution, if it wasn’t that I find it unlikely that upstream will accept that. Beside the fact that rather than coupling autotools more tightly to reduce duplication issues with version compatibility, they seem to be splitting it further and requiring users to use more code that might not be useful at all for them (for instance by deprecating the mktime() checks in favour of using gnulib, as they did with autoconf 2.68), libtool developers seems to have indicated they don’t intend conceding to modern system use cases if they might hinder support of much older systems.
While it is feasible to patch libtool to not emit the .la files, and quite a number of projects will rejoice of that, it cannot be done unconditionally, as I’ll explain further along the post. So this will require either doing it under conditional of -shared flag, or adding a new flag parameter to use. But even assuming that upstream were to merge the flag, fixing all of the packages upstream not to emit the libtool archive files, well, it’s a plan that takes way too many years, with user inconvenience not stopping until all of it is done. So as it is, it’s something that should be sought out, but won’t help on short to mid term.
About usefulness of libtool archives with plugins, which is something I wrote in detail over an year ago. I’m not sure if I can say “each and every software building plugins with automake uses libtool”, as I don’t know all of them (yet), but I can safely assume that most of the automake-based build systems out there can only make use of libtool to build plugins, shared objects that are dynamically loaded into processes to provide expanded features. This means that most of them will emit and install, by default, a number of libtool archive files.
But even if you build the plugins with libtool, it does not mean you’re loading them with it as well; you could be using the libltdl library that libtool provides, but sincerely, most of the time just learning about it is a waste of time. The most important feature it provides you with is a common interface for various dynamic linker interfaces; indeed there are different dynamic linker interfaces on Linux, Windows and Mac OS X (if I recall correctly, at least), but in this usage mode, there is no use of the archive files either, they are not really needed; the interface allows you to load any object file, as long as you use the full basename, including the operating system-specific extension (.so, .dll, .dylib…).
Where the libtool archive files are actually used is dynamic linker emulation, which is another work mode of libltdl; instead of accessing the standard dynamic linker (loader) in the system, the host software only relies on static linking, and the libtool archives are used for knowing what to look for. In this mode, the archives are used even when not using the emulation itself, for they describe the plugin is used from a shared object anyway. In this case you cannot remove the libtool archive without changing the underlying host software architecture considerably.
The result is that for most usage of libltdl, you can remove the .la files without thinking twice once you know that the software uses dlopen() to load the plugins (such is the case of xine-lib, which also uses a nasty Makefile.am hack to remove them altogether). You might require a bit more analysis for the software that does use libltdl.
About using libtool archive files for standard libraries is definitely a different, complex topic that I’ll have to write probably a lot about.
The first problem: static archives (and thus static linking) for libraries hosting plugins (like xine-lib, for instance). Most of these host programs do not support static linking at all, so for instance xine-lib never provided a static archive to begin with. This does not mean that there is no way to deal with static-linked plugins (the so-called built ins that I talked a lot about; heck Apache pulls it off pretty nicely as well). But it means that if the plugin links to the host library (and most do, to make sure proper linking is applied), then you cannot have a statically-linked copy of it.
Anything that uses dlopen(), anything that uses features such as PAM or NSS, will then not have a chance to work correctly with static linking (I admit, I’m oversimplifying here a bit, but please bear with me, getting into the proper details of that will require a different blog post altogether and I’m too tired to start drawing diagrams). After pushing this to an accepted state, we can now look at what libtool archive files provide for the standard libraries.
The libtool archive files were mainly created to overcome the limitations of the original library implementations for GNU and other operating systems, as they did not provide ABI version information, or dependency data. On some operating systems (but please don’t ask me to track down which), neither shared objects nor static archives provide these information; on most modern operating system, Unix, Unix-like, or not at all, at least shared objects are advanced enough to not require support files to provide metadata; ELF (used by Linux, BSD and Solaris) provides them in form of sonames and needed entries; Mach-O (used by OSX) and PE (used by Windows and .NET) have their own ways as well. Static libraries are a different matter.
For most Unix and Unix-like system, static libraries are rather called “static archives” because that’s what they are: archive files created with ar where a number of object files are added. For these, the extra information about static linking is somewhat valuable. It is not, though, as valuable as you might think it is. While it does provide dependency information, there are no warranty that the dependencies used to create the shared object and those to link the static archives are the same; transitive dependencies cover part of the issue, but it doesn’t let you know which builtins you’d have to link statically against. Also, they can be only used by the software that in turns use libtool to link.
With the current tendency at abandoning autotools (for good or bad, we have to accept that this is the current trend), this means that the .la files are getting even more useless, especially because the projects that do build libraries with libtool cannot simply rely on their users to use libtool on their own, and that means they have to provide them with options to link statically (if it’s at all supported) without using libtool. This usually boils down to using pkg-config to do the deed; this also have the positive effect of working for non-autotools and/or non-libtool based projects.
Sincerely, the only relatively big subsystem that relied heavily on libtool archive files was KDE 3; since they switched away from autotools (and thus, libtool) altogether, the rest of the software stack I know of, only consider libtool archives side effects, most of the time not thinking twice about their presence. A few projects are actively trying to avoid the presence of such files, for instance by removing them through an install hook (which is what xine-lib has been doing for years), but also has a drawback: make uninstall does not work if you do that, because it relies on the presence of the .la files (we don’t need those in ebuilds since we don’t use the uninstall target at all).
Taking all of this in consideration, I reiterate my suggestion that we start removing .la files on an ebuild basis (or on subsystem basis like it’s the case for X11 and gnome, as they can vouch for their packages not to idiotic things); doing so in one big swoop will cause more trouble (as a number of packages that do need them will require the change to be reverted, and a switch to be added); and simply adding a f..ine load of USE flags to enable/disable their installation is just going to be distracting, adding to the maintenance burden, and in general not helpful even for the packages that are not in tree at all.
Repeat after me: nobody sane would rely on libtool so much nowadays.
Re: upstream, this thread is informative:http://thread.gmane.org/gma…
Do you think libtool archives still have their place outside of modern desktop distributions? When building for embedded systems, for example?Tesseract has recently started using libtool to do dynamic linking but one of its dependencies, Leptonica, will only build as a shared library from the next release. For this reason, Tesseract is currently looking for the libraries that Leptonica “usually” requires, even though Tesseract doesn’t use them at all. Obviously this is what libtool was designed to prevent. Now that Leptonica will be a shared library, this could be avoided without the use of libtool archives but I get the impression that lots of people still want to build statically for whatever reason.So what I’m unclear on is whether you are declaring war on libtool archives everywhere or whether you’re just saying they do more harm than good on a system built almost entirely around shared libraries, such as Gentoo.
Building for embedded has the same build systems as building for desktop; so yes, .la files tend to be useless. When cross-compiling, they even tend to be _more_ of a nuisance because they use full paths, not relatives one, and libtool has been ignoring the SYSROOT setting altogether.I think they fixed SYSROOT support in 2.4, but as it is for libtool, just updating the system copy is pointless, you have to rebuild the autotools of each of the involved packages to have it respect SYSROOT properly.
I’ve been reading your blog entries for quite a while, and just want to thank you for the wonderful work that you do. You really explain issues in a complete and understandable way (well, sometimes your explanations are even a bit too technical for me, but I’m sure make sense to people who are more directly involved in the particular issue), that makes the matter clear. I have learned a great deal by reading your posts. Thanks again.
I’d forgotten about the SYSROOT problem, that was a real pain. At least there’s been some progress there.You didn’t fully answer my question though. What if you actually want to use static libraries?
Is there some Automake flag that prevents .la files from being installed, and which also keeps the “uninstall” target working ?
James, as long as pkg-config provides the right information and you set @PKG_CONFIG=”pkg-config –static”@ then static linking should work just as fine.Stelian, as I said in the article, there isn’t an option to do that at this point; while useful, adding it won’t solve any of our current problems, though.
I wouldn’t solve any of Gentoo’s problems, but as maintainer of a C library I’d like not to install them anyway
> The result is that for most usage of libltdl, you can remove the .la files without thinking twice> once you know that the software uses dlopen() to load the pluginsYou can also do so if the software uses lt_dlopenext.BTW, the reason why libtool long ignored the sysroot, is because sysroot support was invented _after_ libtool, and those who introduced it didn’t think of upgrading libtool as well. But the new libtool 2.4 is already being tinkered with by OpenEmbedded, and so far they found only one bug (and removed a bunch of hacks from their build system).Paolo