I think this post might be interesting for those people interested in trying to get all the performance power out of a box, without breaking anything in the process.
I’ve blogged before about the problems related to exporting symbols from final executables, but I haven’t really dug deep enough to actually provide useful information to developers and users about what those exported symbols represent, for an executable.
First of all, let’s start to say that an executable under Linux, and on most modern Unixes, is just the same kind of file of a shared object (shared library, if you prefer, those which are called DLL under Windows, if you come from a Windows background). And exactly as shared libraries, they can export symbols.
Exported symbols are resolved by the dynamic – runtime – linker, through the process of dynamic bindings, and thy might collide. I’ll return on the way the runtime linker works at a different moment. For now let’s just say that the exported symbols require some extra step to be taken during the execution of a program, and that the process takes time.
Executables don’t usually need to export symbols, and they usually don’t export symbols at all. Although there are rare cases where executables are required to export symbols, for instance because they are used by some of the libraries they link to as a “callback” from the library to the program, or for C++ programs for RTTI to properly work, most of the times the symbols are exported just because of libtool.
By default when you link a program, it doesn’t get its symbols exported, they are hidden and thus resolved directly at buildtime, for those symbols present in the source files themselves. When you add code to a convenience library that is built with libtool, then something changes, and the symbols defined inside that library are exported even when linking it statically inside the final executable.
This causes quite a few drawbacks. and as I said, is not usually used for anything:
the symbols are resolved at runtime through dynamic binding, which takes time, even if usually very little for a normal system, repeated time wasted during dynamic binding might actually become a good deal of time;
the symbols might collide with a library loaded afterward, this is for instance why recode breaks PHP;
--gc-sectionswon’t help much because exported symbols are seen as always used, and this might increase the amount of code added to the executable with no good reason;
prelinkwill likely set the wrong addresses for symbols that collide, which in turn will drop off the improvement of using prelink entirely, at least for some packages.
The easy solution for this is for software packages to actually check if the compiler supports hidden visibility, and if it does, hide all the symbols but for the ones in the public API of their libraries. In case of software like cmake that install no shared objects, hidden visibility could be forced by the ebuild, but to give back to the community, the best thing is to get as much software as possible to use hidden visibility, thus reducing the amount of symbols that gets exported on both binaries and shared libraries.
I hope these few notes might actually help Gentoo maintainers to understand why I’m stressing on this point. It would be nice if we all could improve the software we maintain, even one step at a time.
As for what concerns my linking collision scripts, the bad packages’ list got a few more entries today: KViewShell with djvulibre, Karbon with gdk-pixbuf, and gcj, with both boehm-gc and libltdl.
And now I can actually start seeing the true collisions, like
gfree symbol, having two different definitions in libgunicode.so (fontforge) and poppler/libkpdfpart (xpdf code), or the
scan_token symbol in ghostscript with a completely different definition in libXfont/libt1.
Talking about libXfont and libt1 (or t1lib). I wonder if there is hope in the future for one to use the other, rather than both use the same parser code for type1 fonts. I’ll have to check the FreeDesktop bugzilla tomorrow to see if it was ever discussed. At the moment they duplicate a lot of symbols one with the other.
I have to say, PostgreSQL is an important speed improvement, that will allow me to complete my task in shorter time. Now I’m waiting for Patrick to run my script over the whole set of packages in Gentoo, that might actually be something. If only there was an easy way to make building and testing code faster (for COW reduction) without changing hardware, that would be awesome. Unfortunately that I need to do locally 🙁
if you need help for running your script on a different environment than your, feel free to contact me; i’ll be available to test your script on all mine machines.
If a program loads plugins and those plugins need access to symbols that are defined in the program that is doing the loading, then the program will need to have it symbols exported. This isn’t a too uncommon scenario.
It’s actually uncommon, to my experience at least. Usually when plugins are involved, there is a shared library involved in those, too. This is the case for xine, esmtp/libesmtp, most of KDE (Amarok included), Wireshark, …Sure there are a few cases of it, but it’s quite uncommon to me. At the same time, while those symbols need to be exported, this is not true unconditionally for all symbols.
our nam, if you need some symbols exported in your binary, why not move them to shared library and then use it in binary?
So, how could I export symbol from an ELF binary?Neither __attribute__ ((visibility(“default”))) nor objcopy –globalize-symbol helps
You need to use @gcc -rdynamic@ or @ld –export-dynamic@.
Back in the late 90’s I actually used this methodology in Bayonne, to export the server symbol space to link a driver plugin. You use “–export-dynamic” when linking your app to do this. Yes, the use cases are rather rare, and most people will never need to do this. Most of the time, when doing plugins one would instead create an intermediary shared library that the host app and the plugins will then both use. I used plugins with base classes defined in the server (or an intermediary runtime), which added the current instance to a linked list in the constructor, and a static instance of derived class in the plugin, so that it would auto-install when the plugin was loaded.
From a Microsoft/Visual Studio perspective, if you decorate a class or function with __declspec(dllimport), then the linker will assume [correctly] you want to export symbols and create a .lib file in addition to the .exe file.