New results from my elven script

My work on ruby-elf is continuing, albeit slowly as I’m currently preparing more envelopes. I’ve improved the nm.rb tool to be a bit more alike to the real nm utility from binutils, by taking properly consideration of the Absolute and Common symbols, and today while working on the conflicts finder I also fixed the symbols’ versions parser. Now the parser and the tool should be pretty much solid; too bad that by fixing the parser to actually look in the libraries recursively I ended up making it quite more slow (well, it has a lot more libraries to look up now), and it eats more than 70MB for the database alone.

Now, let me first talk about symbol versions on ELF files. This is a GNU extension, that is not obviously documented in elf(3), as that man page comes out of BSD. There’s little to no documentation available about it, the only thing that can be somewhat reliable is a page from Ulrich Drepper on his home site on RedHat, which of course is the usual hyped version of the truth, in the usual Drepper style everybody is used to. The elf.h header does not contain much information beside the data structures with a few comments, that are not good enough to actually implement anything without a lot of analysis of the data itself, and some fill ups from Drepper’s hyped page. One of the entries, commented as «Unused» in the header, and also in Drepper’s page in the structure definition, carries the most important data of the structure: the index used to retrieve the version information.

Three tables are used, plus .dynstr that carries the symbols’ names and thus also the versions’ names (they are declared as absolute symbols of value 0); as a symbol can have multiple names if it obsoleted other symbols, the records are of variable length, rather than fixed length, which makes it more difficult to parse them. The versions for defined (exported) and for undefined (imported) symbols are declared in different structure, subtly similar enough to confuse you, then the third table tells you which version every symbol refers to. As the «auxiliary» structures for both defined and undefined symbols are not part of the version definition, but pointed by through an offset, and only carry the name of a symbol, they can be shared. Now don’t ask me why there should be two different version specifications pointing at the same name (the only reason I’d see would be if defined and undefined symbols had the same version name, but also the auxiliary structures are not the same between the two, and are defined in two different sections, so those can’t be shared), the only case I found up to now is Berkley DB that uses —default-symver).

After being able to implement an ELF parser that can read symbols’ versions, without looking at glibc source code – because I don’t want to hurt my eyes looking at it, and also because if I need I can implement it again without being restricted to GPL-2 license – I have to say that I don’t like it and I hope never to rely on it in my life! Seems to me like it would have been possible to use a few more bytes but make the load process faster by using fixed-length records; there are also flags variables pretty much unused that could have been used for defined or non-defined symbols if that matters so much, but the fact that you find the version through the symbols rather than the other way around makes it pretty much pointless: you already know if a symbol is defined or not!

Anyway after this a-bit-offtopic time, let me show a bit of results coming from the script itself:

• bug #178919, #178920, #178921 and #178923: there are three ebuilds (one being python and two being python extensions) that brings in their own copy of expat, 1.95.8, which is incompatible with expat 2.0 that is in ~arch for a while now; Compress-Raw-Zlib instead carries its own copy of zlib, which is a waste as virtually any system at a given time has a copy of zlib already in memory.
• I found a few more common libraries in kdepim-kresources that cut down three more megabytes from the total size of the package installed on my system; note that the size includes full debug information in dwarf format (-ggdb), but no debug code; the memory reduction should be similar in a proportion, but it’s not the same amount of course; still it’s a sensible gain.
• I also prepared a patch for kmix to install two more libraries: libkmixprivate and libkmixuiprivate, as kmix, kmixctrl and kmix_applet were sharing half of their code to begin with; through my patch they share it effectively on disk, and during build the files are built once only.
• Samba would probably make use of a libsambainternal library of its own, as a lot of symbols seems to be shared between different tools from the Samba package. Note that internal libraries are an useful way of sharing code, as you can put there the functions used by everything, and just declare them to be private, making sure that users of the project won’t try to use it, or at least will know that the ABI can change from one release to the other.

With the more widespread look of the script I also had to extend a lot the suppressions’ file, as there is a lot of software using plugin-based interface nowadays.
One problem I’m facing now, though, is that there are a lot of drop-in replacement libraries in a common system, for instance libncurses and libncursesw, OpenSSL and GNU-TLS’s replacement, and so on. I need to write a suppression file for those too, so that the symbols common between those libraries are not reported as actual collisions, but skipped over.

One thing I’m not sure about are the yy* symbols that come out of flex/yacc/whatever it is. There are a lot of libraries that export them, and I’m not sure if it’s correct for them to, as the different flex versions might be incompatible. Unfortunately I never used flex in my life so I can’t tell.

If somebody knows how those functions are supposed to work with shared libraries, I’ll be very grateful to know; and if they are a problem, I’ll see to report and fix all of them.