I’m referring to KDEPIM from KDE 3, I certainly hope this kind of problems is fixed for KDE 4; if it isn’t, that would likely be a good reason why KDE should reform itself in a new way.
So, last night I started working on a simple script (that for now uses a mixture of Ruby and readelf calls) that loads all the symbols present in the libraries on the system, and then checks for duplicates, to see which symbols would collide if loaded in the same address space.
A bit of history on this. If you ever got into Michael Meeks’s patches for the infamous -Bdirect option, he talked about the incompatibility of that with the technique of interposing. Symbol interposing is a way to provide different implementations of the same interface on different libraries, allowing to shift between one and the other by changing the way they are linked or by using LD_PRELOAD. A common case of interposing are the threading libraries; the C library usually has weak symbols for the various
pthread_* functions, beside from
pthread_new that is the one used to actually create the threads, so that a library that can be used both with and without threading can still define its own mutexes and use them, making the calls no-ops when no threads are used.
To use interposing, the same symbol is present in more than one library; usually the library always linked in has a weak symbol, while the implementations have normal symbols. This when used consciously is called interposing, but when this happens (way more often) without a proper conscious use, this is instead symbol collision; a symbol collision is bad, because a library expecting to use a certain function in a certain way might be using it in a totally different way because of the collision.
Even when the symbol is just the same, for the same function, symbol collisions requires more work from the linker to identify them, hash values and prelink not always works, so they usually should be avoided. One way to avoid them, without renaming the functions, if they are used only internally, is to use hidden visibility. This solves the issue, but needs to be properly implemented to be good.
So, my tool checks for symbols’ collisions, by gathering all the symbols in all the libraries installed on my system in a table of an sqlite database, and then counting how many times the same symbol is present. Of course there are false positives, cases in which the presence of different symbols with the same name implies neither interposing nor collisions, and that’s the case of most plugin infrastructures: the plugins export one or more specific symbols that have a given name, the loader knows to look for them and binds them in a per-plugin data structure. In the case of xine, this is achieved through the
xine_plugin_info structure present in every plugin. To avoid reporting those as collisions (they are not) I also added a suppressions file, where I can declare regular expressions of symbols (for some files) that needs not to be counted as collision.
The output of the script shown me this:
Symbol soap_bool2s(soap*, bool) present 3 times /usr/kde/3.5/lib64/kde3/kio_groupwise.so /usr/kde/3.5/lib64/libkcal_groupwise.so.1.0.0 /usr/kde/3.5/lib64/libkabc_groupwise.so.1.0.0 Symbol soap_in_int(soap*, char const*, int*, char const*) present 3 times /usr/kde/3.5/lib64/kde3/kio_groupwise.so /usr/kde/3.5/lib64/libkcal_groupwise.so.1.0.0 /usr/kde/3.5/lib64/libkabc_groupwise.so.1.0.0 Symbol soap_s2bool(soap*, char const*, bool*) present 3 times /usr/kde/3.5/lib64/kde3/kio_groupwise.so /usr/kde/3.5/lib64/libkcal_groupwise.so.1.0.0 /usr/kde/3.5/lib64/libkabc_groupwise.so.1.0.0
and so on for a long time. This means that the same internal library is being linked in those three shared objects, and replicated. This is bad, because it wastes memory (as the code is not shared between the instances of programs using those three libraries) and on-disk space (as the code is replicated on three files when it should be on just one).
I’ve prepared a patch to kdepim-kresources (the package in which these libraries are) that instead of declaring libgwsoap an internal library declares it as an installed library, so that the three libraries links to the libgwsoap rather than linking it in.
flame@enterprise ~ % qsize kdepim-kresources kde-base/kdepim-kresources-3.5.6: 186 files, 25 non-files, 64362.24 KB flame@enterprise ~ % qsize kdepim-kresources kde-base/kdepim-kresources-3.5.6: 191 files, 25 non-files, 37207.401 KB
The first is before the change, the second after the change.
And this is far from being the sole part of kdepim having such a stupid problem.
Sorry KDE guys, but KDEPIM 3.x is a failure, when it comes to properly build and install stuff.
Where is the patch? 🙂
You can find it “here”:http://git.flameeyes.is-a-g… it’s a oneliner, but has to be checked, as it changes the way kdepim gets installed.
I love KDE to bits – I think it’s one of the reasons I stuck with Linux when I was learning my way around and things got tough – but any KDE user knows it’s bloated and overweight, and this example highlights that. 27M of memory freed just by making the one change?I’m not a programmer but common sense says that you link to a file, not link it in. Even if only for ease of maintenance, debugging etc… ?
thanks for being so rigorous about optimizations! If only everyone paid this much attention we could get along with 64MB RAM 🙂
It would be great to have a Strigi analyzer that indexes symbols from elf binaries. Would you care to look into that?
You’re right, there is a problem. But at least the mis-linked version runs, unlike your patched version.Dirk Mueller discovered the problem this week, but didn’t need see the need to rant about it, and just made a fix: As well as the move to lib_LTLIBRARIES you need:-libgwsoap_la_LDFLAGS = $(KDE_RPATH) $(all_libraries)-libgwsoap_la_LIBADD= -lkabc+libgwsoap_la_LDFLAGS = $(KDE_RPATH) $(all_libraries) -no-undefined+libgwsoap_la_LIBADD = $(top_builddir)/libkcal/libkcal.la $(top_builddir)/libkdepim/libkdepim.laAnd consider that next time you find a problem in somebody’s code or build setup, it would be a lot more productive and pleasant to solve the problem with the maintainer of the code (in this case, me) then blog about how we’ve improved things together, instead of drama-queening a mistake into a sucky failure. You might then find people appreciate your work in producing optimisation tools.If we blogged that the sky was falling every time we made an svn ci -m ‘compile++’ fix in someone else’s code, KDE development would be a less fun activity for all of us.
First of all, the script was never designed to be an “optimisation tool”, it was designed to find a different class of problems, and it just happened to finding this.I didn’t test the patch throughout, and I clearly stated so:.bq You can find it here it’s a oneliner, _but has to be checked_, as it changes the way kdepim gets installed.I didn’t contact anybody because the fix was not complete (as it turns out, there are other places that could use similar fixes, so I just wanted to finish investigating about the kdepim package before submitting anything).And sincerely, I do think this is a bit worse than a mistake for a package like kdepim that is far from being a “little” package, and for which there should be more than one maintainer for sure.