Why my Munin plugins are now written in Perl

This post is an interlude between Gentoo-related posts. The reason is that I have one in drafts that requires me to produce some results that I have not yet, so it’ll have to wait for the weekend or so.

You might remember that my original IPMI plugin was written in POSIX sh and awk, rather than bash and gawk as the original one. Since then, the new plugin (that as it turns out might become part of the 2.1 series but not to replace both the old ones, since RHEL and Fedora don’t package a new enough version of Freeipmi) has been rewritten in Perl, so using neither sh nor awk. Similarly, I’ve written a new plugin for sensors which I also wrote in Perl (although in this case the original one also used it).

So why did I learn a new language (since I never programmed in Perl before six months ago) just to get these plugins running? Well, as I said in the other post, the problem was calling the same command so many times, which is why I wanted to go multigraph — but when dealing with variables, sticking to POSIX sh is a huge headache. One of the common ways to handle this is to save to a temporary directory the output of a command and parse that multiple times, but that’s quite a pain, as it might require I/O to disk, and it also means that you have to execute more and more commands. Doing the processing in Perl means that you can save things in variables, or even just parse it once and split it into multiple objects, to be later used for output, which is what I’ve been doing for parsing FreeIPMI’s output.

But why Perl? Well, Munin itself is written in Perl, so while my usual language of choice is Ruby, the plugins are much more usable if doing it in Perl. Yes, there are some alternative nodes written in C and shell, but in general it’s a safe bet that these plugins will be executed on a system that at least supports Perl — the only system I can think of that wouldn’t be able to do so would be OpenWRT, but that’s a whole different story.

There are a number of plugins written in Python and Ruby, some in the official package, but most in the contrib repository and they could use some rewriting. Especially those that use net-snmp or other SNMP libraries, instead of Munin’s Net::SNMP wrapper.

But while the language is of slight concern, some of the plugins could use some rewriting simply to improve their behaviour. As I’ve said, using multigraphs it’s possible to reduce the number of times that the plugin is executed, and thus the number of calls to the backend, whatever that is (a program, or access to /sys), so in many cases plugins that support multiple “modes” or targets through wildcarding can be improved by making them a single plugin. In some cases, it’s even possible to reduce multiple plugins into one, as I did to the various apache_* plugins shipping with Munin itself, replaced on my system with apache_status as provided by the contrib repository, that fetches the server status page only once and then parses it to produce the three graphs that were, before that, created by three different plugins with three different fetches.

Another important trick up our sleeves while working on Munin plugins is dirty config which basically means that (under indication from the node itself), you can make the plugin output the values as well as the configuration itself during the config execution — this saves you one full trip to the node (to fetch the data), and usually that also means it saves you from having to send one more call to the backend. In particular with these changes my IPMI plugin went from requiring six calls to ipmi-sensors per update, for the three graphs, to just one. And since it’s either IPMI on the local bus (which might require some time to access) or over LAN (which takes more time), the difference is definitely visible both in timing, and in traffic — in particular one of the servers at my day job is monitoring another seven servers (which can’t be monitored through the plugin locally), which means that we went from 42 to 7 calls per update cycle.

So if you use Munin, and either have had timeout issues in the past or recently, or you have some time at hand to improve some plugins, you might want to follow what I’ve been doing, and start improving or re-writing plugins to support multigraph or dirtyconfig, and thus improve its performance.

Monitoring HP servers

Sometimes this blog has something like “columns” for long-term topics that keep re-emerging (no pun intended) from time to time. Since I came back to the US last July you can see that one of the big issues I fight with daily is HP servers.

Why is the company I’m working for using HP servers? Mostly because they didn’t have a resident system administrator before I came on board, and just recently they hired an external consultant to set up new servers … the one who set up my nightmare: Apple OS X Server so I’m not sure which of the two options I prefer.

Anyway, as you probably know if you follow my blog, I’ve been busy setting up Munin and Icinga to monitor the status of services and servers — and that helped quite a bit over time. Unfortunately, monitoring HP servers is not easy. You probably remember I wrote a plugin so I could monitor them through IPMI — it worked nicely until I actually got Albert to expose the thresholds in the ipmi-sensors output, then it broke because HP’s default thresholds are totally messed up and unusable, and it’s not possible to commit new thresholds.

After spending quite some time playing with this, I ended up with write access to Munin’s repositories (thanks, Steve!) and I can now gloat be worried about having authored quite a few new Munin plugins (the second generation FreeIPMI multigraph plugin is an example, but I also have a sysfs-based hwmon plugin that can get all the sensors in your system in one sweep, a new multigraph-capable Apache plugin, and a couple of SNMP plugins to add to the list). These actually make my work much easier, as they send me warnings when a problem happens without having to worry about it too much, but of course are not enough.

After finally being able to replace the RHEL5 (without a current subscription) with CentOS 5, I’ve started looking in what tools HP makes available to us — and found out that there are mainly two that I care about: one is hpacucli, which is also available in Gentoo’s tree, and the other is called hp-health and is basically a custom interface to the IPMI features of the server. The latter actually has a working, albeit not really polished, plugin in the Munin contrib repository – which I guess I’ll soon look to transform into a multigraph capable one; I really like multigraph – and that’s how I ended up finding it.

At any rate at that point I realized that I did not add one of the most important checks: the SMART status of the harddrives — originally because I couldn’t get smartctl installed. So I went and checked for it — the older servers are almost all running as IDE (because that’s the firmware’s default.. don’t ask), so those are a different story altogether; the newer servers running CentOS are using an HP controller with SAS drives, using the CCISS (block-layer) driver from the kernel, while one is running Gentoo Linux, and uses the newer, SCSI-layer driver. All of them can’t use smartctl directly, but they have to use a special command: smartctl -d cciss,0 — and then either point it to /dev/cciss/c0d0 or /dev/sda depending on how which of the two kernel drivers you’re using. They don’t provide all the data that they provide for SATA drives, but they provide enough for Munin’s hddtemp_smartctl and they do provide an health status…

For what concerns Munin, your configuration would then be something like this in /etc/munin/plugin-conf.d/hddtemp_smartctl:

[hddtemp_smartctl]
user root
env.drives hd1 hd2
env.type_hd1 cciss,0
env.type_hd2 cciss,1
env.dev_hd1 cciss/c0d0
env.dev_hd2 cciss/c0d0

Depending on how many drives you have and which driver you’re using you will have to edit it of course.

But when I tried to use the default check_smart.pl script from the nagios-plugins package I had two bad surprises: the first is that they try to validate the parameter passed to the plugin to identify the device type to smartctl, refusing to work for a cciss type, and the other that it didn’t consider the status message that is printed by this particular driver. I was so pissed, that instead of trying to fix that plugin – which still comes with special handling for IDE-based harddrives! – I decided to write my own, using the Nagios::Plugin Perl module, and releasing it under the MIT license.

You can find my new plugin in my github repository where I think you’ll soon find more plugins — as I’ve had a few things to keep under control anyway. The next step is probably using the hp-health status to get a good/bad report, hopefully for something that I don’t get already through standard IPMI.

The funny thing about HP’s utilities is that they for the most part just have to present data that is already available from the IPMI interface, but there are a few differences. For instance, the fan speed reported by IPMI is exposed in RPMs — which is the usual way to expose the speed of fans. But on the HP utility, fan speed is actually exposed as a percentage of the maximum fan speed. And that’s how their thresholds are exposed as well (as I said, the thresholds for fan speed are completely messed up on my HP servers).

Oh well, anything else can happen lately, this would be enough for now.

USEless flag: static-libs

You might remember that in my graphical post I tried making it clear that in some cases static libraries are not only unnecessary, but also unusable. It so happens that not everybody has that clear yet, so let me try to explain a bit more where this USE flag is unwarranted entirely.

Language bindings. While it is technically possible to have static extensions built in in Ruby and Python, said situations are very convoluted and aren’t really solid enough to be used for Gentoo packaging; those who want to go with that route are generally using custom build systems, because they require to build in the whole interpreter and binding libraries together.

This means that, to give an idea, these archives shouldn’t have been there to begin with:

/usr/lib/python2.7/site-packages/beagle/beagle.a
/usr/lib/python2.7/site-packages/pygetdata.a
/usr/lib/python2.7/site-packages/numpy/core/lib/libnpymath.a
/usr/lib/python2.7/site-packages/_mathgl.a
/usr/lib/python3.2/site-packages/numpy/core/lib/libnpymath.a

Note: there are no Ruby files in that list, but just because the tinderbox is currently unable to build any Ruby version due to GLIBC 2.14 issues that are yet unresolved upstream.

So if you’re building language bindings, adding a static-libs USE flag is just pointless: remove it and disable static libraries altogether, so that you don’t have to build source files twice (PIC and non-PIC).

Runtime-loaded plugins. It is unfortunate how little developers know about the limitations of runtime-loaded plugins (i.e., all plugins; I’m just making it clear, because buildtime-included plugins are rather built-ins). One of these is the fact that you only need the .so file to load them. So these are certainly smelling funny:

/usr/lib/gstreamer-0.10/libgstmimic.a
/usr/lib/gstreamer-0.10/libgstvp8.a
/usr/lib/libvisual-0.4/morph/morph_alphablend.a
/usr/lib/libvisual-0.4/morph/morph_flash.a
/usr/lib/libvisual-0.4/morph/morph_tentacle.a
/usr/lib/libvisual-0.4/morph/morph_slide.a
/usr/lib/libvisual-0.4/actor/actor_lv_scope.a
/usr/lib/libvisual-0.4/actor/actor_jakdaw.a
/usr/lib/libvisual-0.4/actor/actor_oinksie.a
/usr/lib/libvisual-0.4/actor/actor_bumpscope.a
/usr/lib/libvisual-0.4/actor/actor_gforce.a
/usr/lib/libvisual-0.4/actor/actor_infinite.a
/usr/lib/libvisual-0.4/actor/actor_lv_analyzer.a
/usr/lib/libvisual-0.4/actor/actor_corona.a
/usr/lib/libvisual-0.4/actor/actor_JESS.a

It is true that libtool’s libltdl theoretically allows you to decide at build time whether to build the code in or use it as a plugin, but even then, only one version of the two is ever needed. So if your package only installs one of these, simply remove the USE flag and disable static libs altogether; pretty please.

Also note that while there are packages that allows you to use plugins or builtins, for instance Apache allows you to choose whether to build the modules in or build plugins to load at runtime. But this only works for first-party modules, as building third party modules within Apache requires to have their sources available when building Apache itself.

Libraries relying heavily on plugins. Similarly, if you’re packaging a library that includes a plugins’ system, it is almost certain that the plugins link back to the library. This is actually required to avoid underlinking and to ensure that no undefined symbols are present in the final link of the plugin (well, there are exceptions, but let’s not dig into those right now).

Why in this case you don’t want static libraries? Well, if you were to build a frontend against the static library, it would then load the plugin and the dynamic library as well, not only doubling the loaded code, but causing symbol collisions one with the other, which is fine only until the two versions get out of sync (i.e. you update the library package, without rebuilding the frontend).

This only works if you can convert all the plugins into builtins on the fly, but I know of no project that allows you to do that.

Almost everything else. Don’t get me wrong, I knwo that sometimes we do need static libraries. But it’s also true that in most cases we don’t. I don’t know why people insist on having static-libs an USE flag by default. My personal preference would be to have static-libs as an USE flag when it is actually being used.

This means that unless there are reverse dependencies building against the static libraries, the package should not have the static-libs USE flag at all. Unfortunately this requires the user to enable it forcefully when they want to build something statically (and I admit it happened to me before a time or two).. but let’s be real: most people will not do that.

Indeed, if it wasn’t for PostgreSQL and lvm2/udev (the latter of which I would have avoided with a USE=-static on lvm2), I don’t think I would have noticed the DWARF-4 binutils bug that I had lingering on my system for over a month!

More pointless .la files

The long-time followers of my blog will almost surely know of my fight against the pollution from libtool archive (.la) files in Gentoo packages. I already said that they are often useless, or even dangerous when dealing with cross-compilation, and that they create problems that can be solved only with --as-needed (over-linking).

But in particular, I’ve started making the tinderbox warn me when libtool archive files are found in three particular sub-trees where they are absolutely not useful: Python modules, Ruby extensions and PAM plugins. Neither of these use libtool for loading the extensions (for quite obvious reasons), so the .la file is not going to be used for loading; you also shouldn’t link against dynamically-loaded modules (and libtool warns you about that if you try to do it), so the .la file is not used for build-time linking. It’s simply unused.

But it goes even worse than that: the warning has also made me notice that quite a few packages not only install the libtool archive, but build and install a static archive as well! This is not only an extra file installed (and in the search path of those implementations) but also force libtool to build all the source code twice: with and without Position-Independent Code (the static archives are usually non-PIC but shared objects are PIC).

In the bugs I reported for this kind of problems I also linked my own documentation on building plugins with libtool, so maybe the Gentoo developers reaching the bugs would be able to fix both up- and down-stream the issue of double build of both PIC and non-PIC objects. Not that I count on it but it might be possible.

Plugins aren’t always a good choice

I’ve been saying this for quite a while, probably one of the most on-topic post has been written a few months ago but there are some indications about it in posts about xine and other again.

I used to be an enthusiast about plugin interfaces; with time, though, I started having more and more doubts about their actual usefulness — it’s a tract I really much like in myself, I’m fine with reconsidering my own positions over time, deciding that I was wrong; it happened before with other things, like KDE (and C++ in general).

It’s not like I’m totally against the use of plugins altogether. I only think that they are expensive in more ways than one, and that their usefulness is often overstated, or tied to other kind of artificial limitations. For instance, dividing a software’s features over multiple plugins makes it easier for the binary distributions to package them, usually: they only have to ship a package with the main body of the software, and many for the plugins (one per plugin might actually be too much so sometimes they might be grouped). This works out pretty well for both the distribution and, usually, the user: the plugins that are not installed will not bring in extra dependencies, they won’t take time to load and they won’t use memory for either code nor data. It basically allows binary distribution to have a flexibility to compare with Gentoo’s USE flags (and similar options in almost any other source-based distribution).

But as I said this comes with costs, that might or might not be worth it in general. For instance, Luca wanted to implement plugins for feng similarly to what Apache and lighttpd have. I can understand his point: let’s not load code for the stuff we don’t have to deal with, which is more or less the same reason why Apache and lighttpd have modules; in the case of feng, if you don’t care about access log, why should you be loading the access load support at all? I can give you a couple of reasons:

  • because the complexity of managing a plugin to deal with the access log (or any other similar task) is higher than just having a piece of static code that handles that;
  • because the overhead of having a plugin loaded just to do that is higher than that of having the static code built in and not enabled into configuration.

The first problem is a result of the way a plugin interface is built: the main body of the software cannot know about its plugins in too specific ways. If the interface is a very generic plugin interface, you add some “hook locations” and then it’s the plugin’s task to find how to do its magic, not the software’s. There are some exceptions to this rule: if you have a plugin interface for handling protocols, like the KIO interface (and I think gvfs has the same) you get the protocol from the URL and call the correct plugin, but even then you’re leaving it to the plugin to deal with doing its magic. You can provide a way for the plugin to tell the main body what it needs and what it can do (like which functions it implements) but even that requires the plugins to be quite autonomous. And that means also being able to take care of allocating and freeing the resources as needed.

The second problem is not only tied to the cost of calling the dynamic linker dynamically to load the plugin and its eventual dependencies (which is a non-trivial amount of work, one has to say), also by the need for having code that deals with finding the modules to load, the loading of those modules, their initialisation, keeping a list of modules to call at any given interface point, and two more points: the PIC problem and the problem of less-than-page-sized segments. This last problem is often ignored, but it’s my main reason to dislike plugins when they are not warranted for other reasons. Given a page size of 4KiB (which is the norm on Linux for what I know), if the code is smaller than that size, it’ll still require a full page (it won’t pack with the rest of the software’s code areas); but at least code is disk-backed (if it’s PIC, of course), it’s worse for what concerns variable data, or variable relocated data, since those are not disk-backed, and it’s not rare that you’d be using a whole page for something like 100 bytes of actual variables.

In the case of the access log module that Luca wrote for feng, the statistics are as such:

flame@yamato feng % size modules/.libs/mod_accesslog.so
   text    data     bss     dec     hex filename
   4792     704      16    5512    1588 modules/.libs/mod_accesslog.so

Which results in two pages (8KiB) for bss and data segments, neither disk-backed, and two disk-backed pages for the executable code (text): 16KiB of addressable memory for a mapping that does not reach 6KiB, it’s a 10KiB overhead, which is much higher than 50%. And that’s the memory overhead alone. The whole overhead, as you might guess at this point, is usually within 12KiB (since you got three segments, and each can have at most one byte less than page size as overhead — it’s actually more complex than this but let’s assume this is true).

It really doesn’t sound like a huge overhead by itself, but you have to always judge it compared to the size of the plugin itself. In the case of feng’s access log, you got a very young plugin that lacks a lot of functionality, so one might say that with the time it’ll be worth it… so I’d like to show you the size statistics for the Apache modules on the very server my blog is hosted. Before doing so, though, I have to remind you one huge difference: feng is built with most optimisations turned off, while Apache is built optimised for size; they are both AMD64 though so the comparison is quite easy.

flame@vanguard ~ $ size /usr/lib64/apache2/modules/*.so | sort -n -k 4
   text    data     bss     dec     hex filename
   2529     792      16    3337     d09 /usr/lib64/apache2/modules/mod_authn_default.so
   2960     808      16    3784     ec8 /usr/lib64/apache2/modules/mod_authz_user.so
   3499     856      16    4371    1113 /usr/lib64/apache2/modules/mod_authn_file.so
   3617     912      16    4545    11c1 /usr/lib64/apache2/modules/mod_env.so
   3773     808      24    4605    11fd /usr/lib64/apache2/modules/mod_logio.so
   4035     888      16    4939    134b /usr/lib64/apache2/modules/mod_dir.so
   4161     752      80    4993    1381 /usr/lib64/apache2/modules/mod_unique_id.so
   4136     888      16    5040    13b0 /usr/lib64/apache2/modules/mod_actions.so
   5129     952      24    6105    17d9 /usr/lib64/apache2/modules/mod_authz_host.so
   6589    1056      16    7661    1ded /usr/lib64/apache2/modules/mod_file_cache.so
   6826    1024      16    7866    1eba /usr/lib64/apache2/modules/mod_expires.so
   7367    1040      16    8423    20e7 /usr/lib64/apache2/modules/mod_setenvif.so
   7519    1064      16    8599    2197 /usr/lib64/apache2/modules/mod_speling.so
   8583    1240      16    9839    266f /usr/lib64/apache2/modules/mod_alias.so
  11006    1168      16   12190    2f9e /usr/lib64/apache2/modules/mod_filter.so
  12269    1184      32   13485    34ad /usr/lib64/apache2/modules/mod_headers.so
  12521    1672      24   14217    3789 /usr/lib64/apache2/modules/mod_mime.so
  15935    1312      16   17263    436f /usr/lib64/apache2/modules/mod_deflate.so
  18150    1392     224   19766    4d36 /usr/lib64/apache2/modules/mod_log_config.so
  18358    2040      16   20414    4fbe /usr/lib64/apache2/modules/mod_mime_magic.so
  18996    1544      48   20588    506c /usr/lib64/apache2/modules/mod_cgi.so
  20406    1592      32   22030    560e /usr/lib64/apache2/modules/mod_mem_cache.so
  22593    1504     152   24249    5eb9 /usr/lib64/apache2/modules/mod_auth_digest.so
  26494    1376      16   27886    6cee /usr/lib64/apache2/modules/mod_negotiation.so
  27576    1800      64   29440    7300 /usr/lib64/apache2/modules/mod_cache.so
  54299    2096      80   56475    dc9b /usr/lib64/apache2/modules/mod_rewrite.so
 268867   13152      80  282099   44df3 /usr/lib64/apache2/modules/mod_security2.so
 288868   11520     280  300668   4967c /usr/lib64/apache2/modules/mod_passenger.so

The list is ordered for size of the whole plugin (summed up, not counting padding); the last three positions are definitely unsurprisingly, although it surprises me the sheer size of the two that are not part of Apache itself (and I start to wonder whether they link something in statically that I missed). The fact that the rewrite module was likely the most complex plugin in Apache’s distribution never left me.

As you can see, almost all plugins have vast overhead especially for what concerns the bss segment (all of them have at least 16 bytes used, and that warrants a whole page for them: 4080 bytes wasted each); the data segment is also interesting: only the two external ones have more than a page worth of variables (which also is suspicious to me). When all the plugins are loaded (like they most likely are right now as well on my server) there are at least 100KiB of overhead; just for the sheer fact that these are plugins and thus have their own address space. Might not sound like a lot of overhead indeed, since Apache is requesting so much memory already, especially with Passenger running, but it definitely doesn’t sound like a good thing for embedded systems.

Now I have no doubt that a lot of people like the fact that Apache has all of those as plugins as they can then use the same Apache build across different configurations without risking to have in memory more code and data than it’s actually needed, but is that right? While it’s obvious that it would be impossible to drop the plugin interface from Apache (since it’s used by third-party developers, more on that later), I would be glad if it was possible to build in the modules that come with Apache (given I can already choose which ones to build or not in Gentoo). Of course I also am using Apache with two configurations, and for instance the other one does not use the authentication system for anything, and this one is not using CGI, but is the overhead caused by the rest of modules worth the hassle, given that Apache already has a way to not initialise the unused built-ins?

I named above “third party developers” but I have to say now that it wasn’t really a proper definition, since it’s not just what third parties would do, it might very well be the original developers who might want to make use of plugins to develop separate projects for some (complex) features, and have different release handling altogether. For uses like that, the cost of plugins is often justifiable; and I am definitely not against having a plugin interface in feng. My main beef is when the plugins are created for functions that are part of the basic featureset of a software.

Another unfortunately not uncommon problem with plugins is that the interface might be skewed by bad design, like the case was (and is) for xine: when trying to open a file, it has to pass through all the plugins, so it loads all of them into memory, together with the libraries they depend on, to ask each of them to test the current file; since plugins cannot really be properly unloaded (and it’s not just a xine limitation) the memory will still be used, the libraries will still be mapped into memory (and relocated, causing copy on write, and thus, more memory) and at least half the point of using plugins has gone away (the ability to only load the code that is actually going to be used). Of course you’re left with the chance that an ABI break does not kill the whole program, but just the plugin, but that’s a very little advantage, given the cost involved in plugins handling. And the way xine was designed, it was definitely impossible to have third-party plugins developed properly.

And to finish off, I said before that plugins cannot be cleanly unloaded: the problem is not only that it’s difficult to have proper cleanup functions for plugins themselves (since often the allocated resources are stored within state variables), but also because some libraries (used as dependency) have no cleanup altogether, and they rely (erroneously) on the fact that they won’t be unloaded. And even when they know they could be unloaded, the PulseAudio libraries, for instance, have to remain loaded because there is no proper way to clean up Thread-Local Storage variables (and a re-load would be quite a problem). Which drives away another point of using plugins.

I leave the rest to you.

Plugins and static libraries don’t mix well

There is one interesting thing that, I’m afraid, most devleopers happen to ignore, either spontaneously, or because they really don’t know about them. One of this is the fact that static libraries and plugins usually don’t mix well. Although I have to warn you, that’s not an absolute and they can easily designed to work fine.

The main problem though is to ponder whether it is useful to use static libraries and plugins together, and then it’s to find out if it’s safe to or not. Let’s start from the basis. What are static libraries used for? Mostly for two reasons: performances, and not having to depend on the dynamic version of the same library in the system. If performance of the library is a problem, it’s much more likely that the culprit is the plugins system rather than the dynamic nature of the library; I have said something about it in the past, although I didn’t go much in details and I haven’t had time to continue the discussion yet.

For what concerns dependencies, the plugins usually need a way to access the functions provided by the main library; this means there is an ABI dependency between the two; now the plugins might not link against the library directly, to support static libraries usage, but it also means that if the internal ABI changes in any way between versions, you’re screwed.

What does this mean? That in most cases when you have plugins, you don’t want to have static libraries around; this means that you also don’t need the .la files and so you have quite a bit of cleanup.

More to the point, if you’re building a plugin, you don’t want to build the static version at all, since the plugin will be opened with the dlopen() interface, from the dynamic version of the library (the module). Since not always upstream remember to disable the static archive building in their original build system, ebuild authors should take care of disabling them, either with --disable-static or by patching the build system (if you don’t want to stop all static lib building). And this is not my idea but a proper development procedure (and no, it does not mean any discussion: if it’s a plugin — and it’s not possible to make it a builtin — you shouldn’t install the archive file! Full stop!).

Now, you can see where this brings us again: more .la files that are often currently installed and are not useful at all. Like .la files for PAM modules (libpam does not load them through the .la so they are not useful — and this is definitely the word of the PAM maintainer! And for PAM-related packages, that word is The Word). Let’s try to continue this way, shall we? From the leaves.

Identifying pointless .la files for plugins

At this point I expect most users to know that .la files are evil and they are often useless and that removing them can save you from having to do work when packages drop them, either upstream or in Gentoo. Unfortunately, most developers tend to be overly conservative and keep them around even when not needed even remotely.

One of the cases for which I have said times and times again that .la files should be removed is for plugins; the .la files for plugins are useful when you use libltdl to wrap around plugins loading (and not even needed), but are totally pointless when using the straight dlopen() call.

Unfortunately even when that’s the case, it’s hard for ebuild developers to feel confident that the files are unneeded, so here it comes a practical case study to identify when they are not used at all. First step is to decide which package to test; I’ll go with eog since I have noticed this before and I know they are not used.

The eog package installs some .la files:

% qlist eog | fgrep .la
/usr/lib64/eog/plugins/libreload.la
/usr/lib64/eog/plugins/libstatusbar-date.la
/usr/lib64/eog/plugins/libfullscreen.la

Now we can see that the eog/plugins directory is where it’ll be looking for plugins, so we’ll start eog through strace and see if it tries to load any of that:

% strace -e open eog |& fgrep eog/plugins
open("/usr/lib64/eog/plugins/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 14
open("/usr/lib64/eog/plugins/reload.eog-plugin", O_RDONLY) = 15
open("/usr/lib64/eog/plugins/statusbar-date.eog-plugin", O_RDONLY) = 15
open("/usr/lib64/eog/plugins/fullscreen.eog-plugin", O_RDONLY) = 15

A quick look at the strace outout can let us see that it’s not loading the plugins at all; indeed in this case eog was started without any plugin enabled, and it only opened the .eog-plugin files, which are ini-like files describing the plugins and their information; I’ll write more about this in the future when I’ll resume my posts about plugins which I’ve been slacking off from for a while. So let’s enable some plugins (all three of them!) and try again.

% strace -e open eog |& fgrep eog/plugins
open("/usr/lib64/eog/plugins/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 14
open("/usr/lib64/eog/plugins/reload.eog-plugin", O_RDONLY) = 15
open("/usr/lib64/eog/plugins/statusbar-date.eog-plugin", O_RDONLY) = 15
open("/usr/lib64/eog/plugins/fullscreen.eog-plugin", O_RDONLY) = 15
open("/usr/lib64/eog/plugins/libreload.so", O_RDONLY) = 16
open("/usr/lib64/eog/plugins/libstatusbar-date.so", O_RDONLY) = 16
open("/usr/lib64/eog/plugins/libfullscreen.so", O_RDONLY) = 16

Now it looks better: it loads the .so directly; what does this mean? Simple: the package is most likely using dlopen() and not at all the libltdl wrapper. The .la files are not looked at, at all, so they can be removed without thinking twice.

With this method you can ensure that the .la files are just and only a side effect of using libtool rathre than something that is actually used by the software; to do that, though, you’ve got to make sure that the .so is at least loaded, otherwise you might not have loaded plugins at all (see the first output above).

Another common mistake is to consider the .la files needed or not depending on libltdl linking; this is not true in either case: software not linking to libltdl might make use of the .la files (kdelibs 3 for instance have their internal libltdl modified copy), and software using libltdl might not need .la files (PulseAudio for instance, that does not install .la files at all, in Gentoo).

This post brought to you by having so much work to do that just 2% of my actual mind was free!

Plugins design: interface style

In my previous post of plugins versus builtins I’ve given a few reasons to make use of plugins, explaining the bad side of them, which left up in the air the fact that you can very well decide to support both plugins and builtins – I think VLC does that by the way – and let the user (or the distributor) choose at build-time whether to link them in statically or link them externally.

To make this possible, you’ve obviously got to deal with a few different design decisions; while plugins have more or less the same basic idea for design, their implementation may be quite different. Since usually you need more than one function from a plugin (at least setup, action and teardown), you have different methods to handle them.

For instance, you can have one symbol per function, and use dlopen() (or similar) for each of them to fill in a structure to call the various interfaces, or you can have a single structure already filled in with the pointers to the (static) functions, or finally you can have a single initialization function that fills in the structure and return it to the calling application.

While the difference between these approaches may seem to be minimal, it actually can be quite a difference: calling into the dynamic loader multiple times can waste time, so having a single entrypoint makes perfect sense from a design point of view. But the remaining two approaches are also very different.

From a logic point of view of a theoretical programmer, having a structure, an object, already pre-filled with the pointers of the static member functions is the best choice. This is, after all, how the C++ vtables work: they are little more than a structure with functions’ addresses. But if your background is more practical than theoretical, you might have already noticed a problem here; when using position independent code (PIC or PIE) all the objects containing pointers (doesn’t matter whether they point to functions or data) need to be relocated at runtime (that’s what .data.rel is used for).

When relocating objects, the .data.rel section of an executable object (shared or not), will become dirty, causing it not to be shared among processes any longer. As I’ve written in the previous post, prelink does not work on them which means it won’t be able to alleviate the effect of relocation on the object (like it would be for normal shared libraries). Also, since plug-ins are isolated shared objects, they don’t share the sections of the main program – I already said this before – the relocation of small “vtable-like” objects will cause the copy-on-write of a full page (4KiB) for each plugin loaded, even when the object itself is just a few bytes big.

My preferred solution is to have a single interface function, which, once called, will fill-in a heap-allocated function with the pointers to the rest of the interface function. This option allows to just have a single symbol per plugin that needs to be looked up, and at the same time sidesteps the problem of relocation (the pointers will be calculated at runtime, just like a relocation, but without the memory hit).

The next post in the series will try to focus more on the style of the initialisation function, to be shared between plugins and builtins.

Plugins and builtins what’s the best?

Summary: this post is going to address a request from a reader, about the best way to implement plugins and builtins. If everything goes as I plan (but it really happens very rarely for multi-post topics!), I’ll be discussing today the actual differences between the approach (focusing mostly on explaining the correct reasoning behind the use of plug-ins), then I’ll move to giving a few pointers about their implementations in C, and finally write something about the needed build system changes (which are already partially documented in my guide ).

There is lot of software out there that makes use of plug-in systems of different kinds. For what concerns this blog post, I’m going to focus on plug-in systems where a main application or library loads smaller, semi-autonomous modules for executing primary or secondary tasks. But to be even more focused, I’m not going to talk about plugin systems for interpreted languages like Ruby, Python, Perl, or virtual machine languages like Java and C#/Mono. My reason to exclude these from the list is that they have different rules for loading code, and indeed for them the plugins come to be near-free. On the other hand, compiled languages like C, C++ and similar have harder barriers to work through.

If you follow my blog, are a Gentoo user, or have had to deal with libtool ever before, you probably know already that building shared objects (which is what compiled plug-ins are!) is not as easy as building a standard executable. Indeed shared objects at least in Linux and *BSD, require to be built with PIC, or will cause text relocations; in both cases they require more memory (they both are Copy-on-Write, the former for .data.rel relocations, the other for .text relocations). PIC code, also, require the x86 register ebx to be reserved for use as base pointer for the global offset table, which means that the compiler and the hand-written assembly code have one less register to use.

With plug-ins, you not only have to deal with the usual problems related to shared objects, like the already-mentioned copy-on-write sections and the loss of one register, but you also have a few more issues, which can make them pretty expensive in term of resources, just as a shortlist:

  • you cannot mitigate the problem with prelink, as I’ve stated previously the current implementation of prelink does not take into consideration plugins at all, which not only means it won’t be able to reduce the copy on write caused by the actual plugin objects, but also that it cannot take the proper step to make sure that all the libraries don’t end up trying to use the same address (when trying to preserve the address space); this because it cannot understand that the libraries linked to by the various plugins will be loaded in the same memory area;
  • the plugins will call into the dynamic loader and force it to load further libraries in the address space if they are not there yet, this actually requires a non trivial amount of work from one part of the operating system that usually stops after the program is loaded and running;
  • plugins will require accessing otherwise private functions of the software that is loading them; this means that the library (or the final executable if there is no intermediate library) will have to export more symbols; exported symbols also have tighter rules for their optimisation (since they will be called form outside of the currently-built module), and require bigger PLTs (Procedure Linking Tables) as well as hash tables and so on;
  • while plugins share the address space of the process loading them, they don’t share neither on-disk nor in memory sections; this can be easily seen in plugins like the one shipping with xine-lib: any process that will load them will end up wasting 4KiB of .data.rel per plugin as they declare an exported data structure (usually much smaller than 4KiB, but that’s the page size) which suffers from copy-on-write; since all the sections are mapped into multipliers of the page size (4KiB on mos systems), even if each plugin were to use just 1KiB of private memory areas it will end up using at least four times as much of resident memory;
  • finding symbol collisions can easily become tricky when they happens between two libraries loaded by two different plugins since they might never seen to be loaded together, just by looking at the dependencies on the files, this is the same problem as prelink above.

At this point you might think that plugins are inherently bad and should always be avoided; on the other hand you might notice that, in a standard desktop system, you find lots of software that actually use plug-ins. Why is it this way? Well the problem is that while you do have lots of nasty side-effects with the use of plugins, they have lots of advantages over building all the support in the software, especially they can work when there is no way to build the support in the software itself.

For instance, browser plugins cannot really be built into the browser; things like the Flash Player or other similar tools need to be loaded from outside. Themes for Gtk+ and Qt that don’t ship with the libraries themselves cannot be built in them, since you cannot merge and separate the modules that easily, so they also need to be designed as plugins. But it does not stop here. If you do load the plugins conditionally, they can save quite a bit of memory.

If you think that simply linking against a library, without using it explicitly, can actually execute code that wastes cpu and memory resources, through constructor functions and static initializators, you can easily understand that being able to just load a subset of the modules at any given time can easily be a save in memory, both shared and static. For this to work, though, you need to make sure that the plugins are loaded only if explicitly needed, may it be via an explicit request to load them (themes, applets) or via smart databases (browser plugins). Just being able to unload them might not be enough: there are libraries that once loaded cannot be unloaded until the process completes; PulseAudio for instance does that. And xine, well, is a good example how not to do it: not only, as I said, each plugin uses up between 8 and 12 KiB of memory without compelling reasons but being the design of the plugin systems, but also lacks a proper way to decide which subset of plugins to load, which results in all the plugins being loaded at all time.

Of course, sometimes the plugins seem to just be pointless because, as for the case of xine, they are always loaded, or they are entirely shipped with the application, with no header or interface to actually add more. In this case, one would probably be expecting to just being able to choose which feature to build in (in Gentoo via USE flags) and be done with it. In cases like these, the choice of using plugins over optional build-ins can easily be more political than technical.

Once you think of it, USE flags in Gentoo are very easy to use to choose what to load and what not, but for binary distributions it’s not that easy to provide packages for an arbitrary number of combinations of selected build-ins for the same basic application; on the other hand for them is very easy to provide a number of sub-packages with the various plugins. In these terms, the ability to choose at build time whether to build multiple plugins or merge all them in a single executable is likely a desirable feature; source-based distributions like Gentoo, FreeBSD ports and pkgsrc will prefer the built-ins, providing their users with quite nicely optimised software, while binary distributions like Fedora and Debian can provide multiple binary packages with plugins.

Plugins should not go in /usr/lib

Although most of the software in Portage already follows the correct rule of not putting their plugins in /usr/lib or equivalent directories, during my collision detection analysis I did find a few that instead pollute the standard library path with their plugins.

In general, I see way too much pollution in the library path, and that is quite bad since the loader will have to iterate through all of it whenever it has to find a library that is not in the cache, or if environment variables are set that forces it to look for libraries in a different order. Also, the linker (which is quite slow already) needs to look there to find the libraries to link to, which means more and more work every time, to scan everything.

For this reason, the amount of files in /usr/lib that are not needed to be there should really be reduced drastically, this means for instance not installing shell scripts there, like quite a bit of software does, and especially not installing plugins, since those would be picked up by the ldconfig utility and written in a cachefile that should really be as small as possible.

It’s not just that though, there are more problems with software installing plugins, for instance I noticed that in the PAM modules directory /lib/security there is one plugin with a full-versioned name (that is with the final .0.0 which is not needed by PAM modules) and another that uses the lib prefix (also unneeded); both are likely built with libtool by somebody who knows not how to properly build plugins.

I really should be writing another further test to make sure that the packages get installed following the usual layout, I just don’t know how for now. It would be a nice way to identify “rogue” packages that install outside of the standard-defined directories.

In general, whenever you want to install plugins you should do so by putting them in a separate directory inside /usr/lib, usually pkglibdir when using automake (defaults to /usr/lib/PACKAGE_NAME). This also allows a much simpler approach to plugins loading. And if you’re not really allowing plug-ins to be added after the ones compiled at build time, then you should probably not use plugins but rather build them in.

I guess there is more work for my tinderbox now…