Plugins and builtins what’s the best?

Summary: this post is going to address a request from a reader, about the best way to implement plugins and builtins. If everything goes as I plan (but it really happens very rarely for multi-post topics!), I’ll be discussing today the actual differences between the approach (focusing mostly on explaining the correct reasoning behind the use of plug-ins), then I’ll move to giving a few pointers about their implementations in C, and finally write something about the needed build system changes (which are already partially documented in my guide ).

There is lot of software out there that makes use of plug-in systems of different kinds. For what concerns this blog post, I’m going to focus on plug-in systems where a main application or library loads smaller, semi-autonomous modules for executing primary or secondary tasks. But to be even more focused, I’m not going to talk about plugin systems for interpreted languages like Ruby, Python, Perl, or virtual machine languages like Java and C#/Mono. My reason to exclude these from the list is that they have different rules for loading code, and indeed for them the plugins come to be near-free. On the other hand, compiled languages like C, C++ and similar have harder barriers to work through.

If you follow my blog, are a Gentoo user, or have had to deal with libtool ever before, you probably know already that building shared objects (which is what compiled plug-ins are!) is not as easy as building a standard executable. Indeed shared objects at least in Linux and *BSD, require to be built with PIC, or will cause text relocations; in both cases they require more memory (they both are Copy-on-Write, the former for .data.rel relocations, the other for .text relocations). PIC code, also, require the x86 register ebx to be reserved for use as base pointer for the global offset table, which means that the compiler and the hand-written assembly code have one less register to use.

With plug-ins, you not only have to deal with the usual problems related to shared objects, like the already-mentioned copy-on-write sections and the loss of one register, but you also have a few more issues, which can make them pretty expensive in term of resources, just as a shortlist:

• you cannot mitigate the problem with prelink, as I’ve stated previously the current implementation of prelink does not take into consideration plugins at all, which not only means it won’t be able to reduce the copy on write caused by the actual plugin objects, but also that it cannot take the proper step to make sure that all the libraries don’t end up trying to use the same address (when trying to preserve the address space); this because it cannot understand that the libraries linked to by the various plugins will be loaded in the same memory area;
• the plugins will call into the dynamic loader and force it to load further libraries in the address space if they are not there yet, this actually requires a non trivial amount of work from one part of the operating system that usually stops after the program is loaded and running;
• plugins will require accessing otherwise private functions of the software that is loading them; this means that the library (or the final executable if there is no intermediate library) will have to export more symbols; exported symbols also have tighter rules for their optimisation (since they will be called form outside of the currently-built module), and require bigger PLTs (Procedure Linking Tables) as well as hash tables and so on;
• while plugins share the address space of the process loading them, they don’t share neither on-disk nor in memory sections; this can be easily seen in plugins like the one shipping with xine-lib: any process that will load them will end up wasting 4KiB of .data.rel per plugin as they declare an exported data structure (usually much smaller than 4KiB, but that’s the page size) which suffers from copy-on-write; since all the sections are mapped into multipliers of the page size (4KiB on mos systems), even if each plugin were to use just 1KiB of private memory areas it will end up using at least four times as much of resident memory;
• finding symbol collisions can easily become tricky when they happens between two libraries loaded by two different plugins since they might never seen to be loaded together, just by looking at the dependencies on the files, this is the same problem as prelink above.

At this point you might think that plugins are inherently bad and should always be avoided; on the other hand you might notice that, in a standard desktop system, you find lots of software that actually use plug-ins. Why is it this way? Well the problem is that while you do have lots of nasty side-effects with the use of plugins, they have lots of advantages over building all the support in the software, especially they can work when there is no way to build the support in the software itself.

For instance, browser plugins cannot really be built into the browser; things like the Flash Player or other similar tools need to be loaded from outside. Themes for Gtk+ and Qt that don’t ship with the libraries themselves cannot be built in them, since you cannot merge and separate the modules that easily, so they also need to be designed as plugins. But it does not stop here. If you do load the plugins conditionally, they can save quite a bit of memory.

If you think that simply linking against a library, without using it explicitly, can actually execute code that wastes cpu and memory resources, through constructor functions and static initializators, you can easily understand that being able to just load a subset of the modules at any given time can easily be a save in memory, both shared and static. For this to work, though, you need to make sure that the plugins are loaded only if explicitly needed, may it be via an explicit request to load them (themes, applets) or via smart databases (browser plugins). Just being able to unload them might not be enough: there are libraries that once loaded cannot be unloaded until the process completes; PulseAudio for instance does that. And xine, well, is a good example how not to do it: not only, as I said, each plugin uses up between 8 and 12 KiB of memory without compelling reasons but being the design of the plugin systems, but also lacks a proper way to decide which subset of plugins to load, which results in all the plugins being loaded at all time.

Of course, sometimes the plugins seem to just be pointless because, as for the case of xine, they are always loaded, or they are entirely shipped with the application, with no header or interface to actually add more. In this case, one would probably be expecting to just being able to choose which feature to build in (in Gentoo via USE flags) and be done with it. In cases like these, the choice of using plugins over optional build-ins can easily be more political than technical.

Once you think of it, USE flags in Gentoo are very easy to use to choose what to load and what not, but for binary distributions it’s not that easy to provide packages for an arbitrary number of combinations of selected build-ins for the same basic application; on the other hand for them is very easy to provide a number of sub-packages with the various plugins. In these terms, the ability to choose at build time whether to build multiple plugins or merge all them in a single executable is likely a desirable feature; source-based distributions like Gentoo, FreeBSD ports and pkgsrc will prefer the built-ins, providing their users with quite nicely optimised software, while binary distributions like Fedora and Debian can provide multiple binary packages with plugins.