RDEPEND safety

I’m hoping this post is going to be useful for all the devs and devs to be that want to be sure their ebuilds have proper runtime dependencies. It has sprouted by the fact it seems at least a few developers were oblivious of the implications of what I’m going to describe (which I described briefly on gentoo-core a few days ago, without any response).

First of all, I have to put my hands forwards and say that I’m going to focus on just the binary ELF packages, and this is far from a complete check for proper runtime dependencies. Scripting code is much more difficult to check, while Java is at least somewhat simpler thanks to the Java team’s script.

So you got a simple software that installs ELF executable fils or shared libraries, and you want to make sure all the needed dependencies are listed. The most common mistake there is to check the link chain with ldd (which is just a special way to invoke the loader, dumping out the called libraries). This would most likely show you a huge amount of false positives:

yamato ~ # ldd /usr/bin/mplayer
    linux-gate.so.1 =>  (0xf7f8d000)
    libXext.so.6 => /usr/lib/libXext.so.6 (0xf7eec000)
    libX11.so.6 => /usr/lib/libX11.so.6 (0xf7dfd000)
    libpthread.so.0 => /lib/libpthread.so.0 (0xf7de5000)
    libXss.so.1 => /usr/lib/libXss.so.1 (0xf7de1000)
    libXv.so.1 => /usr/lib/libXv.so.1 (0xf7ddb000)
    libXxf86vm.so.1 => /usr/lib/libXxf86vm.so.1 (0xf7dd4000)
    libvga.so.1 => /usr/lib/libvga.so.1 (0xf7d52000)
    libfaac.so.0 => /usr/lib/libfaac.so.0 (0xf7d40000)
    libx264.so.65 => /usr/lib/libx264.so.65 (0xf7cae000)
    libmp3lame.so.0 => /usr/lib/libmp3lame.so.0 (0xf7c37000)
    libncurses.so.5 => /lib/libncurses.so.5 (0xf7bf3000)
    libpng12.so.0 => /usr/lib/libpng12.so.0 (0xf7bcd000)
    libz.so.1 => /lib/libz.so.1 (0xf7bb9000)
    libmng.so.1 => /usr/lib/libmng.so.1 (0xf7b52000)
    libasound.so.2 => /usr/lib/libasound.so.2 (0xf7a9a000)
    libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0xf7a13000)
    libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0xf79e6000)
    libmad.so.0 => /usr/lib/libmad.so.0 (0xf79cd000)
    libtheora.so.0 => /usr/lib/libtheora.so.0 (0xf799b000)
    libm.so.6 => /lib/libm.so.6 (0xf7975000)
    libc.so.6 => /lib/libc.so.6 (0xf7832000)
    libxcb-xlib.so.0 => /usr/lib/libxcb-xlib.so.0 (0xf782f000)
    libxcb.so.1 => /usr/lib/libxcb.so.1 (0xf7815000)
    libdl.so.2 => /lib/libdl.so.2 (0xf7810000)
    /lib/ld-linux.so.2 (0xf7f71000)
    libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0xf77ef000)
    librt.so.1 => /lib/librt.so.1 (0xf77e6000)
    libexpat.so.1 => /usr/lib/libexpat.so.1 (0xf77bf000)
    libogg.so.0 => /usr/lib/libogg.so.0 (0xf77b9000)
    libXau.so.6 => /usr/lib/libXau.so.6 (0xf77b4000)
    libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0xf77ae000)

In this output, for instance, you can see listed the XCB libraries, and Expat, so you could assume that MPlayer depends on those. On the other hand, it really doesn’t, and they are just indirect dependencies, that the loader will have to load anyway. To avoid being fooled by that the solution would be to check the file itself for the DT_NEEDED entries in the .dynamic section of the ELF file. This can be achieved by checking the output of readelf -d or much more quickly by using scanelf -n:

yamato ~ # scanelf -n /usr/bin/mplayer
 TYPE   NEEDED FILE 
ET_EXEC libXext.so.6,libX11.so.6,libpthread.so.0,libXss.so.1,libXv.so.1,libXxf86vm.so.1,libvga.so.1,libfaac.so.0,libx264.so.65,libmp3lame.so.0,libncurses.so.5,libpng12.so.0,libz.so.1,libmng.so.1,libasound.so.2,libfreetype.so.6,libfontconfig.so.1,libmad.so.0,libtheora.so.0,libm.so.6,libc.so.6 /usr/bin/mplayer 

As you can see here MPlayer does not use either of those libraries, which means that they should not be in MPlayer’s RDEPEND. There is, though, another common mistake here. If you don’t use --as-needed (especially not forcing it), you’re going to get indirect and misguided dependencies . So you can only trust DT_NEEDED when the system has been built with --as-needed from the start. This is not always the case and thus you can get polluted dependencies. And thanks to the fact that now the linker silently ignores --as-needed on broken libraries this is likely to create a bit of stir.

One of the entries in my ever so long TODO list (explicit requests for tasks during donation helps, just so you know) is to write a ruby-elf based script that can check the dependencies without requiring the whole system to be built with --as-needed. It would probably be a lot like the script that Serkan pointed me at for Java, but for ELF files.

After you got the required dependencies are seen by the loader right, though, your task is not complete yet. A program has more dependencies that it might appear to have, since it might require data files to be opened, like icon themes and similar, but also more important dependencies in form of other programs or libraries. And that is not always too obvious. While you can check if the software is using the dlopen() interface to load dynamically further libraries, again using scanelf, that is not going to tell you much and you have to check the source code. Also the program can call another through way of the exec family of functions, or through system(). And even if your program does not call any of these functions you cannot be sure that you got the complete dependencies right without opening it

This is because libraries adds indirection to these things too. The gmodule interface in glib allows for dynamically loading plugins, and can actually load plugins you don’ t see and check, and Qt (used to) provide a QProcess class that allows to execute other software.

All in all, even for non-scripting programs, you really need to pay attention to the sources to be safe that you got your dependencies right and you should never ever rely purely on the output of a script. Which is another reason why I think that most work in Gentoo cannot be fully automated, not just yet at least. At any rate, I’m hoping to provide developers with an usable script one day soonish, at least it’ll be a step closer than it is now.