Unfriendly open source projects

While working on the RaspberryPi setup, which I decided is going to use wview as the main component, I’ve started wondering once again why some upstream projects really feel like they don’t want to be friendly with distributions and other developers at all.

It’s not like they are hostile, just really unfriendly.

First of all, wview comes with its own utility library (more on that in a different post); both projects don’t have a public repository, they are developed behind closed doors, but you can send patches. This is bothersome but not a big deal. But when you send patches, and it goes weeks without answers, and without any idea if the patch was merged, well, things get bothersome.

But this is not the only project lately that seems to have this kind of problems. Google’s PageSpeed has binary package snapshots for Debian and CentOS but they have no related source release. Indeed, if you want the sources, you’re supposed to fetch them from the repository using Google’s own repo tools, rather than any standard SCM software. While I was interested in trying the module, the idea of having to deal with repo tools once again didn’t enthusiasm me, so I stopped looking into it at all.

This, to me, looks like a common problem: who cares about the policies that distribution had for years, make it agile, make it easy, and stop doing releases, git is good for everybody and so is bzr, hg and monotone! And then the poor distribution packages need to suffer.

Who cares about pkg-config and the fact that it supports cross-compilation out of the box? Just rely on custom -config scripts!

Why waste time making things work once installed at system level? They work good enough out of a repository!

Sigh.

Cross-compilation and pkg-config

As it happens – and as I noted yesterday – one of my current gigs involves cross-compilation of Gentoo packages; this should also be obvious for those monitoring my commits. This sometimes involves simply fixing the ebuilds; and other times it needs to move up the stream and fix the buildsystem of the original project. Lately, I hit a number of the latter cases.

In the previous post, I noted that I fixed upower’s build system to not choose the backend based on the presence of files in the build’s machine. Somehow, checking for presence of files in the system to choose whether to enable stuff, or to install stuff, seems to be the thing to do for build system; a similar situation happened with SystemTap, that tries to identify the prefix of NSS and Avahi headers by checking for the /usr/include/nss and /usr/include/avahi-common directories, among others. Or, I should have said, tried, since luckily my two patches to solve those issues have also been merged today.

Another package I had to fix for cross-compilation recently was ecryptfs-utils, and again NSS is involved. This time rather than testing the presence of files, the build system did almost the right thing, and used the nss-config script that ships with the package to identify which flags and libraries to use. While it is a step in the right direction, compared to SystemTap’s, it is not yet the real right thing to do.

What is the problem? When you’re cross-compiling, simply calling nss-config will not do: the build machine’s script would be used, which will report the library paths of that system and not your target, which is what you really want. How do you solve this trouble then? Simple, you use the freedesktop-developed pkg-config utility, that can be easily rigged to handle cross-compilation nicely, even though it is not as immediate as you’d like, and if you ever tried using it in a cross-compilation environment, you probably remember the problems.

It is for this reason that I started writing cross-compile documentation for pkg-config in my Autotools Mythbuster quasi-book. Speaking about which, I’d like to find some time to spend focusing on writing more sections of it. To do so, though, my only chance is likely going to be if I get to take some vacation, and book a hotel room while I work on it: the former part is needed as right now my gigs tend to occupy my whole day leaving me with scarce to no free time, and the latter because at home I have too many things to take care of to actually focus on writing when I’m not working.

At any rate, please people, make sure you support pkg-config rather than rely on custom scripts that are not really cross-compilable.. it is the best thing you can do, trust me on this please.

Fixing CFLAGS/LDFLAGS handling with a single boilerplate Makefile (maybe an eclass, too?)

So, in the last few weeks I’ve been filing bugs for packages that don’t respect CFLAGS (or CXXFLAGS) using the beacon trick. Beside causing some possibly false positives, the testing is going well.

The problem is that I found more than a couple of packages that either do call gcc manually (I admit I’m the author of a couple of ebuilds doing that) or where the patch to fix the Makefile would be more complex than just using a boilerplate makefile.

So what is the boilerplate makefile I talk about? Something like this:

$(TARGET): $(OBJS)
        $(CC) $(LDFLAGS) -o $@ $^ $(LIBS)

Does it work? Yes it does, and it will respect CFLAGS, CXXFLAGS and LDFLAGS just fine, the invocation on an ebuild (taking one I modified earlier today) would be as easy as:

src_compile() {
    emake CC="$(tc-getCC)" 
        TARGET="xsimpsons" 
        OBJS="xsimpsons.o toon.o" 
        LIBS="-lX11 -lXext -lXpm" || die "emake failed"
}

Now of course this would suck if you had to do it for every and each ebuild, but what if we were to simplify it to an eclass? Something like having an ebuild just invoking it this way:

ESIMPLE_TARGET="xsimpsons"
ESIMPLE_OBJS="xsimpsons.o toon.o"
ESIMPLE_LIBS="-lX11 -lXext -lXpm"

inherit esimple

For slightly more complicated things you could make it use PKG_CONFIG too…

ESIMPLE_TARGET="xsimpsons"
ESIMPLE_OBJS="xsimpsons.o toon.o"
ESIMPLE_REQUIRED="x11 xext xpm"

inherit esimple

so that it would call pkg-config for those rather than using the libraries directly (this would allow to simplify also picoxine’s ebuild for instance that uses xine-lib).

Even better (or maybe I’m getting over the top here ;)), one could make the eclass accept a possibly static USE flag that would call pkg-config --static instead of standard pkg-config and append -static to the LDFLAGS, so that the resulting binary would be, well, static…

If anybody has comments about this, to flesh it out before it could actually be proposed for an eclass, it would be a nice time to say here so we can start with the right foot!

Building against non-installed libraries

If you have seen my repositories lately, you’d probably have noticed that I started working more on third-party projects, in particular you can see repositories for libxdg-basedir (which I’ve been using already for a while for xine-lib) and libxr (which is new, recently added to portage by Luca), together with those you can find a bunch of repositories from lscube and some from Lennart .

The reason for this is that I started considering taking a more active stance when I need something fixed, in particular, I’m just making my changes and then sending upstream the repositories, hoping they get merged quickly. This is the nice thing about having repositories available with GIT and all the like. Unfortunately a lot of projects are still managed with Subversion or, even worse, CVS, or other DVCS which I don’t have set up on midas (yet).

At any rate, one of the changes that is tied with all these project is probably the introduction of the -uninstalled variant of pkg-config files. Simon McVittie writes more about it than I could probably write myself.

There are a few interesting notes about this though. The first is that -uninstalled variants of pkg-config files gets often in the way of libtool; for instance you’d have to point them to the .libs/ directory used internally by libtool, or you’d be able to link against them just with libtool (as they’d point to the .la file). The other problem is that, if you’re building with libtool, it will properly set the rpath/runpath to make sure that the resulting binary is executed against the uninstalled version, but this does not happen, by default, outside of libtool. I’m still pondering whether I should change the -uninstalled.pc files to set the runpath by themselves. Unfortunately it would then work only with ld compatible with GNU syntax.

The nice part of building against non-installed libraries is that it makes it much easier to run a series of tests on the project, which is very nice once you set up unit tests .

Now if only I can solve the unit testing problem, I could be focusing on working on unit testing and possibly to extend unit testing coverage in as many software as possible, which I use, so that I don’t have to worry about broken behaviour. But this is work for another day, right now I have enough to do, I should be writing docs and articles, and converting to Glib and writing tests.

And I just came out of the hospital! I hope the games I ordered from Amazon will arrive soon so I can at least relax a bit during the weekends.

And what about imported libraries?

Following the previous blog here also a list of projects that seem to like importing libraries, causing code duplication even for code that was designed to be shared.

  • cdrkit, again, contains a stripped down version of libdvdread, added, of course, by our beloved Jörg Schilling; bug #206939; additionally it contains a copy of cdparanoia code; bug #207029

  • ImageMagick comes with a copy of libltdl; bug #206937

  • not even KDE4 seems to have helped libkcal which even in its newest incarnation ships with an internal copy of libical, causing me to have three copies of it installed in my system;

  • libvncserver comes with a copy of liblzo2; actually there are two, one in libvncserver and one in libvncclient; even the source files are duplicated!; bug #206941

  • SDL_sound, Wine and LAME seem to share some mp3 decoding code, which seems to come originally from mpg123;

  • cmake couldn’t stay out of this, it comes with a copy of libform (which is part of ncurses); follow bug #206920

  • I’m not sure what it is, but DigiKam, Numeric (for Python) and numpy have a few functions in common; the latter seems to have even more than that in common; bug #206931 per Numeric and numpy, and bug #206934 for DigiKam.

  • ghostscript comes with internal copies of zlib, libpng, jpeg and jasper; unfortunately jasper is also modified, for the other three there’s bug #206893; by the way, the copies are present in both the gs command and in the libgs library;

  • OpenOffice comes with loads of duplicated libraries; in particular, it comes with its own copy of icu libraries; see on bug #206889

  • TiMidity++ comes with a copy of libmikmod; bug #206943

  • Korundum for KDE3 has a copy of qtruby embedded, somehow; I wonder if it isn’t a fluke of our buildsystem; bug #206936

  • gdb contains an internal copy of readline; –bug #206947

  • tork contains a copy of some functions coming from readline; bug #206953

  • KTorrent contains a copy of GeoIP (and to think I removed the one in TorK as soon as I’ve spotted it); bug 206957

  • both ruby and php use an internal copy of – I think – oniguruma; I haven’t looked if it’s possible to add that as a system library and then use it; bug #206963

  • MPlayer seems to carry a copy of libogg together with tremor support; bug #206965

  • pkg-config ships with an internal copy of glib; bug #206966

  • tor has an internal copy of libevent’s async dns support; funny, as it links to libevent; bug #206969

  • gettext fails to find the system copy of libxml2, falling back to use the internal copy; at least it has the decency of using a proper commodity library; bug #207018

  • both Perl and Ruby have a default extension based on SDBM, a NDBM workalike; there seems not to be a shared version of it, so they just build the single source file in their own extensions directly, without hiding the symbols; beside the code re-use not being available, if a process loads both libperl and libruby, and in turn they load their sdbm extension, stuff’s gonna hurt;

  • enchant has an internal copy of Hunspell; probably due to the fact that old Hunspell built only static non-PIC libraries, and enchant uses plugins; bug #207025; upstream fixed this in their Subversion repository already;

  • gnome-vfs contains an internal copy of neon; funny as it depends on neon already, in the ebuild; bug #207031

  • KOffice’s Karbon contains an internal copy of gdk-pixbuf; bug #209561;

  • kdegraphics’s KViewShell contains an internal copy of djvulibre; bug #209565;

  • doxygen contains internal copies of zlib and libpng; bug #210237 ; this time I used a different method to identify it as doxygen does not export the symbols;

  • rsync contains an internal copy of zlib; bug #210244 ;

Unfortunately making sure that what I’m reading is true data and not false positive, looking at the output of my script, becomes more difficult now for the presence of multiple Sun JDK versions; I have to add support for alternatives, so that different libraries implementing the same interface don’t show up as colliding (they are that way by design).