Always factor Murphy in

Sometimes, Murphy make you curse the wrong software, or the wrong people, or both.

With kernel 2.6.29, the iec958 output from my soundcard (ICE1712-based) decided to stop working properly and started getting white noise out after a while. I never had enough time to properly debug it so I left it alone. Now 2.6.30 landed, and magically the output seemed to work fine. To a point; after a while playing music I could start hearing screeches, both with PulseAudio and with direct output on the soundcard.

Changing resampler settings, format setting, all the possible setting, nothing worked. I started cursing ALSA, the soundcard producer, possible bugs along the way…. then I tried one thing out of desperation:

% speaker-test -c 2 -t sine

The default for speaker tests, both from ALSA and the internal one for self-diagnostic of my A/V receiver (which is what gets the final output) use by default pink noise. The result is that you cannot identify a distortion along the way, at least not easily and not if you’re not a technician to begin with. The sine method instead uses a sinusoidal soundwave. A single-tone, continuous beep. Much nicer to find the failure point.

Indeed, this let me find a nasty screeching sound on the left channel speaker. From that, it was just a metter of getting dirty (literally) with trying the speaker on a different channel, with a different cable, and the cable with a different speaker to make sure that the problem was, indeed, the cable itself. Luckily, because if it was the speaker it wouldn’t have been as cheap, and also because I do know Murphy well enough to have cables at home, so I change the cable, and the noise disappears.

For a while at least because now it resumed again. I’m not sure what the problem is at this point, I’ll doublecheck with an analog input in case the problem is actually server side, but I’m afraid the problem is in the receiver itself, maybe the temperature got it to fry up something. Whatever the problem, it’s a nasty one.

So now I’m again without audio on Yamato (which makes maintaining PulseAudio a bit of a problem to be honest), and relying on Merrimac to play music, not the best choice I have to say.

Suggestions for a new sound system, stereo, good stereo, no (strict) need for surround or subwoofer, digital interface with computer required, are very welcome.

Update: after some more fighting with cables, receiver, ground, speakers, computers, it turns out the problem isn’t, after all, the receiver: plugged in the iMac with the optical interface it’s fine, and even the coaxial S/PDIF interface, together with the cable connected to it, are fine once connected to Enterprise (booting Fedora 11 live) and the via82xx integrated soundcard.

So the problem is either in the ICE1712 soundcard or the related driver. I don’t want to track down the issue further at this time (my time is all allotted up already) so I’ve added an USB soundcard to my wishlist and will wait a bit, using the iMac with the receiver. Further details at the end of this other post if you care!

Back to origins

Thanks to the luck with QEmu this week I finally got a Gentoo/FreeBSD VM working again, so I can actually resume working on the one thing I joined Gentoo for, initially. The nice thing about this is that the project is, in itself, mostly an experimentation, which means it’s quite easygoing. But it also has some very interesting and useful results.

Every time people ask me why do I think Gentoo/FreeBSD is useful to something, I point out that by not using ports, we’re ensuring that the software builds out of the original sources, and if it doesn’t, we can provide the patches upstream, since we have to write them in a way that is compatible with other systems anyway. This hasn’t changed the slightest in the last two years I didn’t work on the project: ports maintainers still don’t seem to provide upstream with patches, and lots of software is quite broken.

Indeed, in the last couple of days I identified quite a few issues both in and outside of Gentoo: bsdtar from libarchive failed to work with latest Portage version (thanks to Tim who provided me a patch within the hour!), pambase was putting nologin in the wrong chain (fixed and pushed a new release out), sandbox does not compile (still broken, need to be investigated yet), PulseAudio was totally borked upstream (now I made it build but it still fails tests, need to fix and port some areas, and if I had the time there is also the OSS driver to fix), libSM has a dependency over libuuid (which collides in FreeBSD, where the system already provides a different, incompatible interface; I submitted a patch to use the FreeBSD uuid interface when available), and more.

I cannot blame the Gentoo/FreeBSD team for this, because, well, it’s just Alexis right now I guess; I’m getting my hands dirty and making sure I can get the thing to work as it’s supposed to, and this is the important part, I guess. On the other hand, I wonder why is it that FreeBSD developers don’t seem to care about this kind of problems at all. PulseAudio might not have the best OSS support, but that’s just because Lennart obviously don’t care about it (Fedora now also disabled it by default, good for them!), but if somebody were to actually mind PulseAudio (more than I can do, since I don’t have audio in my VM anyway), I don’t think it would be impossible for it to provide proper support for the FreeBSD OSS options.

At any rate, I guess I’m now back to my original plans as well, at least part time, hopefully it won’t be too bad on the long run. Going to try GCC 4.4 with the system packages, and the kernel, later on today. Or rather I’ll leave it to test the build since I’m actually supposed to be out of here to a friend’s house for some photo shoots (long story…).

Oh by the way, if you haven’t noticed I’m still making some changes to the blog, in particular now the tags and categories pages show decent titles; I have made some changes to Typo that allows me to set the titles in a more human-readable way. If I can find time I’ll be also cleaning up the tags, since I have lots of tags with one post each, and there are some synonyms that I should really get rid of. To do the latter, though, I’m going to write a script that can merge the tags’ contents and then set up redirection, since I dislike very much to break the links in my blog, as you may know already.

Oh well!

The Mono problem

In the past I have said that I find C# a very nice language; I still maintain this position, the C# language, taken alone, is probably one of my favourites, this does not mean that I intend to rewrite my ruby-elf suite in C#, but I like it and if I were to have to write a GUI application for the GNOME desktop, not relying on any particular library, I’d probably choose C# and Mono.

Why do I say “not relying on any particular library”? Well, today I wanted to try writing some C# bindings for PulseAudio to make use of it for a project I was proposed (and I don’t think now is really feasible), and I went to read the documentation from the Mono project. Took me a while to digest it; even though I had some experience with writing bindings for Ruby with my “Rust”, and a long time ago I worked on implementing Python extensions to a program, the C# method of bindings software really escaped me for quite a while.

In all the interpreted languages I know to write bindings you start from the language the interpreter is written in (usually, C) and then define an interface to the language that calls into “native” functions that in turn can call the library you want to bind. This is how Ruby bindings are written, and how Rust works, it tells the Ruby interpreter that there is a class called in a certain way and it defines its method through C functions that are called back; then it takes care of marshalling and translating parameters around.

The C++ bindings work in a slightly different way: since C++ can be described as a superset of C from one point of view, and the design of the language allows a very high type compatibility between the two, included the calling conventions, you usually write C++ class interfaces that wrap around C functions calling. It’s a completely different paradigm than Ruby and Python, but it works because there is really no interpreter or library barrier between C and C++ after all.

Considering how Mono is implemented, I sincerely expected the bindings writing to be a mix of these two methods; it seems instead that it’s almost entirely a C+­+-style bindings interface. But with a nasty twist: in C+­+ you got direct access to the interface of the library you’re writing bindings of (its headers) through direct inclusion and an eventual extern "C" block; with C# you don’t have that at all, as far as I can see.

This means that you got to describe all the interfaces inside the C# code, and then write the marshalling code that can translate the parameters from C# objects to C types. The way the functions are loaded is similar to the standard dynamic linker interface (dlopen()) with all the problems connected to that: C++ interfaces are almost impossible to load, and you got to get the parameters just right, if you don’t it’s a catastrophe doomed to happen. And this is even trickier than linking libraries with pure dynamic linking.

The most obvious problem for those who had to deal with dlopen() idiosyncrasies, is that C# has fixed-sized basic integer types. This is good, but not all software uses fixed-sized parameters; off_t, size_t, long are all types that change their size depending on the operating system and the hardware architecture; off_t is not even of the same size on the same system because it depends on whether large file support is enabled or not, at least on glibc-based systems (most BSDs should have it fixed-sized but that’s about it). Since the C# code is generic and is supposed to be built just once, it’s not easy to identify the right interface for the system. You cannot just #ifdef it around like you would with C++ bindings.

But this is not the only problem; the one problem I noticed first is, again because of the lack of access to the #include headers, that constants might not be constant. Since I wanted to write bindings for PulseAudio, I started first with the “simple” interface, and I started finding the problem right away with the pa_stream_direction_t enumeration. While I could create my own C# enumeration for it, I have no guarantee that Lennart does not decide to change the values; while that is going to change the ABI of the package, for both native implementations and Ruby-style bindings, a rebuild is just enough, there is usually no need to change the sources when this kind of changes are made; for the C# bindings, you’d have to adapt the bindings every time.

This is probably why there aren’t many C# bindings for libraries that don’t use GObject (for that, you got the gapi tool that takes care of most of the work for you), and why the banshee project preferred to reimplement TagLib in C# rather than bind it (indeed, binding TagLib is far from an easy thing, like I can testify first hand). I’m afraid this is the “Achilles’ heel” of C# and Mono. While this makes it less likely to produce the “java crap effect” that I have written about almost four years ago by now (jeez has so much time passed?), it does reduce the ability of Mono to blend in with the rest of the modern Linux systems.

The effort required to maintain proper bindings for C projects in C# is even higher than it is to reimplement the same code, and that is really a big problem for blending; the only thing that it works well for is portability, especially when it comes to portability on Microsoft platforms. This is all fine and dandy if you need your software to bend that way, and I have to say I do know a couple of cases where that might be one of the important factors, but it comes to a pretty high toll. On the other hand, Ruby, Python, Vala and Java seem to have better chances for integration. All in all, I’m sincerely unimpressed. I like the language, I just don’t like the runtime or the way that’s going.

Coming soon, in a sync tree near you

I’ve been meaning to write about the 32-bit emulation libraries for a while, with all the problems they come with, included a lot of security problems. The chance come now since last night solar pushed (not yet to the tree though) the new test version of the new emul-linux ebuilds.

The first problem to know of is that at the moment the whole of the set of emulation libraries is a collection of security bugs; I cannot even start to count how many issues there are but for sure there are two for PAM, the two that caused the recent stabilisation of 1.0.4 version. So if you need a system where you count security first, you should not use the multilib profile for AMD64 and the emul-linux libraries. You can still use it without emul-linux for stuff like grub, but that should be it; anything bringing in emul-linux will bring in security-impaired software.

On the other hand, thanks to Daniel’s work on 32-bit userland, it’s likely that building these libraries will start being less of a problem. Some of the fixes are already in, others will go in in time; I fixed Perl and Qt4 last night, the next step is to add a sub-profile for building those libraries.

There is more work that needs to be done though, for instance PulseAudio needs to have a way to disable the whole daemon part and just build the library used by software, which is what you want on both the 32-bit compatibility library list and for daemon-less clients (for instance when you want to just use a remote host to send audio to). These will require some changes in PulseAudio itself and will have, thus, to wait for the new release and probably the one afterwards. I could start working on it already (I would probably be able to get it to work before end of the day) but since I cannot even use it yet it doesn’t make much sense. The problem is that the latest test versions fail nastily on my system, and I was unable to ask Lennart about them, since he’s currently at BOSSA.

Hopefully, one day, the emul-linux libraries wouldn’t be needed at all, and instead we’d be building the packages, in either 32-bit or 64-bit mode, directly as needed. The true multilib support is, currently, one of the most important lacking features of Gentoo; on the other hand, it seems like that what gets discussed for implementation is stuff that has relatively little importance to users, and a lot of importance on “who is right” about implementation details.

Frankly, I really wish we were finally discussing a way to express the difference between same-ABI and any-ABI dependencies between packages, it would be really quite useful especially if the concept of ABI was expanded further to encompass also implementations like Ruby 1.8, Ruby 1.9, JRuby and so on so forth, so that we all could decide whether to build dev-ruby/ruby2ruby for one, the other, or all three together with its dependencies; or whether to install PulseAudio as 64-bit only or both 64-bit and 32-bit.

Discovering the environment versus knowledge repository

For what concerns users, the main problem with build systems based on autotools is the long delay caused by executing ./configure scripts. It is, indeed, a bit of a problem from time to time and I already expressed my resentment regarding superfluous checks. One of the other proposed solution to mitigate the problem in Gentoo is running the script through a faster, more basic shell rather than the heavy and slow bash, but this has a few other side issues that I’m not going to discuss today.

As a “solution” (but to me, a workaround) to this problem, a lot of build systems prefer using a “knowledge repository” to know how to deal with various compiler and linker flags, and various operating systems. While this certainly has better result for smaller, standards-based software (as I write in my other post, one quite easy way out from a long series of tests is just to require C99), it does not work quite that well for larger software, and tends to create fairly problematic situations.

One example of such a buildsystem is qmake as used by Trolltech, sorry Qt Software. I had to fight with it quite a bit in the past when I was working on Gentoo/FreeBSD since the spec files used under FreeBSD assumed that the whole environment was what ports provided, and of course Gentoo/FreeBSD had a different environment. But without going full-blown toward build systems entirely based on knowledge, there can easily be similar problems with autotools-based software, as well as cmake-based software. Sometimes it’s just a matter of not knowing well enough what the future will look like, sometimes these repositories are simply broken. Sometimes, the code is simply wrong.

Let’s take for instance the problem I had to fix today (in a hurry) on PulseAudio: a patch to make PulseAudio work under Solaris went to look for the build-time linker (ld) and if it was the GNU version it used the -version-script option that it provides. If you look at it on paper, it’s correct, but it didn’t work and messed up the test4 release a lot. In this case the cause of the problem is that the macro used has been obsoleted and thus it never found the link as being GNU, but this was, nonetheless, a bad way to deal with the problem.

Instead of knowing that the GNU ld supported that option, and just that, the solution I implemented (which works) is to check if the linker accepts the flag we needed, and if it does it provides a variable that can be used to deal with it. This is actually quite useful since as soon as I make Yamato take a break from the tinderbox I can get the thing to work with the Sun linker too. But it’s not just that: nobody tells me whether in the future a new linker will support the same options as the GNU ld (who knows, maybe gold).

A similar issue applies to Intel’s ICC compiler, that goes to the point as passing itself as GCC (defining the same internal preprocessor macros), for the software to use the GCC extension that ICC implements. If everybody used discovery instead of knowledge repository, this would have not been needed (and you wouldn’t have to workaround when ICC does not provide the same features as GCC — I had to do that for FFmpeg some time ago).

Sure, knowledge repository is faster, but is it good just the same? I don’t think so.

User Services

A little context for those reading me; I’m writing this post a Friday night when I was planning to meet with friends (why I didn’t is a long story and not important now). After I accepted that I wouldn’t have a friendly night I decided to finish a job-related task, but unfortunately I’ve had some issues with my system. Somehow the latest radeon driver is unstable (well it’s an experimental driver after all), and it messes up compiz; in turn after a while either X crashes or I’m forced to restart it. This wouldn’t be a problem if the emacs daemon worked as expected. Since it doesn’t, I lose my workspace, with the open files, and everything related to that. It’s obnoxious. Since this happened four times already today I decided to take the night off, but I wasn’t in the mood for playing, so I settled for watching Pirates of the Carribean 2 in Blu-Ray, and write out some notes regarding the topics I wanted to write about for quite a while.

The choice of topic was related to the actual context I’ve just written above. As I said GNU Emacs is acting badly when it comes to the daemon. While the idea of the daemon would be to share buffers (open files) between ttys, network and graphical session, and to actually allow restarting those sessions without losing your settings, your data, and your open files, it’s pretty badly implemented.

A few months ago I reported that as soon as X was killed by anything (or even closed properly), the whole emacs daemon went down. After some debugging it turned out to be a problem with the handling of message logging. When the clients closed they sent a message to be logged by the emacs daemon, but since it had no way to actually write it to a TTY session, it died. That problem have been solved.

Now the problem appear to be just the same mirrored around: after X dies, the emacs daemon process is still running, but as soon as I open a new client, it dies. I guess it’s still trying to logging. As of today the problem still happens with the CVS version.

So anyway, this reminded me of a problem I already wanted to discuss with a blog: user-tied services. Classically, you had user-level software that i s started by an user and services that are started by the init system when the system starts up. With time, software became less straightforward. We have hotplugged services, that start up when you connect hardware like, for instance, a bluetooth dongle, and we have session software that is started when you login and is stopped once you exit.

Now, most of the session-related software is started when you log into X, and is stopped when you exit, sometimes, though, you want processes to persist between sessions. This is the case of emacs, but also my use case for PulseAudio since I want for it to keep going from before I login to before I shut down the system straight. There are more cases of similar issues but let’s start with this for now.

So how do we handle these issues? Well for PulseAudio we have an init script for the systemwide daemon. It works, but it’s not the suggested method to handle PulseAudio (on the other hand is probably the only way to have a multi-user setup with more than one user able to play sound, but that’s for another day too). For emacs, we have a a multiplexed init script that provides one service per user; a similar method is available for other service. Indeed, in my list of things to work on regarding PulseAudio there is to add a similar multiplexed init script to run per-user sessions of PulseAudio without using the system wide instance (should solve a bit of problems).

So the issue should be solved with he multiplexed per-user init script, no? Unfortunately, no. To be able to add the init scripts to the runlevels to be started, you need to have the root privileges. To start, stop and restart the services, you also need root privileges. While you can use sudo to allow users to run the start/stop commands to the init script, this is far from being the proper solution.

What I’d like to have one day is a way to have user-linked services, in three type of runlevels: always running (start when the machine starts up, stop when the system shuts down), after the first login (the services are started at the first login and never stop till shutdown), and while logged in (the services start at the first login, and stop when the last session logs out).

At that point it would be then possible to provide init scripts capable of per-user multiplexing for stuff like mpd too, so that users could actually have the flexibility f choosing how to run any software in the tree.

Unfortunately I don’t have any idea on how to implement this right now, but I guess I could just throw this in for the Summer of Code ideas.

Why would anybody need PulseAudio?

That’s a very common question as of lately, and somehow I feel like most people who haven’t dealt with ALSA in the past would find it very difficult to properly answer to it. Even myself I would have ignored one particular issue till last night, when I hit another reason why I want to keep PulseAudio as my main and only audio system as soon as possible, reducing direct ALSA access.

With modern systems you mostly get onboard sound cards, rather than PCI cards and similar, unless you’re somewhat crazy and want a proaudio card (like I did) or a Creative card (not yet sure what’s the fuzz about that). To be honest, my motherboard really does have a limited soundcard so I would have needed a PCI card anyway to get digital S/PDIF output, and I want S/PDIF because the room has too much electric noise to the point I can hear it on the speakers.

Somehow, ALSA seem to hide to most users that there are indeed a lot of limitations with the HDA hardware. This because the idea for the HDA was probably to shift as much work as possible into software space rather than doing so in hardware; this is why you need a driver fix for the headphones jack to work properly most of the time. While this can be seen as a way to produce cheap cards, it has to be said that software always had to compensate for hardware defects since it’s more flexible. And having most of the processing done in software means that you can actually fix it if there is a bug, rather than having to find a workaround if anything.

So, no hardware mixing, and ALSA has to use dmix, no hardware volume handling, and ALSA has to use softvol, no hardware resampling, and ALSA has its own resamplers, and so on so forth. But while ALSA has to cope with doing these things in the same process that are doing the audio output, and thus is not so good at coping with different processes accessing the audio device with different parameters.

Having a separate process doing the elaboration does not mean that you add more work to the system, you can actually make it do less work if, for instance, you ask that process to play a (cached) sound rather than having it opened, converted, and played back.

At the same time, the fact that PulseAudio handles mixing, volume and resampling in software does not mean that it adds to the work done by the software for most cards, since the most common cards nowadays, the HDA-based ones, already do all that in software, just you don’t see that explicitly because ALSA do them in process.

In my opinion, ALSA is really doing way too much as a library, since each time you open a device it has to parse a number of configuration files, to identify which definition to use (and even then, by default they are far from perfect, for instance for the ICE1712-based cards, which depending on the way they are wired may have two, four, six or eight channel outputs, the definition only suits the 8-channel model, and does not make sense with the lower end cards). And once it found the definition it has to initialise a number of plugins, internal or external, to perform the software functions.

And this is all without putting in the mix the LISP interpreter that alsa-lib ships with (I’d leave to Lennart to explain that one).

So if you think that PulseAudio is an over-engineered piece of software that performs functions in software that should be done in hardware, you better be a FreeBSD user. At least there the OSS subsystem does not try to do all the things that ALSA does (and on the other hand FreeBSD is trying to go the Microsoft way for the HDA cards, implementing the UAA specifications rather than having a huge table of quirks; as far as I can see that would be quite useful, if it’s going to work that is).

But even in that case you should probably find PulseAudio a good thing since it tries not to be Linux-specific and thus would allow for performing all the needed functions with a single cross-platform software without having to reimplement it in platform-specific systems like OSS or ALSA. That was my main reason to look into PulseAudio when I was working on Gentoo/FreeBSD.

For everyone who cares about users, let’s try to work all together to make PulseAudio better rather than attacking Lennart for trying to do the right thing, okay?

The battles and the losses

In the past years I picked up more than a couple of “battles” to improve Free Software quality all over. Some of these were controversial, like `–as-needed and some of them have been just lost causes (like trying to get rid of C++ strict requirements on server systems). All of those though, were fought with the hope of improving the situation all over, and sometimes the few accomplishments were quite a satisfaction by themselves.

I always thought that my battle for --as-needed support was going to be controversial because it does make a lot of software require fixes, but strangely enough, this has been reduced a lot. Most of the newly released software works out of the box with --as-needed, although there are some interesting exceptions, like GhostScript and libvirt. On the positive exceptions, there is for instance Luis R. Rodriguez, who made a new release of crda just to apply an --as-needed fix with a failure that was introduced in the previous release. It’s very refreshing to see that nowadays maintainers of core packages like these are concerned with these issues. I’m sure that when I’ve started working on --as-needed nobody would have made a new point release just to address such an issue.

This makes it much more likely for me to work on adding the warning to the new --as-needed and even more needed for me to find why ld fails to link PulseAudio libraries even though I’d have expected him to.

Another class of changes that I’ve been working on that have shown more interest around than I would have expected is my work on cowstats which, for the sake of self-interest, formed most of the changes in the ALSA 1.0.19 release for what concerns the userland part of the packages (see my previous post on the matter).

On this case, I wish first to thank _notadev_ for sending me Linkers and Loaders, that is going to help me improve Ruby-Elf more and more; thanks! And since I’m speaking of Ruby-Elf, I finally decided its fate: it’ll stay. My reasoning is that first of all I was finally able to get it to work with both Ruby 1.8 and 1.9 adding a single thin wrapper (that is going to be moved to Ruby Bombe once I actually finish that), and most importantly, the code is there, I don’t want to start from scratch, there is no point in that, and I think that both Ruby 1.9 and JRuby can improve from each other (the first losing the Global Interpreter Lock and the other one trying to speed up its starting time). And I could even decide to find time to write a C-based extension, as part of Ruby-Bombe, that takes care of byteswapping memory, maybe even using OpenMP.

Also, Ruby-Elf have been serving its time a lot with the collision detection script which is hard to move to something different since it really is a thin wrapper around PostgreSQL queries, and I don’t really like to deal with SQL in C. Speaking about the collision detection script, I stand by my conclusion that software sucks (but proprietary software stinks too).

Unfortunately while there are good signs to the issue of bundled libraries, like Lennart’s concerns with the internal copies of libltdl in both PulseAudio (now fixed) and libcanberra (also staged for removal) the whole issue is not solved yet, there are still packages in the tree with a huge amount of bundled libraries, like Avidemux and Ardour, and more scream to enter (and thankfully they don’t always do). -If you’d like to see the current list of collisions, I’ve uploaded the LZMA-compressed output of my script.- If you want you can clone Ruby-Elf and send me patches to extend the suppression files, to remove further noise from the file.

At any rate I’m going to continue my tinderboxing efforts, while waiting for the new disks, and work on my log analyser again. The problem with that is I really am slow at writing Python code, so I guess it would be much easier if I were to reimplement the few extra functions that I’m using out of Portage’s interface in Ruby and use those, or find a way to interface with Portage’s Python interface from Ruby. This is probably a good enough reason for me to stick with Ruby, sure Python can be faster, sure I can get better multithreading with C and Vala, but it takes me much less time to write these things with Ruby than it would take me in any of the other languages. I guess it’s a problem with the mindset.

And on the other hand, if I have problems with Ruby I should probably just find time to improve the implementation; JRuby is enough evidence to show that my beef against Ruby 1.9 runtime not supporting multithreading are an implementation issue and not a language issue.

A softer –as-needed

Following my previous blog about un-released autoconf I wanted to write a bit about an unreleased change in binutils’ ld, that Sébastien pointed me at a few days ago. Unfortunately, since things piled up, the code is now actually released, and I briefly commented about it in the as needed by default bug. The change is only in the un-keyworded snapshot of pre-2.20 binutils so it’s not released to users, which makes it worth commenting before hand anyway.

The change is as follows:

--as-needed now links in a dynamic library if it satisfies undefined symbols in regular objects, or in other dynamic libraries. In the latter case the library is not linked if it is found in a DT_NEEDED entry of one of the libraries already linked.

If you know how --as-needed works and the ELF-related terms, you should be able already to guess what it’s actually doing. If you’re not in the known with this, you should probably read again my old post about it. Basically the final result of this is that the first situation:

Messy linking diagram

gets expanded in the wished linking situation:

Hoped linking situation

instead of the broken one that wouldn’t work.

This is all good, you’d expect, no? I have some reserves about it. First of all, the reason for this change is to accommodate the needs of virtual implementation libraries like blas and similar. In particular the thread refers to the requirements of gsl to not link its blas implementation leaving it to the user linking the final application. While I agree that’s a desired feature, it has to be noted that all the libraries needs to keep the same ABI, otherwise just changing it on the linker call is not going to work. Which means that you can technically change the implementation by using the LD_PRELOAD environment variable to interpose the new symbols at runtime, allowing to change the possible implementation at runtime without having to relink anything.

Of course, using LD_PRELOAD is not very handy especially if you want to do it on a per-command basis or anything like that. But one could probably wonder “Why on Earth didn’t someone think of a better method for it before?” and then answer to himself after a bit of search “Someone already did!”. Indeed a very similar situation arouse on FreeBSD 5 series since there were multiple PThread implementations available. Since the ABI of the implementations is the same, they can be switched at both link editing time and at runtime linking. And to make it easier to switch it at runtime, they created a way to configure it through the /etc/libmap.conf file.

Indeed, the method to choose different implementations of PThread under FreeBSD used before libmap.conf introduction was the same that gsl wants to use. The result of which already shown that --as-needed was unusable on FreeBSD because of a similar problem: libpthread was never added to the dependencies of any library and was supposed to be linked on the final executable, that might not have any requirement for PThread by itself.

So basically the whole reasoning for softening up --as-needed is to allow working around a missing feature in the GNU runtime linker. Which is what, to me, makes it wrong. Not wrong in the sense of the wrong thing to do, but the wrong reason to do it. But it’s not that simple. Indeed this change means that there will be much less build failures with --as-needed, making it much much more likely to become part of the default options of the compiler once binutils 2.20 is released. On the other hand, I think I’ll submit a patch for ld to warn when the new code is triggered.

My reasoning is quite simple: libraries should be, as much as possible, be linked completely so that all their dependencies are well stated, especially since leaving them to the final executable to link can create a huge mess (think if the final executable is linked on a system where a different, ABI-incompatible, dependency is present, the final executable will have subtle problems running, like unexpected crashes and the like), and also, if ten executables need a single library, which forgets to state its dependency on, just as an example, libexpat, you get ten links to libexpat that needs to be re-created (while the original library will not be picked up at all by the way, so will still expect the ABI of the previous version), rather than just one.

Since indeed the softer --as-needed makes it much simpler to enable it by default, I think it’s not a good idea to revert from the new behaviour, but having a warning that would say something like “-lexpat salvaged for -lfoo” would make it easy to identify the issue and assess on a case by case basis whether this is an intended situation or just a bug. So that the latter can be corrected.

On the other hand I also have a case of failure with recursive linking, coming out of the next PulseAudio release, which I need to get fixed, hopefully before PulseAudio is released.

Prank calls and threats

Disclaimer: please take this post with a grain of salt and consider it tongue-in-cheek. While the situation from which it sprouts is very real, the rest is written with a jokingly tone. In particular I wish to state beforehand that I’m not trying to tie anything to Ciaran and his group, or any other group for what matters.

As it turns out, last night some very funny guy thought that it was a nice idea to call me and tell me I’ll die, obviously with the caller ID withheld. It’s almost certainly a prank call (as a good rationalist, it’s 99.98% a prank call, 0.02% you never know about), but with the cold and the meds in me, I didn’t have the quickness of response to say “you go first and don’t spoilt it to me how it is”.

Just to cover all basis, I’m now considering who might actually want me dead. Which turns out that, if we consider Hans Reiser’s extreme personality cases, might be quite a bit of people. I wouldn’t count Ciaran in the list though, since a) I respect him enough to trust he wouldn’t do it anonymously if he wanted to and b) Stephen is more the person to slander rather than threaten. Beside, that area has been quiet for quite a bit of time that I almost forgot about it.

Last time I was threatened was the time of XMMS removal and it was more than two years ago by now. I don’t think this is related to that at all. But staying on the multimedia side of the fence, I can see a possible issue in people disliking PulseAudio with no good reason (the link is for a positive post); but even though I do lend a hand to Lennart with autotools, I sincerely doubt that my involvement is enough for people to want to get rid of me just for that.

It could have been some anti-Apple activist gone crazy for my previous post praising some of Apple’s products, I guess I should have started with a list of things I don’t like about Apple, or with a list of things that I do for Free Software each day and which is not going to stop just because I can settle for now with Apple products. But there are more chances that if somebody wants me dead is for dissing some projects he likes, it’s not like I didn’t criticise quite a few before, like cmake.

But if we expect this to be tied to something that happened recently, I shouldn’t rule out my criticism of Ruby 1.9 as well as my not-so-recent move from KDE to GNOME (I have to say, why if I move from KDE to GNOME, and I have been a KDE developer, it does not even make a news site, and if Linus does he gets on LWN ? I dare to say this is unjust!). These sound more likely for crazy guys just because they might feel “betrayed” since I was on their side before and then turned away, while for what concerns XMMS, PulseAudio, Apple and CMake I haven’t changed (much) my opinion.

Another option, if we follow what the mass-media has shown of black-hat hackers (even outside our general Western culture, is that somebody got upset about my recent security-oriented work, either because I found some security issue that they tried to hide around. Or it’s another security analyst that is upset because I found so many issues all at once.

All in all, I guess I could have enough reasons to worry if enough FOSS people suffer from the Reiser syndrome. Hopefully this is not the case. The good news is that nobody left me threats on the blog or via e-mail so I really don’t think I have to worry about FOSS people. And please if you think it would be fun to leave them now, just don’t, okay?