Blast from the recent past; two years ago

Today I was feeling somewhat blue, mostly because I’m demotivated to do most stuff, and I wanted to see what it was like to work in Gentoo two years ago.

One thing I read is that a little shy of exactly two years ago, ICQ broke Kopete, just like they did yesterday. Interestingly enough, even though a workaround has been found for Kopete 0.12 (the one shipped with KDE 3.5), there is no bump I see in the tree this time. Sign that the KDE support in Gentoo has changed, most likely.

There is also the whole thing with ALSA problems, which span so many posts that it’s not worth listing all them. The current ALSA maintainer simply gave up on providing something that, at least for some users, ended up being quite important.

And all the work on Gentoo/FreeBSD! Although Javier is doing a huge work now to support FreeBSD 7.0, he’s not prone to blog about it, and you can see that Gentoo/FreeBSD is easily ending up in the “historical” memory, rather than being discussed and tried out by users daily.

What didn’t change at all is my insomnia, it’s almost 2AM and I’m still up. And this time I don’t even have Antiques Roadshow to watch. I’m currently working on xine, just like two years ago.

In general, I think a lot of areas in Gentoo did go downhill from two years ago, rather than improving. While Portage is certainly improved, thanks to Zac, Genone and the rest of the team, and we can see that in the new extended repoman checks, that also helps QA. But the general user support seems, to me, lacking.

This is a direct consequence, in my opinion, of leaving open doors for people who are just driving Gentoo’s energy away, by taking over projects to make them stall, by discussing details over and over and over, by repeating the same request even when people reject it as it stands, and so on.

I hope things will improve in the next months, thanks to a new council that can finally grow some balls, straightening up the situation, but if this does not happen, I’m already preparing for my plan B…

How many implementations of MD5 do you have in your system?

Anybody who ever looked into protocols and checksums or even downloaded the ISO file of a distribution a few years ago knows what an MD5 digest looks like. It’s obvious that the obnoxious presence of MD5 in our lives up to a few years ago (when it was declared non secure, and work started to replace it with SHA1 or SHA256) caused a similar obnoxious presence of it in the code.

Implementing MD5 checksumming is not really an easy task; for this reason there is almost the same code that gets reused from one library to another, and so on. This code has an interface that is more or less initialise, update, finish; the name of the functions might change a bit between implementation and implementation, but the gist is that.

Now, the most common provider of these functions is certainly OpenSSL (which also implements SHA1 and other checksum commands), but is not limited to. On FreeBSD, the functions are present in a system library (I forgot which one), and the same seems to happen in the previous Linux C library (libc.so.5, used before glibc). A common problem with OpenSSL is its GPL-incompatibility, for instance.

Now this means that a lot of software reimplemented their own MD5 code, using the same usual interface, with slightly different names: MD5Init, MD5_init, md5_init, md5init, md5_update, md5_append, md5_final, md5_finish and so on. All these interfaces are likely slightly different one with the other, to the point of not being drop-in replacements, and thus causing problems when they collide one with the other.

On every system, thus, there are multiple implementations of MD5, which, well, contributes to memory waste, as having a single implementation would be nicer and be easily shared between programs.

These packages implement their own copy of MD5 algorithms, and export them (sometimes correctly, sometimes probably by mistake): OpenSSL (obviously), GNUTLS (obviously, as it’s intended as semi-drop-in replacement for OpenSSL), GeoIP (uh?), Python (EH!? Python already links to OpenSSL, why on earth doesn’t it use SSL for MD5 functions really escapes me), python-fchksum (and why does it not use Python’s code?), Wireshark (again, it links to both GNUTLS and OpenSSL, why it does implement its own copy of MD5 escapes me), Kopete (three times, one for Yahoo plugin, one for Oscar – ICQ – plugin, and a different one for Jabber, it goes even better as KDE provides an MD5 implementation!), liblrdf, Samba (duplicated in four libraries), Wine (for advapi32.dll reimplementation, I admit that it might be requested for advapi32 to export it, I don’t know), pwdb, and FFmpeg (with the difference that FFmpeg’s implementation is not triggering my script as it uses its own interface).

I’m sure there are more implementations of MD5 on a system, as I said they are obnoxiously present in our lives still, for legacy protocols and data, and the range of different areas where MD5 checksums are used is quite wide (cryptography, network protocols, backup safety checks, multimedia – for instance the common checksum of decoded data to ensure proper decoding in lossless formats – and authentication). Likely a lot of implementations are hidden inside the source code of software, and it is likely impossible to get rid of them. But it would certainly be an interesting task if someone wants: sharing MD5 implementations means that optimising it for new CPUs will improve performance on all software using it.

If I wasn’t sure that most developers would hate me doing that, I’d pretty much like to open bugs for all the packages giving possible area of improvement of upstream code. As it is, contacting all upstreams, and creating a good lot of trackers’ accounts is something I wouldn’t like to do in my free time, but I can easily point out improvement areas for a lot of code. I just opened python-fchksum (which is used by Portage, which in turn means that if I can optimise it, I can optimise Portage), and beside the presence of MD5 code, I can see a few more things that I could improve in it. I’ll likely write the author with a patch and a note, but it’s certainly not feasible for me to do so for every package out there, alone and in my free time…

I’m not disappearing…

… just saying for who was wondering why I wasn’t blogging lately, that I’ve been occupied with my real life… mostly work, as I now have two jobs to take care of, well, one is less timeconsuming than the other, but they are still two, an and thus I’m trying to use my free time to relax myself.. not like I have much of that free time.

A particular thanks to Robin by the way, who today shown me how I can avoid restarting this box if xine crashing upsets my via82xx soundcard and disallow me from hearing sound from hw:0 (and just allowing iec958 output): iecset audio on do the trick.

Right now I’m unfortunately rebuilding OpenOffice, as I need to read a proprietary XLS file that drives my KSpread crazy (really, why do I have it installed? I hardly ever used a spreadsheet in years how… I think last time I did it seriously it was for an electronic class…), which means that also my testing is not that easy ;)

But, I’ve managed to commit already the fixes for the new ICQ lock out to Kopete 3.5.5-r2 (and kdenetwork-3.5.5-r1), I hope it can help.

By the way, Linux Kernel Development by Robert Love is proving an interesting book, Love’s writing style is anything but obnoxious with technicalisms, and the way he does not enter the debate between Linux and GNU/Linux, or Free and Open Source, made me take a deep satisfactory breath (I was expecting a boring discussion about that in the first chapter of the book). Even if it wasn’t cheap, I feel I spent my money in a sane way this time.

The ICQ mess, part N+M+K+1

So all the Kopete users will have noticed that I’ve just bumped it in portage, both for the standalone version and the two from kde-base. Reasoning? For the third time in less than a month, the ICQ protocol changed, or dropped some support.

So what happens? The stable versions are all broken, and I won’t be calling for a new stable for this, as we’re probably going to assist to newer breakages in the next months too. What I can only suggest people is to use net-im/kopete (that is not tied to the KDE releases and is right now ~arch only) if they want ICQ support working. Next time a patch will be needed it will be updated only on net-im/kopete.

Anyway, I have to thanks all the Kopete developers who made a really good job with fixing it promptly once again.

Now, for the “Why one has to bother to do this?” part, I want to suggest everybody here to use a Free as in Freedom protocol, an open protocol, like Jabber, as much as they can. We do have some good Jabber providers now, like KDETalk and of course Google Talk, that since a few months is interoperable with the other Jabber networks, and a few other (local) servers are listed on the Wikipedia page.

As the Wikipedia article also refer, there’s the problem of not having your friends on there. Well, just make them use Google Talk, Gaim, or whatever else, but let them join the Jabber network and you’ll be safe with protocol changes!

Problems with Kopete and ICQ

I e just finished having dinner, in front of emacs, while fixing today’s Kopete problems. For who didn’t notice, Kopete stopped working with ICQ today because of a change in ICQ’s policies, that kicked out older versions of the clients, included Kopete.

Luckily upstream reacted almost immediately and I was contacted by an user (padde on Freenode) with the information, I got the patch for net-im/kopete and I started working on it.

Unfortunately kdehiddenvisibility does not seem to work that well with that version, so I had to disable it for now. I also took the time this time to add a few useflags to disable some plugins and protocols, that should make Kopete take less time to build, and allows to have proper runtime dependencies for plugins like the LaTeX and the Cryptographic ones, that needs tetex and gnupg respectively.

Anyway, right now I patched net-im/kopete-0.12.0-r2, kde-base/kopete-3.5.3-r2 and kde-base/kdenetwork-3.5.3-r2 with the patch to fix ICQ, the older versions of kde-base stuff got an elog message to let users know of the problem. I might ask for a new stable in the next days, but I’m not sure yet.

Talking about early stables, xine-lib-1.1.2 is going to be stable sooner than I expected, this because of a bug in MiMMS code, that could be exploited and that will require a new stable. This means that I should be able to finally clean up the state of the xine-lib versions to something completely working and stable, hopefully.

You wake up and… oh another bump

Okay, last night I gone to sleep at 2am after unmasking KDE 3.5.3, this morning I wake up and find… another bump to do :P While I was sleeping seems like mattr found the time to release Kopete 0.12 final.

So I’m here, waiting for Kopete to build and then install. Also, the release notes seems to state that there won’t be a 0.12.1… this is from one side bad (because there is much work to do on it still) and from another good, as if I won’t be able to get wengophone working I might spend some more time to make Kopete’s plugins configurable at build time, so for instance no more LaTeX plugin if you don’t want to use LaTeX at all :)

Also this is conditioned to my free time of course. And to being unable to build wengophone with GCC 4.1 as I’m quite interested in it :)

To remain on topic of KDE applications, I have to say that the dear amaroK developers gave me quite a bit of work in preparation of amaroK 1.4.1… luckily I found time to handle most of the requirements already.

They dropped support for libvisual 0.2, so I had to put 0.4 in portage (for glory’s sake it is slottable so no big problems), they changed xine engine so that it asks to xine if it can play a given stream before trying to play it, and that uncovered a bug in xine’s FLAC handling (that is fixed for you all in -r8); they added an inotify backend for the directories rescan that required new linux-headers available.

Finally, I have to ask SUSE people about their patch to Helix Player to build on AMD64, so I can at least try to put that back in portage to provide a helix engine.

Okay, now it’s time to do some work.

New kopete

I’m happy to see that Matt decided to go with a new release of kopete out of KDE’s release cycle.

This is probably going to help quite a bit the development IMHO, as there won’t be any more 3.x series releases a part bugfixes, but Kopete really need more work on features and protocol support, not just bug fixing.

The new release, 0.12-alpha1, really impressed me: the new chat window engine, using CSS instead of XSL is way faster, although I had to edit the style I choose (from AdiumXtras, that I already knew as I use AdiumX on OSX) to remove my own icon from appearing, this because every time I sent a message it appeared to take my avatar image, and scale it down to what the skin required… pretty CPU intensive, especially as it wasn’t caching at all.
Well not a big deal, I don’t like watching those avatars anyway :)

Anyway for who wants to test the ebuild is in in my overlay. I’ll wait a bit more before looking into adding this in portage, just to solve the possible big issues with it.

I’m anyway good impressed with the work that’s being done. Although I didn’t enable the jingle (Google Talk’s talk support) as it requires an old version of oRTP (0.7.1) that would create a new dependency up-downgrading, and I’d rather wait to see if somebody is going to make it work with the current 0.8 series.

This version seems really to improve the responsiveness of the application in its entirety, so it’s a good step forward. Only thing I found strangely long was the shutdown… and I have no idea why, it starts a long series of munmap, takes a long time before it actually closes…

Oh well, I’m really looking forward to beta1.. and I really like being able to use Adium’s styles.