Multiple SSL implementations

If you run ~arch you probably noticed the a few days ago that all Telepathy-based applications failed to connect to Google Talk. The reason for this was a change in GnuTLS 3.1.7 that made it stricter while checking security parameters’ compliance, and in particular required 816-bit primes on default connections, whereas the GTalk servers provided only 768. The change has since been reverted, and version 3.1.8 connects to GTalk by default once again.

Since I hit this the first time I tried to set up KTP on KDE 4.10, I was quite appalled by the incompatibility, and was tempted to stay with Pidgin… but at the end, I found a way around this. The telepathy-gabble package was not doing anything to choose the TLS/SSL backend but deep inside it (both telepathy-gabble and telepathy-salut bundle the wocky XMPP library — I haven’t checked whether it’s the same code, and thus if it has to be unbundled), it’s possible to select between GnuTLS (the default) or OpenSSL. I’ve changed it to OpenSSL and everything went fine. Now this is exposed as an USE flag.

But this made me wonder: does it matter on runtime which backend one’s using? To be obviously honest, one of the reasons why people use GnuTLS over OpenSSL is the licensing concerns, as OpenSSL’s license is, by itself, incompatible with GPL. But the moment when you can ignore the licensing issues, does it make any difference to choose one or the other? It’s a hard question to answer, especially the moment you consider that we’re talking about crypto code, which tends to be written in such a way to be optimized for execution as well as memory. Without going into details of which one is faster in execution, I would assume that OpenSSL’s code is faster simply due to its age and spread (GnuTLS is not too bad, I have some more reserves for libgcrypt but nevermind that now, since it’s no longer used). Furthermore, Nikos himself noted that sometimes better algorithm has been discarded, before, because of the FSF copyright assignment shenanigan which I commented on a couple of months ago.

But more importantly than this, my current line of thought is wondering whether it’s better to have everything GnuTLS or everything OpenSSL — and if it has any difference to have mixed libraries. The size of the libraries themselves is not too different:

        exec         data       rodata        relro          bss     overhead    allocated   filename
      964847        47608       554299       105360        15256       208805      1896175   /usr/lib/libcrypto.so
      241354        25304        97696        12456          240        48248       425298   /usr/lib/libssl.so
       93066         1368        57934         2680           24        14640       169712   /usr/lib/libnettle.so
      753626         8396       232037        30880         2088        64893      1091920   /usr/lib/libgnutls.so

OpenSSL’s two libraries are around 2.21MB of allocated memory, whereas GnuTLS is 1.21 (more like 1.63 when adding GMP, which OpenSSL can optionally use). So in general, GnuTLS uses less memory, and it also has much higher shared-to-private ratios than OpenSSL, which is a good thing, as it means it creates smaller private memory areas. But what about loading both of them together?

On my laptop, after changing the two telepathy backends to use OpenSSL, there is no running process using GnuTLS at all. There are still libraries and programs making use of GnuTLS (and most of them without an OpenSSL backend, at least as far as the ebuild go) — even a dependency of the two backends (libsoup), which still does not load GnuTLS at all here.

While this does mean that I can get rid of GnuTLS on my system (NetworkManager, GStreamer, VLC, and even the gabble backend use it), it means that I don’t have to keep loaded into memory that library at all time. While 1.2MB is not that big a save, it’s still a drop in the ocean of the memory usage.

So, while sometimes what I call “software biodiversity” is a good thing, other times it only means we end up with a bunch of libraries all doing the same thing, all loaded at the same time for different software… oh well, it’s not like it’s the first time this happens…

GNU is actually a totalitarian regime

You probably remember that I’m not one to fall in line with the Free Software Foundation — a different story goes for the FSFe which I support and I look forward for the moment when I can come back as a supporter; for the moment I’m afraid that I have to contribute only as a developer.

Well, it seems like more people are joining in the club. After Werner complained about the handling of GNU copyright assignments – not much after my covering of Gentoo’s assignments which should probably make those suggesting a GNUish approach to said copyright assignment think a lot – Nikos of GnuTLS decided to split off the GNU project.

Why did Nikos decide this? Well, it seems like the problem is that both Werner and Nikos are tired of the secrecy in the GNU project and even more of the inability to discuss, even in a private setting, some topics because they are deemed taboo by the FSF, in the person of Richard Stallman.

So, Nikos decided to move the lists, source code and website on its own hosting, and then declared GnuTLS no longer part of the GNU project. Do you think that this would have put the FSF in a “what are we doing wrong?” mood? Hah, naïve are you! Indeed the response from the FSF (in the person of Richard Stallman, see a pattern?) was to tell Nikos (who wrote, contributed to GNU, and maintained the project) that he can’t take his own project and take it out of GNU, and that if he wants he can resign from the maintainer’s post.

Well, now it seems like we might end up with a “libreTLS” package, as Nikos is open to renaming the project… it’s going to be quite a bit of a problem I’d say, if anything because I want to track Nikos’s development more than GNU’s, and thus I would hope for the “reverse fork” from the GNU project to just die off. Considering I also had to sign the assignment paperwork (and in the time that said paperwork was being handled, I lost time/motivation for the contributions I had in mind, lovely isn’t it?).

Well, what this makes very clear to me is that I still don’t like the way the GNU project, and the FSF are managed, and that my respect for Stallman’s behaviour is, once again, zero.

Gentoo Linux Health Report — October 2012 Edition

I guess it’s time for a new post on what’s the status with Gentoo Linux right now. First of all, the tinderbox is munching as I write. Things are going mostly smooth but there are still hiccups due to some developers not accepting its bug reports because of the way logs are linked (as in, not attached).

Like last time that I wrote about it, four months ago, this is targeting GCC 4.7, GLIBC 2.16 (which is coming out of masking next week!) and GnuTLS 3. Unfortunately, there are a few (biggish) problems with this situation, mostly related to the Boost problem I noted back in July.

What happens is this:

you can’t use any version of boost older than 1.48 with GCC 4.7 or later;
you can’t use any version of boost older than 1.50 with GLIBC 2.16;
many packages don’t build properly with boost 1.50 and later;
a handful of packages require boost 1.46;
boost 1.50-r2 and later (in Gentoo) no longer support eselect boost making most of the packages using boost not build at all.

This kind of screwup is a major setback, especially since Mike (understandably) won’t wait any more to unmask GLIBC 2.16 (he waited a month, the Boost maintainers had all the time to fix their act, which they didn’t — it’s now time somebody with common sense takes over). So the plan right now is for me and Tomáš to pick up the can of worms, and un-slot Boost, quite soon. This is going to solve enough problems that we’ll all be very happy about it, as most of the automated checks for Boost will then work out of the box. It’s also going to reduce the disk space being used by your install, although it might require you to rebuild some C++ packages, I’m sorry about that.

For what concerns GnuTLS, version 3.1.3 is going to hit unstable users at the same time as glibc-2.16, and hopefully the same will be true for stable when that happens. Unfortunately there are still a number of packages not fixed to work with gnutls, so if you see a package you use (with GnuTLS) in the tracker it’s time to jump on fixing it!

Speaking of GnuTLS, we’ve also had a smallish screwup this morning when libtasn1 version 3 also hit the tree unmasked — it wasn’t supposed to happen, and it’s now masked, as only GnuTLS 3 builds fine with it. Since upstream really doesn’t care about GnuTLS 2 at this point, I’m not interested in trying to get that to work nicely, and since I don’t see any urgency in pushing libtasn1 v3 as is, I’ll keep it masked until GNOME 3.6 (as gnome-keyring also does not build with that version, yet).

Markos has correctly noted that the QA team – i.e., me – is not maintaining the DevManual anymore. We made it now a separate project, under QA (but I’d rather say it’s shared under QA and Recruiters at the same time), and the GIT Repository is now writable by any developer. Of course if you play around that without knowing what you’re doing, on master, you’ll be terminated.

There’s also the need to convert the DevManual to something that makes sense. Right now it’s a bunch of files all called text.xml which makes editing a nightmare. I did start working on that two years ago but it’s tedious work and I don’t want to do it on my free time. I’d rather not have to do it while being paid for it really. If somebody feels like they can handle the conversion, I’d actually consider paying somebody to do that job. How much? I’d say around $50. Desirable format is something that doesn’t make a person feel like taking their eyes out when trying to edit it with Emacs (and vim, if you feel generous): my branch used DocBook 5, which I rather fancy, as I’ve used it for Autotools Mythbuster but RST or Sphinx would probably be okay as well, as long as no formatting is lost along the way. Update: Ben points out he already volunteered to convert it to RST, I’ll wait for that before saying anything more.

Also, we’re looking for a new maintainer for ICU (and I’m pressing Davide to take the spot) as things like the bump to 50 should have been handled more carefully. Especially now that it appears that it’s basically breaking a quarter of its dependencies when using GCC 4.7 — both the API and ABI of the library change entirely depending on whether you’re using GCC 4.6 or 4.7, as it’ll leverage C++11 support in the latter. I’m afraid this is just going to be the first of a series of libraries making this kind of changes and we’re all going to suffer through it.

I guess this is all for now.

GNU software (or, gnulib considered harmful and other stories)

The tinderbox right now seems to be having fun trying to play catch-up with changes in GNU software: GCC, Automake, glibc and of course GnuTLS. Okay it’s true that compatibility problems are not a prerogative of GNU, but there are some interesting problems with this whole game, especially for what concerns inter-dependencies.

So, there is a new C library in town, which, if we ignore the whole x32 dilemma, has support for ISO C11 as its major new feature. And what is the major change in this release? The gets() function has finally been removed. This is good, as it was a very nasty thing to use, and nobody in his sane mind would use it…

tbamd64 ~ # scanelf -qs -gets -R /bin /opt /sbin /lib /usr
gets  /opt/flexlm/bin/lmutil

Okay nevermind those who actually use it, the rest of the software shouldn’t be involved, should it? You wish. What happens is that since gnulib no longer only carries replacements for GNU extensions but also includes code that is not present on glibc itself, and extra warnings about use of deprecated features, it now comes with its own re-declaration of gets() to feature a prominent warning if it’s used. And of course, that makes it fail badly with the new gnulib.

Obviously, this has been fixed in gnulib already, since it was planned for gets() to be removed, but it takes quite a bit of time for a fix in gnulib to trickle down to the packages using it, which is one of my previous main complaints about it. Which means that Gentoo will have to patch the same code over and over again in almost all GNU software, since almost all of it uses gnulib.

Luckily for me only two packages have hit (with this problem, at least) that I’m in the herd of: GnuTLS and libtasn1 which is a dep of it. The former is fixed in version 3 which is also masked but I’m testing as well, while the latter is fixed in the current ~arch (I really don’t care about 2.16 in stable yet!), so there is nothing to patch there. The fact that GCC 4.6 itself fails to build with this version of glibc is obviously a different problem altogether, and so is the fact that we need Boost 1.50 for a number of packages to work with the new glibc/gcc combination, as 1.49 is broken with the new C library and 1.48 is broken with the new compiler.

Now to move on to something different: Automake 1.12 was released a couple of months ago and is now in ~arch, causing trouble although not as bad as it could have been. Interestingly enough, one of the things that they did change in this version was removing $(mkdir_p) as I wrote in my guide — but that seems to have been a mistake.

What should have been removed in 1.12 was the default use of AM_PROG_MKDIR_P, while the mkdir_p define should have been kept around until version 1.13. Stefano said he’s going to revert that change in automake 1.12.2, but I guess it’s better if we deal with it right away instead of waiting for 1.13 to hit us….

Of course there is a different problem with the new automake as well: GNU gettext hasn’t been updated to support the new automake versions, so using it causes deprecation warnings with 1.12 (and will fail with 1.13 if it’s not updated). And of course a number of projects are now using -Werror on automake, like it wasn’t enough trouble to use it on the source code itself.

And of course the problem with Gentoo, according to somebody, is the fact that my tinderbox bugs (those dozen a day) are filed with linked build logs instead of attached ones. Not the fact that the same somebody commits new versions of critical packages without actually doing anything to test them.

Choosing an MD5 implementation

_Yes this post might give you a sense of deja vu read but I’m trying to be more informative than ranty, if I can, over two years and a half after the original post…

Yesterday I ranted about gnulib and in particular about the hundred and some copies of the MD5 algorithm that it brings with it. Admittedly, the numbers seem more sensational than they would if you were to count the packages involved, as most of the copies come from GNU octave and another bunch come from GCC and so on so forth.

What definitely surprises me, though, is the way that people rely on said MD5 implementation, like there was no other available. I mean, I can understand GCC not wanting to add any more dependencies that could make it ABI-dependent – just look at the recent, and infamous, gmp bump – but did GNU forget that it has its own hash library ?

There are already enough MD5 implementations out there to fill a long list, so how do you choose which one to use, rather than add a new one onto your set? Well, that’s a tricky question. As far as I can tell, the most prominent factors in choosing an implementation for hash algorithms are non-technical:

Licensing is likely to be the most common issue: relying on OpenSSL’s libcrypto means that you rely on a software the license of which has been deemed enough incompatible with GPL-2 that there is a documented SSL exception that is used by projects using the GNU General Public License together with those libraries. This is the reason why libgcrypt exists, for the most part, but this continues GNU’s series of “let’s reinvent the wheel, and then reinvent it, and again”, GnuTLS (which is supposed to be a replacement to OpenSSL itself) also provides its own implementation. Great.

Availability can be a factor as well: software designed for BSD systems – such as libarchive – will make use of the libcrypto-style interface just fine; the reason is that at least FreeBSD (and I think NetBSD as well) provides those functions in its standard libraries set, making it the obvious available implementation (I wish we had something like that). Adding dependencies on a software is usually a problem, and that’s why gnulib’s used often times (being imported in the project’s sources, it adds no further dependency). So if your average system configuration already contains an acceptable implementation, then that’s what you should go for.

Given that all the interfaces are almost identical one to the other with the exception of the names and structure, and that their actual implementation has to follow the standard to make sense, the lack of many technical reasons in the prominent factors for choosing one library over another is generally understandable. So how should one proceed to choose which one to use? Well, I have some general rule of thumbs that could be useful.

The first is to use something that you already have available or you’re using already in your project: OpenSSL’s libcrypto, GLIB and libav/ffmpeg all provide an implementation of most hash functions. If you already rely on them for some of your needs, simply use their interfaces rather than looking for new ones. If there are bugs in those, or the interface is not good enough, or the performances are not as expected, try to get those fixed instead of cooking your own implementation.

If you are not using any other implementation already, then try to look at libgcrypt first; the reason why I suggest this is because it’s a common library (it is required among others by GnuPG), implementing all the main hashing algorithms, it’s LGPL so you have very few license concerns, and it doesn’t implement any other common interface, so you don’t risk symbol collision issues, as you would if you were to use a libcrypto-compatible library, and at the same time bring in anything else that used OpenSSL — I have seen that happening with Drizzle a longish time ago.

If you’re a contributor to one of the projects that use gnulib’s md5 code… please consider porting it to libgcrypt, all distributors will likely thank you for that!

Too many alternatives are not always so good

I can be quite difficult to read for what concerns alternative approaches to the same problem; while I find software diversity to be an integral part of the Free Software ideal and very helpful to find the best approach to various situations, I also am not keen on maintaining the same code many time because of that, and I’d rather have projects to share the same code to do the same task. This is why I think using FFmpeg for almost all the multimedia projects in the Free Software world is a perfectly good thing.

Yesterday, while trying to debug (with the irreplaceable help of Jürgen) a problem I was having with Gwibber (which turned out to be an out-of-date ca-certificates tree), I noted one strange thing with pycurl, related to this fact, that proves my point to a point.

CURL can make use of SSL/TLS encryption using one out of three possible libraries: OpenSSL, GnuTLS and Mozilla NSS. The first option is usually avoided by binary distributions because it is incompatible with some licensing terms; the third option is required for instance by the Thunderbird binary package in Gentoo as it is. By default Gentoo uses OpenSSL, that you like it or not.

When CURL is built against OpenSSL (USE="ssl -gnutls -nss"), PyCURL linked to libcrypto; given that my system is built with forced --as-needed, it also means it uses it. I found it quite strange so I went to look at it; if you rebuild CURL (and then PyCURL) with GnuTLS (USE="ssl gnutls -nss") you’ll see that it only links to libgnutls, but if you look closer, it’s using at least one libgcrypt symbol. Finally if you build it with Mozilla NSS (USE="ssl -gnutls nss") then it will warn that it didn’t detect the SSL library used.

The problem here is that CURL seems not to provide a total abstraction of the SSL implementation it uses, and for proper threading support, PyCURL needs to run special code for the crypto-support library (libcrypto for OpenSSL; libgcrypt for GnuTLS). I’m sincerely not sure how big the problem would be when you mix and match the CURL and PyCURL implementations, I also have no idea what would happen if you were to use CURL with NSS and PyCURL with that (which will not provide locking for crypto at all). What I can tell you, is that if you change the SSL provider in CURL, you’d better rebuild PyCURL, to be on the safe side. And there is currently no way to let Portage do that automatically for you.

And if you are using CURL with NSS and you see Portage asking you to disable it in favour of GnuTLS or OpenSSL, you’ll know why: PyCURL is likely to be your answer. At least once the bug will be addressed.

How many implementations of MD5 do you have in your system?

Anybody who ever looked into protocols and checksums or even downloaded the ISO file of a distribution a few years ago knows what an MD5 digest looks like. It’s obvious that the obnoxious presence of MD5 in our lives up to a few years ago (when it was declared non secure, and work started to replace it with SHA1 or SHA256) caused a similar obnoxious presence of it in the code.

Implementing MD5 checksumming is not really an easy task; for this reason there is almost the same code that gets reused from one library to another, and so on. This code has an interface that is more or less initialise, update, finish; the name of the functions might change a bit between implementation and implementation, but the gist is that.

Now, the most common provider of these functions is certainly OpenSSL (which also implements SHA1 and other checksum commands), but is not limited to. On FreeBSD, the functions are present in a system library (I forgot which one), and the same seems to happen in the previous Linux C library (libc.so.5, used before glibc). A common problem with OpenSSL is its GPL-incompatibility, for instance.

Now this means that a lot of software reimplemented their own MD5 code, using the same usual interface, with slightly different names: MD5Init, MD5_init, md5_init, md5init, md5_update, md5_append, md5_final, md5_finish and so on. All these interfaces are likely slightly different one with the other, to the point of not being drop-in replacements, and thus causing problems when they collide one with the other.

On every system, thus, there are multiple implementations of MD5, which, well, contributes to memory waste, as having a single implementation would be nicer and be easily shared between programs.

These packages implement their own copy of MD5 algorithms, and export them (sometimes correctly, sometimes probably by mistake): OpenSSL (obviously), GNUTLS (obviously, as it’s intended as semi-drop-in replacement for OpenSSL), GeoIP (uh?), Python (EH!? Python already links to OpenSSL, why on earth doesn’t it use SSL for MD5 functions really escapes me), python-fchksum (and why does it not use Python’s code?), Wireshark (again, it links to both GNUTLS and OpenSSL, why it does implement its own copy of MD5 escapes me), Kopete (three times, one for Yahoo plugin, one for Oscar – ICQ – plugin, and a different one for Jabber, it goes even better as KDE provides an MD5 implementation!), liblrdf, Samba (duplicated in four libraries), Wine (for advapi32.dll reimplementation, I admit that it might be requested for advapi32 to export it, I don’t know), pwdb, and FFmpeg (with the difference that FFmpeg’s implementation is not triggering my script as it uses its own interface).

I’m sure there are more implementations of MD5 on a system, as I said they are obnoxiously present in our lives still, for legacy protocols and data, and the range of different areas where MD5 checksums are used is quite wide (cryptography, network protocols, backup safety checks, multimedia – for instance the common checksum of decoded data to ensure proper decoding in lossless formats – and authentication). Likely a lot of implementations are hidden inside the source code of software, and it is likely impossible to get rid of them. But it would certainly be an interesting task if someone wants: sharing MD5 implementations means that optimising it for new CPUs will improve performance on all software using it.

If I wasn’t sure that most developers would hate me doing that, I’d pretty much like to open bugs for all the packages giving possible area of improvement of upstream code. As it is, contacting all upstreams, and creating a good lot of trackers’ accounts is something I wouldn’t like to do in my free time, but I can easily point out improvement areas for a lot of code. I just opened python-fchksum (which is used by Portage, which in turn means that if I can optimise it, I can optimise Portage), and beside the presence of MD5 code, I can see a few more things that I could improve in it. I’ll likely write the author with a patch and a note, but it’s certainly not feasible for me to do so for every package out there, alone and in my free time…