Network Security Services (NSS) and PKCS#11

Let’s clear first a big mess. In this post I’m going to talk about dev-libs/nss or, as the title suggests, Network Security Services which is the framework developed by Netscape first, and Mozilla Project now, for implementing a number of security layers, including (especially) SSL. This should not be confused with many others similar acronym, especially with the Name Service Switch which is the interface that allows your applications to resolve hosts and users against database they aren’t designed to use in the first place.

In my previous posts about smartcard-related software components – first and second – I started posting an UML components diagram that was not very detailed but generally readable. With time, and with the need to clarify my own understanding of the whole situation, the diagram is getting more complex, more detailed, but arguably less readable.

In the current iteration of the diagram, a number of software projects are exploded in multiple components, like I originally did with the lone OpenCryptoki project (which I should have been writing about but I hadn’t had enough time to finish cleaning off yet). In particular, I split the NSS component in two sub-components: libnss3 (which provides the actual API for the applications to use), and libnssckbi that provides access to the underlying NSS database. This is important because it shows how the NSS framework actually communicates with itself through the use of the standard PKCS#11 interface.

Anyway, back to NSS proper. To handle multiple PKCS#11 providers – which is what you want to do if you intend to use a hardware token, or a virtual one for testing – you need to register them with NSS itself. If you’re a Firefox user, you can do that from its settings windows, but if you’re a Chromium user, you’re mostly out of luck for what concerns GUI: the official way to deal with certificates et simila with Chromium is to use the NSS command-line utilities available with the utils USE flag for dev-libs/nss.

First of all, by default Mozilla, Evolution and Chromium, and the command-line utilities use three different paths to find their database: one depending on the Mozilla profile, ~/.pki/nssdb and .netscape respectively. Even more importantly, by default the first and last will use an “old” version of the db, based on the Berkeley DB interface, while the other two will use a more modern, SQLite-based database. This is troublesome.

Thankfully, the Mozilla Wiki has an article on setting up a shared database for NSS which you might want to do to make sure that you use the same set of certificates between Firefox, Chromium, Evolution and the command-line utilities. What it comes to be is just a bunch of symlinks. Read the article yourself for the instructions; on the other hand I have to note you to do this as well:

~ % ln -s .pki/nssdb .netscape

This way the nss utilities will use the correct database as well. Remember that you have to logout and log back in to tell the utilities and Firefox to use the SQL database.

Unfortunately I haven’t been able to get a token to work in this environment; from one side I’m afraid I might have busted the one Eva sent me (sigh! but at least it served the purpose of getting most of this running); from the other, Scute does not allow to upload an arbitrary certificate, but only to generate a CSR, which I obviously can’t get signed by StartSSL (which is my current certificate provider). Since I’m getting paranoid about security (even more so since I’ll probably be leaving my servers in an office when I’m not around), I’ll probably be buying an Aladdin token from StartSSL though (which also means I’ll be testing out their middleware). At that point I’ll give you more details about the whole thing.

Smartcards again

People seem to know by now that I have a particular passion for the security devices called smartcards. I’m not sure why myself, to be honest, but a few years ago I decided to look more into this, and nowadays I have three smartcard readers at home connected to the three main computers I use, and I use a FSFe card to store my GnuPG keys and to login to local and remote SSH services.

In Gentoo, unfortunately, most of the smartcard-related software has been vastly ignored for years, or was and still is only considered for the small use cases of developers and users, rather than in the general picture of it all. I have been trying to improve the situation ever since I first experimented with token-based login over one year and a half ago, but even my results are not really good.

The last hard work I did on the subject has been directed toward pcsc-lite improvements which brought me to hack at the code to improve support for two of the three devices I have here: the blutronics bludrive II CCID – which has a firmware quirk, requiring to look up the CCID description in the “wrong place” – and a Broadcom BCM5880 security device that provides dual-interface access to standard smartcards and for contact-less cards as well — I have to thank my trip to London two years ago for having a RFID card available at home to try it out!

Since my personal smartcard setup has been mostly complete and working fine for a while now, I wasn’t planning on working hard on anything in particular for a while, unless, like OpenCryptoki a couple of months ago, my job required me to. On the other hand, after my complaining about stable testing from last week, I started wondering if I couldn’t leverage the work I’ve been doing on OpenCryptoki to allow an easy way to test PKCS#11 software for people without the required hardware devices. Between that and a messed-up bump of OpenSC (0.12.0) in tree, I have been looking hard at the situation again.

Before moving on to describe the recent developments on the topic, though, I’d like to give an insight on why you cannot blame it on anyone in particular if the whole handling of smartcards in Gentoo. The following UML diagram is a schematic, vastly simplified component view of the software (and, very selectively, hardware) involved in smartcard access:

Smartcard Components UML diagram

In this diagram, the deep-green interfaces (circles) are those that are standardized by multiple organisations:

  • CCID is defined by the USB Forum;
  • CT-API is defined by a number of German organisations;
  • PC/SC is specified by its own workgroup which also defines the IFD interface;
  • PKCS#11 is defined by RSA.

The red components are implement as long-running services (daemons) on your Linux (or other Unix) system, the white ones are hardware devices, the blue ones are software libraries and finally the green ones are the applications the users use directly! Almost each one of those components is a standalone package (only package split in two components is GnuPG, and that’s just because Alon’s alternative SCD implementation makes it necessary to explicit the interface providers/consumers there.

This whole complexity not only makes it very difficult for distributions to manage software correctly, but also introduces a number of sensitive points of contacts between the software components, much more than one would like to have in a security-sensitive context such as Smartcards handling. Sometimes I wonder if they are really secure at all.

Back to what I have been doing in Gentoo, though. My first desire was to leverage The tpm-emulator and OpenCryptoki combo to allow arch testers to test PKCS#11 packages, such as pam_pkcs11 and pam_p11 (both of which are not part of the component diagram above by choice: to add those to the diagram, I would have had to add another indirection layer – libpam – to reach an user-accessible application like login) without the need of rare, and expensive, hardware devices. I’ve been working on OpenCryptoki’s ebuild and build system for a while, rewriting its build system and doing other general improvements — unfortunately it seems to me like it still doesn’t work as it is supposed to. I thought it could have been a problem with the software token emulation implementation, so I thought it might have been better to use the emulated TPM device, but even that method is not viable: even the latest version of the package does not seem to build properly against the current 2.6.38 Linux version, let alone the ancient version we have in the tree right now. I have a possibly-working ebuild for the 0.7 series (which uses cmake as basic build system), but since I can’t get the module to build, I haven’t committed it yet. This is likely one good candidate for the Character Device in UserSpace (CUSE) interfaces.

With the emulator being unbuildable, and the software-emulated token seemingly not working, using OpenCryptoki was thus slated for later review. I then switched my focus from that to OpenSC: version 0.12.0 was a major change, but in Gentoo it seems to have been bumped without proper consideration: for instance, the ebuild was committed with an optional pcsc-lite support, but without switches for any other alternative interface, and without any more support for the OpenCT interface that for some devices – including the iKey 3000 device that Gilles provided me with – is the only viable solution. Thanks to Alon (who’s a former Gentoo developer and an upstream developer for OpenCT/OpenSC), I was able to fix this up, and now OpenSC should be properly working in Gentoo — what is not currently implemented is support for non-OpenCT implementations of the CT-API interface; since I don’t know of other software that implements it that are available in Portage; if you know any of those, let me know and I’ll see to add support.

Now, for whatever reason, last time I worked on this, I ended up using pcsc-lite as my main hardware access provider – possibly because it is the easiest way to set it up for GnuPG and OpenPGP – and I didn’t want to throw it off right now, especially since I have a relatively good relationship with Ludovic (upstream) and I had already spent time fixing support for two of my three readers, as I said before. Thankfully, as the diagram suggests, OpenCT not only provides a CT-API interface, but also an IFD one, that can be used with pcsc-lite, providing a layered access to OpenCT-supported readers, including the ikey3k that I have here. Support for that in Gentoo, though, was not really sufficient: OpenCT didn’t install a bundle file for pcscd to discover, and the recent changes to pcscd to run without privileges disallowed the service from accessing the OpenCT sockets — I wouldn’t mind at some point moving all of the daemons to run under the same privileges, but that might not be so good an idea, and definitely not an easy one: while I can easily change the user and group settings that pcscd runs through – thanks to the Gentoo deviation I set the privileges just once, in the pcsc-lite udev rules file – it would probably require a bit of work to make sure that OpenCT and the other smartcard-enabled services didn’t step over each other’s toes. In the ~arch version of the two packages these issues are all solved, and indeed I can access the ikey3k device with pcsc_scan just fine, and from OpenSC as well.

I am unfortunately quite far from making use of the keys stored on the PKCS#11 devices in any other software than the PAM modules I have already written about. Alon’s alternative SCD implementation should make it possible to use any PKCS#11-compatible device (token or smartcard) to handle signatures for GnuPG and keys for SSH. What I’d be interested in, though, would be providing PKCS#11 interface to the OpenPGP card I have already, so to be able to mix devices. This should have been possible with OpenSC, as it implements an interface for openpgp applications and should expose it with PKCS#11 compatibility; reality, though, tends to disagree; I’m not sure whether it is simply a bug in the current code, or OpenPGPv2 cards not being supported by the project. I don’t think I’ll have enough time to work on that code anytime soon.

Alon suggested an alternative approach, by using “Scute’:http://www.scute.org/ — a project that aims at adding PKCS#11 interfaces to OpenPGP cards to be usable with Mozilla products. Unfortunately a quick check shows that the package does not build with the current version of its dependencies. And this is another task that would require me more time than I have, as I ”noted before“:https://flameeyes.blog/2011/04/storing-packing-and-disposing-of and thus will be simply slated for an undefined ”later”.

It’s not all gold that shines — Why underlinking is a bad thing

I have written a few days ago a rant about the gold linker, that really messed up the 64-bit tinderbox to the point I have to recreate it from scratch entirely. Some people asked me why I was interested in using gold, given I’m the first one who was sceptic about it — but not the only one as it happens.

Well, there are two reasons for me to be interested in this; the most obvious one is that I’m being paid to care (but I won’t/can’t go in further details, let’s just say that I have a contract that requires me to work at close contact with it), the other is that gold implements what I’ll be calling “underlinking protection” similar to what is done by the Apple linker, and similar to what would be achieved with --no-copy-dt-needed-entries on the good old ld (but in the case of ld, the softer --as-needed is actually going to make it quite moot.

Let me explain what the issue is here with a simple, real-world example to be found in git (this is one of the patches I worked on to get gold running). Since I’m really getting quite used to draw UML diagrams lately, I’m going to start with a diagram of the current situation. Please do note that I’m actually simplifying a lot since there are a number of components in GIT, of course, and also both libcrypto and libssl implement a number of interfaces. Instead, I have reduced the whole situation at a total of three interfaces, two libraries and one component. But it should do.

Example package diagram for GIT and OpenSSL

Yes I like pastel colours on diagrams.

So in this diagram we got two real packages (git and OpenSSL), that for what we’re concerned install respectively one application (imap-send) and two libraries (libssl and libcrypto). Of the two libraries, the former exports only SSL functions, while the latter exports HMAC methods and the EVP interface (which is a higher-level interface for multiple crypto and hash algorithms) — before somebody comments on this, yes I know they both export a lot more than that, I sincerely don’t care right now, I’m simplifying enormously to make this bearable.

As a specific note, I’m going to define the <<access>> and <<import>> terms in this way:

  • <<access>> means the first package is requiring symbols from the second, so it uses the second; this dependency is finalised during compile phase;
  • <<import>> means the first package includes a NEEDED entry for the second, so it links to the second, and thus it is finalised during link phase.

Now that we have an idea of the situation, let’s analyse the dependencies between the parties:

  • first of all we can rule out problems between the two libraries: the SSL functions access the EVP interface, but then again, libssl imports libcrypto, solving the dependency correctly;
  • then there is the dependency between imap-send and libssl: since it access the SSL interface, the component imports libssl, and that’s also correct for the dependencies;
  • on the other hand, while imap-send also accesses HMAC and EVP interfaces, it doesn’t import the libcrypto library; here is our problem.

In the case of the Apple linker and gold, this is an error condition: you cannot make use of the transitive import to produce a stable binary, so the link is rejected; you have to balance the two, by importing (linking in) both libssl and libcrypto. To be honest, in a situation like this, though, there is little to gain beside properness by restoring the balance — after all, the two libraries always come together and are linked one with the other.

There are, though, a number of situations where this does really matter a lot. Dependencies finalised during the compile phase cannot be swapped out at runtime if the binary interface changes, which is why we have the link phase and why we have sonames to express the binary interface compatibility. After seeing a real-world example, let’s go back to an abstract example.

Package diagram of the first abstract example

In this situation we have some given component (can be either an executable, another library, a plugin, it doesn’t really matter to what I’m going to talk about), that accesses interfaces from two different libraries: libfoo, and a dependency of that, libbar, but only links to the former — this is the minimal example of the GIT/OpenSSL interaction above. Just like in the previous real-world case, this situation, as is, is stable: since libfoo.so.1 links to libbar.so.1, the interfaces that the component access are to be found.

Now let’s assume that libbar gets updated to a new version, which is no longer ABI compatible (libbar.so.2); for the sake of argument, libfoo.so.1 still works, either because it was designed to support both source interfaces, or because only ABI changed, but API didn’t (for instance, when 32-bit parameters are replaced with 64-bit ones, you break ABI, but not API), or because there is some source-level compatibility interface. Or, to put it very very bluntly, you only need to rebuild the packages for libbar.so.2 to be usedreminds you of something?

Package diagram of the second abstract example

If you were to run revdep-rebuild (or any other similar approach), it would only rebuild libfoo.so.1 to link to the new dependency; at that point, you’d have libfoo.so.1 updated to both access the new interface and import the new library, but the original component would still be accessing the interfaces from libbar.so.1!

At this point, it depends on how libbar is designed what would happen; if the library was designed with symbol-versioning in mind from the start, or if it avoid re-using the same names between functions with the same ABI, then the original component would fail to load (or execute) because of missing symbols – of course with due caveats – but if they didn’t ensure this, and actually two interfaces with different ABIs shared the same name, you’d get the same effect of a symbol collision: crash, corruption, or generally unpredictable results.

Keeping all I explained in this post in mind, I’m pretty sure you can see the advantage in gold not allowing underlinking to begin with, and why I’m so scared of the softer --as-needed approach. Even if we’re not going to allow gold to be used as default linker anytime soon, we definitely have a number of good reasons why we should at least start using it to run the tinderbox — after all, even if it’s just triggered by gold, fixing underlinking will make it nicer for good old ld users.

Tomorrow, I’ll probably post something about the technical issues regarding testing gold, since there are at least a few.

Implementing the future! (or, how feng is designing secure RTSP)

While LScube is no longer – for me – a paid job project (since I haven’t been officially paid to work on it since May or so), it’s a project that intrigues me well enough to work on it on free time, rather than, say, watch a movie. The reason is probably to find in situations like those in the past few days, when Luca brings up a feature to implement that either no one else has implemented before, or simply needs to be re-implemented from scratch.

Last time I had to come up with a solution to implement something this intricate was when we decide to implement Apple’s own HTTP tunnelling feature whose only other documentation I know of, beside the original content from Apple is the one I wrote for LScube itself.

This time, the challenge Luca shown me is something quite more stimulating because it means breaking new ground that up to now seems to have been almost entirely missing: RTSP recording and secure RTSP.

The recording feature for RTSP has been somewhat specified by the original RFC but it was left quite up in the air as to what it should have been used for. Darwin Streaming Sever has an implementation, but beside the fact that we try not to look at it too closely (to avoid pollination), we don’t agree on the minimum amount of security that it provides as far as we can tell. But let’s start from the top.

The way we need RECORD to work, is to pass feng (the RTSP server) a live stream, which feng will then multiplex to the connected users. Up to now, this task was care of a special protocol between flux (the adaptor) and feng, which also went to a few changes over time: originally it used UDP over localhost, but we had problems with packet loss (no I can’t even begin to wonder why) and in performance; nowadays it uses POSIX message queues, which are supposedly portable and quite fast.

This approach has, objectively, a number of drawbacks. First of all the MQ stuff is definitely not a good idea to use; it’s barely implemented in Linux and FreeBSD, it’s not implemented on Darwin (and for a project whose aim is to replace Darwin Streaming Server this is quite a bad thing) and even though it does work on Linux, its performances aren’t exactly exciting. Also, using POSIX MQs is quite tricky: you cannot simply write an eventloop-based interface with libev because, even though they are descriptors in their implementation, an MQ descriptor does not support the standard poll() interface (nor any of the more advanced ones) so you end up with a thread blocking in reading… and timing out to make sure that flux wasn’t restarted in the mean time, since in that case, a new pair of MQs would have been created.

This situation was also not becoming any friendlier by the fact that, even though the whole team is composed of fewer than five developers, flux and feng are written in two different languages, so the two sides of the protocol implemented over MQs are also entirely separated. And months after I dropped netembryo from feng it is still used within flux, which is the one piece of the whole project that needed it the least, as it uses none of SCTP, TCP and SSL abstractions, and the interfaces of the two couldn’t be more far apart (flux is written in C++ if you didn’t guess, while netembryo is also written in C).

So it’s definitely time to get rid of the middleware and let the producer/adapter (ffmpeg) talk directly with the server/multiplexer (feng). This is where RECORD is necessary: ffmpeg will open a connection to feng via the same RTSP protocol used by the clients, but rather than ask to DESCRIBE a resource, it will ANNOUNCE one; then it will SETUP the server so that it would open RTP/RTCP ports, and start feeding data to the server. Since this means creating new resources and making it available to third parties (the clients) – and eventually creating files on the server’s disk, if we proceed to the full extent of the plan – it is not something that you want just any client to be able to do: we want to make sure that it is authorised to do so — we definitely don’t want to pull a Diaspora anytime soon.

Sequence Diagram of feng auth/record support

And in a situation like this, paying for a decent UML modeler, shows itself as a pretty good choice. A number of situations became apparent to me only when I went to actually think of the interaction to draw the pretty diagrams that we should show to our funders.

Right now, though, we don’t even support authentication, so I went to look for it; RTSP was designed first to be an HTTP-compatible protocol, so the authentication is not defined in RFC2326, but rather to be defined by HTTP itself; this means that you’ll find it in RFC 2616 instead — yes it is counterintuitive, as the RFC number is higher; the reason is simple though: RFC2616 obsoletes the previous HTTP description in 2068 which is the one that RTSP refers to; we have no reason to refer to the old one knowing that a new one is out, so…

You might know already that HTTP defines at least two common authentication schemes: Basic (totally insecure) and Digest (probably just as insecure, but obfuscated enough). The former simply provides username and password encoded in base64 — do note: encoded, not encrypted; it is still passed in what can be called clear text, and is thus a totally bogus choice to take. On the other hand, Digest support provides enough features and variables to require a very complex analysis of the documentation before I can actually come up with a decent implementation. Since we wanted something more akin to a Proof-of-Concept, but also something safe enough to use in production, we had to find a middle ground.

At this point I had two possible roads in front of me. On one side, implementing SASL looked like a decent solution to have a safe authentication system, while keeping the feng-proper complexity at bay; unfortunately, I still haven’t found any complete documentation on how SASL is supposed to be implemented as part of HTTP; the Wikipedia page I’m linking to also doesn’t list HTTP among the SASL-supported protocols — but I know it is supposed to work, both because Apache has a mod_authn_sasl and because if I recall correctly, Microsoft provides ActiveDirectory-authenticated proxies, and again if my memory serves me correctly, ActiveDirectory uses SASL. The other road looked more complex at first but it promised much with (relatively) little effort: implementing encrypted, secure RTSP, and require using it to use the RECORD functionality.

While it might sound a bit of overkill to require an encrypted secure connection to send data that has to be further streamed down to users, keep in mind that we’re talking only of encrypting the control channel; in most cases you’d be using RTP/RTCP over UDP to send the actual data, and the control stream can even be closed after the RECORD request is sent. We’re not yet talking about implementing secure RTP (which I think was already drafted up or even implement), but of course if you were to use a secure RTSP session and interleaved binary data, you’d be getting encrypted content as well.

At any rate, the problem with this became how to implement the secure RTSP connection; one option would have been using the oldest trick in the book and use two different ports, one for clear-text RTSP and another for the secure version, as is done for HTTP and HTTPS. But I also remembered that this approach has been frown upon and deprecated for quite a bit. Protocols such as SMTP and IMAP now use TLS through a STARTTLS command that replaces a temporarily clear-text connection with an encrypted TLS on; I remember reading about a similar approach for HTTP and indeed, RFC 2817 describes just that. Since RTSP and HTTP are so similar, there should be no difficulty into implementing the same idea over RTSP.

Sequence Diagram of feng TLS/auth/RECORD support

Now in this situation, implementing secure ANNOUNCE support is definitely nicer; you just refuse to accept methods (in write-mode) for the resource unless you’re over a secure TLS connection; then you also ensure that the user authenticated before allowing access. Interestingly, here the Basic authentication, while still bad, is acceptable for an usable proof of concept, and in the future it can easily be extended to authenticate based on client-side certificates without requiring 401 responses.

Once again, I have to say that having translated Software Engineering wasn’t so bad a thing; even though I usually consider UML being just a decent way to produce pretty diagrams to show the people with the gold purse, doing some modelling of the interaction between the producer and feng actually cleared up my mind enough to avoid a number of pitfalls, like doing authentication contextually to the upgrade to TLS, or forgetting about the need to provide two responses when doing the upgrade.

At any rate, for now I’ll leave you with the other two diagrams I have drawn but wouldn’t fit the post by themselves, they’ll end up on the official documentation — and if you wonder what “nq8” stands for, it’s the default extension generated by pngnq: Visual Paradigm produces 8-bit RGB files when exporting to PNG (and the SVG export produces files broken with RSVG, I have to report that to the developers), but I can cut in half the size of the files by quantising them to a palette.

I’m considering the idea of RE part of the design of feng into UML models, and see if that gives me more insight on how to optimise it further, if that works I should probably consider spending more time over UML in the future… too bad that reading O’Reilly PDF books on the reader is so bad, otherwise I would be reading through all of “Learning UML 2.0” rather than skimming through it when I need.

An insider view of some future addition to the PAM documentation

Okay, so today I got the definitive answer: the disk is no good: some of the files I was able to recover from lost+found disappeared again, and that included not only Python libraries (thanks to solar’s tinderbox I was able to fix that at least temporarily), but also libstdc++, and some of the glibc includes, that made it impossible to actually fix the installation.

For now I am “stuck” with the MacBook Pro, which is nice as I can still work on my job stuff (like for instance trying to write yet another article that will be rejected by a magazine, or some kind of software I can actually sell to someone, for specific tasks), but I can’t really help with most of the software I resumed maintaining for Gentoo in the past weeks (PAM, Proaudio, …).

Anyway, this doesn’t stop me from at least work on some of the PAM documentation companions I was working on, in particular, the following diagram is supposed to explain the interaction between the different pieces of a software during authentication:

PAM Component Interactions

Please tell me what you think, and yes I know it’s far from being a proper explanation, I’m planning on adding a more detailed diagram with the detailed timeline, following UML sequence diagrams style. This way you can get both a generic view for newbies, and a more detailed one for those who actually know how to read an UML diagram.

I think this is something I should have done before, and I have to thank Mike to suggesting this to me. Graphical companions to technical documentation is something important, but it’s not quite that easy; while UML provides good basis to documenting design and implementation, there are very few people and projects who actually use them, and publish them, before implementation, and after implementation it’s quite silly to start writing these things down, or is it?

Quite sometimes this might come more useful for people touching the code afterward, like new developers, or people somehow interested in a fork. While some people might see this as a good reason not to document the design and interface, to avoid people “stealing” their code, or design, there are two considerations to do: the first is that Free Software projects are about sharing code and experience, and documentation is part of it; the latter is that even when the developers are new developers for the same project, it is often likely that the new developers have little or no contact with the original developers after a few years, this is my case in xine. Let alone the cases where you’re “forced” to make a fork because the original developers disappeared.

Anyway, I’ll try to make good use of the time I’m stuck on the laptop, so I won’t waste it.. but first, I want to take a few days off, as what should have been my week of vacation last week was ruined by the data entry job (that completed today.. if I’m lucky, considering who I worked for, I’ll have my money three months for now).

Too bad that most of the anime series I was planning to see are on the currently-unavailable disks of Enterprise :(