My thoughts on Keybase

Keybase is one of a series of new services that appear to have come up in the wake of the publication of Snowden’s document, and the desire for more and simpler crypto technologies. I may disagree with the overall message attached to the Church of Snowden (Jürgen phrases it much better than me), easier crypto is something I’m generally happy with.

Unfortunately I’m not sure if Keybase’s promise of making GnuPG easier and at the same time keep it safer is actually being maintained. It appears to make it easier, at least under certain conditions, but I disagree with it staying just as secure, particularly if you follow their “default flow.”

The first problem that comes to mind is that they even suggest you to upload your private key to their system so that you can use the browser for interacting with it! I hoped they were kidding me, but no, it seems like that’s an option, actually the first of three options when you try to do anything at all with the website.

The second is the fact that for a lot of the features to make even remote sense you have to use the command line at that point, either through their tool or through a combination of curl and gnupg. It might seem strange that I’m complaining at both ends, but it’s because I would have preferred for them to provide, say, a Chrome extension that interfaces with gnupg, than a command line tool. Even more so when you realize that the command line tool depends on NodeJS, and it includes a -TSR- background service.

The command line tool is also not great. Indeed when you try to log in with it, by default it’ll use pinentry, which, if started with a DISPLAY environment set, will use the graphical version (in my case, Qt.) The graphical version do not allow you to paste, which makes sense for the passphrase of a private key, or the PIN of a smartcard (if you save those in a password manager in the same system, there is very little protection provided anyway, you can leave them unkeyed.) But if you’re trying to access a service… significantly less so. I worked this around by unsetting the DISPLAY environment variable and using the console Pinentry, and just paste the password in Konsole.

But it goes more interesting when you start noticing things that are significantly broken. Keybase requires you prove access to the key you want to mark as yours, which is the obvious thing to do, and that’s good. Unfortunately they don’t seem to cope well with the idea of key expiration. From what I read in various related issues the reason is that they think key expiration is an useless concept thanks to Keybase. That may be the case if you have no other environment, but I’d think this is a myopic point of view. By the way it doesn’t matter if you extend your expiration date in time, you still have to re-prove it to Keybase because you don’t seem to be able to provide them with an updated copy of the key (like you would with a normal keyserver.)

Once I got access to my account back I managed to re-prove my website; this was needed because I moved providers (long story) for the blog and everything, and so the proof (which for whatever reason I forgot to add to the git repository I store my website on) went… poof. Unfortunately it was a bit more involved than just generating a new proof. Mostly because the fetcher that should verify said proof does not actually respect the HTTP standard requirements and provide no Accept header, which meant ModSecurity kicked it out. You’d expect that a service that is all about security and trust would at least be able to implement the protocol correctly.

To finish this off, I really dislike the “limited invites” options in general. I understand why that’s needed, but it just feels a bit useless to me, particularly when, just because I logged back in, the system granted me more invites — with the “cute and whimsical” notion that it’s the founder of the service to “grant” you those invites. Heh.

All in all, I don’t have much real use out of this system. I signed up because it was suggested it’s a nice way to prove my identity but I don’t feel it’s any better than the Web of Trust, and I’m not saying the WoT is good.

Oh well, if it takes off I’ll be there, if not, I have only spent a minimum amount of time on it.

Privacy Theatre

I really wish I could take credit for the term, but Jürgen points out he coined the term way before me, in German: Datenschutztheater. I still like to think that the name fits many behaviours I see out there, and it’s not a coincidence that it sounds like the way we think of TSA’s rules at airports, security theatre.

I have seen lots and lots of people advocating for 100% encryption of everything, and hiding information and all kind of (in my opinion) overly paranoid suggestions for everybody, without understanding any threat model at all, and completely forgetting that your online privacy is only a small part of the picture.

I have been reminded of this as I proceeded sorting out my paperwork here in Dublin, which started piling up a little too much. My trick is the usual I used in Italy too: scan whatever is important to keep a copy of, and unless the original is required for anything, I destroy the hard copy. I don’t trash it, I destroy it. I include anything that has my address on it, and when I was destroying it with my personal shredder, I always made sure to include enough “harmless” papers in the mix to make it more difficult to filter out the parts that looked important.

As I said in my previous post, I’m not worried about “big” corporations knowing things about me, like Tesco knowing what I like to buy. I find it useful, and I don’t have a problem with that. On the other hand, I would have a problem with anybody, wanting to attack me directly, decided to dumpster-dive me.

Another common problem I see that I categorize as Privacy Theatre is the astounding lack of what others would call OpSec. I have seen plenty of people at conferences, even in security training, using their laptop without consideration for the other people in the room, and without any sort of privacy screen. In one of the past conferences I’ve seen mail admins from a provider that will go unnamed, working on production issues in front of my eyes: if I had mischievous intents I would have learnt quite a bit about their production environment.

Yes I know that the screens are a pain, and that you have to keep taking them in and out, and that they take away some of the visual space on your monitor. Myself, for my personal laptop I decided for a gold privacy screen by 3M, which is bearable to use even if you don’t need it, as long as you don’t need to watch movies on your laptop (I don’t, the laptop’s display is good but I have a TV and a good monitor for that).

But there are tons of other, smaller pieces that people who insist they are privacy advocates really don’t seem to care about. I’m not saying that you should be paranoid, actually I’m saying the exact opposite: try to not be the paranoid person that wants everything encrypted without understanding why. In most cases, Internet communication needs to be encrypted indeed. And you want to encrypt your important files if you put them in the cloud. But at the same time there are things that you don’t really care about that much and you’re just making your life miserable because Crypto-Gods, while the same energy could be redirected to save you from more realistic petty criminals.

Random quality

RFC 1149.5 specifies 4 as the standard IEEE-vetted random number.

xkcd’s Random Number comic © Randall Munroe

We all know that random numbers might not be very random unless you are very careful. Indeed, as the (now old) Debian OpenSSL debacle, a not-enough-random random number generator can be a huge breach in your defences. The other problem is that if you want really random numbers you need a big pool of entropy otherwise code requiring a huge chunk of random bytes would stall until enough data is available.

Luckily there are a number of ways to deal with this; one is to use the EntropyKey while other involves either internal sources of entropy (which is what timer_entropyd and haveged do), or external ones (audio_entropyd, but a number of custom circuitry and software exist as well). These fill in the entropy pool, hopefully at a higher rate than it is depleted, providing random data that is still of high quality (there are other options such as prngd, but as far as I can tell those are slightly worse in term of quality).

So, the other day I was speaking with Jaervosz, who’s also an EntropyKey user, and we were reflecting on whether, if there is not enough entropy during crypto operations, the process would stall or cause the generation to be less secure. In most cases, this shouldn’t be a problem: any half-decent crypto software will make sure not to process pseudo-random numbers (this is why OpenSSL key generation tells you to move your mouse or something).

What we ended up wondering about, was how much software uses /dev/urandom (that re-uses the entropy when it’s starving) rather than /dev/random (which blocks on entropy starvation). Turns out there are quite a few. For instance on my systems, I know that Samba uses /dev/urandom, and so does netatalk — neither of which make me very happy.

A few ebuilds allow you to choose which one you want to use through the (enabled-by-default) urandom USE flag… but these I noted above aren’t among those. I suppose, one thing we could be doing would be going over a few ebuilds and see if we can make it configurable which one to use.. for those of us who make sure to have a stable source of entropy, this change should be a very good way to be safe.

Are you wondering if any of your mission-critical services are using /dev/urandom ? Try this:

# fuser -v /dev/{,u}random
                     USER        PID ACCESS COMMAND
/dev/random:         root      12527 F.... ekey-egd-linux
/dev/urandom:        root      10129 f.... smbd
                     root      10141 f.... smbd
                     root      10166 f.... afpd
                     flame     12356 f.... afpd

Also, if you want to make sure that any given service is started only after the entropy services, you can simply make it depend on the virtual service entropy (provided by haveged, or ekeyd if set to kernel output, or ekey-egd-linux if set to EGD output). A quick way to do so without having to edit the init script yourself, is to add the following line to /etc/conf.d/$SERVICENAME:

rc_after="entropy"

The entropy factor

I have been dedicating my last few posts to the EntropyKey limitations and to shooting down Entropy Broker so I guess it is time for me to bring good news to my users, or at least some more usable news.

Let’s start with some positive news about EntropyKey; I have already suggested in the latter post that the problem I was getting with the key not working after a reboot could have been linked to the use of the userland USB access method. I can now confirm this: my router reboots fine with the CDC-based access. Unfortunately this brings up a different point: Linux’s CDC-ACM driver is far from perfect; as I pointed out, Simtec has a compatibility table for the kernel versions that are known to be defective. And even the latest release (2.6.38) is only partially fine — there are fixes in 2.6.38.2 that relates to possible memory corruption and null pointer dereferences in the drivers. But it definitely makes the EntropyKey much more useful than it might have been suggested by my post there.

I also wanted to explain a bit better what I’m trying to look at with entropy, processes creation, and load. One of the two EntropyKey devices I own is connected to Yamato, which is the same box that runs the tinderbox. The entropy on Yamato tends to be quite low in general, and when I was using it as my main workstation, it was feeling sluggish — thus why I got the Key. I started monitoring it lately because I was noticing the Key not behaving properly, and beside the already noted issues with the userland access method, I have noticed a different pattern: when the load (and the running processes) spikes, the entropy fell for a while, without it being replenished in time. I’m still not sure if it’s simply the process creation eating away the entropy (for the ASLR mostly, since Yamato, unlike my router, doesn’t use SSP and thus doesn’t require canaries), or if the build process for ChromiumOS (which is what triggers those spikes) eats up the entropy by itself.

Thanks to upstream as I said before, I know there are somewhere around 4KiB of entropy gathered by the key per second, which means if something drains more than 4KiB/second, /dev/urandom will start re-using entropy. I have discarded the notion that it might have been a priority issue since I made sure to decrease the niceness of the ekeyd daemon, and it didn’t make any significant issue; on the other hand I’m now experimenting with using the same interface designed for vservers that Jaervosz helped getting into Gentoo. This method splits the process reading data from the key and that of sending it to the kernel — a quick check shows me that it does have better results, which would suggest that there are indeed performance-related problems, more than high-level of crypto consumption, at least on basic usage — I have yet to run a full rebuild of ChromiumOS in this configuration, since it’s over 6am and I’ll probably not sleep today.

Actually, moving to the split EGD server/client interface hopefully should also help me by providing more entropy for Raven (my frontend system) without using timer_entropyd, which has a nasty, completely load-linked dip in entropy generated when the load spikes. as Alex points out in the comments of the latter post, the quality of entropy also matters.

In parallel with the entropy issues in Yamato, I have entropy issues with my two vservers, one of which runs this blog, the other runs xine’s bugzilla — and both of which seem to have huge trouble with missing entropy, although in slightly different ways. The former, Vanguard, never reached 1KiB of entropy during the last four days when I monitored it, and its daily range is usually below 300 bytes. The latter, Midas, has a spike in entropy availability during the night, when maintenance tasks run (mostly Bugzilla’s); I’m not sure if it’s because Apache then throttles HTTPS access or something else, but otherwise it also has the same range of entropy.

Since both Vanguard and Midas have an average amount of accesses over HTTPS, it made sense to me to consider this lack of entropy as a problem. Unfortunately running timer_entropyd there is not an option: the virtualisation software used does not let me inject further entropy into the system, which is quite a pain in the lower back side. When I first inspected Entropy Broker, I was thrilled to read that OpenSSL supports accessing EGD sockets directly rather than through the kernel-exposed /dev/yrandom device, since the EGD sockets are handled entirely in userland, and thus wouldn’t hit the limitation of the virtualised envrionment. With Entropy Broker being.. broken, I was still able to track down an alternative to that: PRNGD a pseudo-random number generation daemon that works entirely in user-space by using a number of software sources to provide fast and decent random number sources.

OpenSSL, OpenSSH and other software explicitly provide support for accessing EGD sockets, and explicitly refer to PRNGD as an alternative to the OS-provided interfaces. Unfortunately, neither would work out of the box for Gentoo — the former will look into EGD sockets only if the /dev/urandom and /dev/random devices wouldn’t have enough data to provide enough random bytes as requested (which is never going to happen with /dev/urandom), and has to have its sources’ edited to actually have a chance to use EGD, even so it would (rightly) privilege the /dev/random device; OpenSSH instead requires for an extra parameter to be passed to the ./configure call. For those interested, the ekey-egd-linux package can inject this data back into the kernel, like Entropy Broker would have done if it worked.

A tentative ebuild for prngd is in my overlay but I’m not sure if and when I’ll add it to the main tree. First of all, I haven’t compared the entropy’s quality yet, and it might well be that the PRNGD method is going to give very bad results, but I’m also quite concerned about the sources’ license: it seems to be a mess of as-is, MIT and other licenses altogether. While Gentoo is not as strict as Debian in the license handling, it doesn’t sound like something I’d love to keep around with my name tagged on it.

Sigh, 7am, I’ll probably be awakened in two hours again. I really need to get myself some kind of vacation, somewhere I can spend time reading and/or listening to music all day long for a week. Sigh. I wish!

Entropy Broken

In my previous post I noted the presence of Entropy Broker — software designed to gather entropy from a number of sources, and then concentrate it to be sent to a number of consumers. My main interest in this was to make use of the EGD interface to feed new entropy to OpenSSL so that it wouldn’t deplete the little one available on my two vservers — where I cannot simply push it via timer_entropyd.

Unfortunately it turned out not to be as simple as building it and writing init scripts for it. The software seems abandoned, with the last release back in 2009, but most importantly it doesn’t work.

I have already hinted in the other post that the website lies about the language the software is written in. In particular, the package declares itself as being written in C, when instead it is composed of a number of C++ source files, and uses the C++ compiler to build. Looking at the code, all of it is written in C style; a quick glance to the compiled results shows that nothing is used from the STL; the only two C++ language symbols that the compiled binaries rely on are the generic new and delete operators (_Znwm and _ZdlPv in mangled form). This alone spells bad.

After building, and setting up the basic services needed by Entropy Broker (the eb hub, server_timers ­­– that takes the place of timer_entropyd – and, in a virtual machine, client_linux_kernel), the results aren’t promising. The entropy pool is not replenished on the virtual machine, ever; network traffic is very very limited. The same more or less goes when using the EGD client (which is actually an EGD server acting as an eb-client). Even worse with the server_audio that seems to exit with error after reading a few data points. server_video doesn’t even build since it relies on V4L1 that has been dropped out of Linux 2.6.38 and later.

Returning a moment about my EntropyKey problems with the entropy not staying full, I’ve spoken with the EntropyKey developers briefly today. If those downward spikes happen, it usually is because something is consuming entropy faster than EntropyKey can replenish it, and since the EntropyKey can produce around 4KiB/s of entropy, that means a fast consumption of random data.

As an example, I was told that spawning a process eats 8 bytes of entropy, so something around 500 process spawned in a second would be enough to beat the Key’s ability to replenish the entropy. This might sound a lot but it really isn’t, especially when doing parallel builds, just think that a straight gcc invocation in Gentoo spawns about five processes (the gcc-config wrapper, gcc as the real frontend, cpp to preprocess, cc1 as the real compiler, and as which is the assembler), and that libtool definitely calls many more for handling inputs and outputs. And remember that the tinderbox builds with make -j12 whenever it can.

This seems to match the results I see from the Munin graphs, where entropy is depleted when load spikes, for instance when kicking off a build for ChromiumOS. But now I’m also wondering if the problem is that the ekeyd daemon gets a too low priority when trying to replenish it, which leaves it uncovered — I guess my next step is to add Munin monitoring for the ekeyd data as well as the entropy to see if I can link the two of them. Do note that the load on Yamato can easily reach 60 and over…

And a final word about timer_entropyd… a quick check seems to suggest that it only works correctly on systems that are mostly idle… my frontend system seems to be just fine in such a context, and indeed it does seem to do a good job there (load during the day never reached 1). It doesn’t seem to be a good idea for Yamato with its high load.

What’s up with entropy? (lower-case)

You might remember me writing about the EntropyKey — a relatively cheap hardware device that is designed to help the kernel generating random numbers sold by SimTec for which I maintain the main software package (app-crypt/ekeyd).

While my personal crypto knowledge is pretty limited, I have noticed before how bad a depleted entropy can hit ssh sessions and other secure protocols that rely on good random numbers, which is why I got interested in the EntropyKey when I read about it, and ordered it as soon as it was possible, especially as I decided to set up a Gentoo-based x86 router for my home/office network, which would lack most of the usual entropy-gathering sources (keyboard, mouse, disk seeks).

But that’s not been all I cared about, with entropy, anyway: thanks to Pavel, Gentoo gathered support for timer_entropyd as well, which uses the system timers to provide further entropy for the pool. This doesn’t require any special software to work (contrarily to the audio_entropyd and video_entropyd software), even though it does make use of slightly more CPU. This makes it a very good choice for those situations where you don’t want to care about further devices, including the EntropyKey.

Although I wonder if it wouldn’t have made more sense to have a single entropyd software and modules for ALSA, V4L and timer sources. Speaking about which, video_entropyd needs a bit of love, since it fails to build with recent kernel headers because of the V4L1 removal.

Over an year after having started using the EntropyKey, I’m pretty satisfied, although I have a couple of doubts about it, which are mostly related to the software side of the equation.

The first problem seems to be mixed between hardware and software: both on my router and on Yamato when I reboot the system, the key is not properly re-initialised, leaving it in “Unknown” state until I unplug and re-plug it; I haven’t found a way to do this entirely in software, which, I’ll be honest, sucks. Not only it means that I have to have physical access to the system after it reboots, but the router is supposed to work out of the box without me having to do anything, and that doesn’t seem to happen with the EntropyKey.

I can take care of that as long as I live in the same house where the keys are, but that’s not feasible for the systems that are at my customers’. And since neither my home’s gateway (for now) nor (even more) my customers’ are running VPNs or other secure connections requiring huge entropy pools, timer_entropyd seems a more friendly solution indeed — even on an Atom box the amount of CPU that it consumes is negligible. It might be more of an issue once I find a half-decent solution to keep all the backups’ content encrypted.

Another problem relates to interfacing the key with Linux itself; while the kernel’s entropy access works just fine for sending more entropy to the pool, the key needs to be driven by an userland daemon (ekeyd); that daemon can drive the key in either of two options: one is by using the CDC-ACM interface (USB serial interface, used by old-school modems and cellphones), and the other is by relying on a separate userland component using libusb. The reason for two options to exist is not that it provides different advantages, as much as neither does always work correctly.

For instance the CDC-ACM option depends vastly on the version of Linux kernel that one is running, as can be seen by the compatibility table that is on the Key’s website. On the other hand, the userland method seems to be failing on my system, as the userland process seems to be dying from time to time. To be fair, the userland USB method is considered a nasty workaround and is not recommended; even a quick test here graphing entropy data shows that moving from the libusb method to CDC-ACM provides a faster replenishing of the poll.

Thanks to Jeremy I’ve now started using Munin and keeping an eye on the entropy available on the boxes I admin, which includes Yamato itself, the router, the two vservers, the customer’s boxes but, most importantly here, the frontend system I’ve been using daily. The reason why I consider it important to graph the entropy on Raven is that it doesn’t have a Key, and it’s using timer_entropyd instead. On the other hand, it is also a desktop system, and thanks to the “KB SSL” extension for Chrome, most of my network connection go through HTTPS, which should be depleting the entropy quite often. Indeed, what I want to do, once I have a full day graph of Raven’s entropy without interruptions, is comparing that with the sawtooth pattern that you can find on Simtec’s website.

For those wondering why I said “without interruptions”. — I have no intention to let Munin node data be available to any connection, and I’ve not yet looked into binding it up with TLS and verification of certificates, so for now the two vservers are accessed only through SSH port forwarding (and scponly). Also, it looks like Munin doesn’t support IPv6, and my long time customer is behind a double-NAT, with the only connection point I have is creating an SSH connection and forward ports with that. Unfortunately, this method relies on my previously noted tricks which leaves processes running in the background when connection to a box needs to be re-established (for instance because routing to my vservers went away or because the IPv6 address of my customer changed). When that happens, fcron considered the script still running and refuses to start a new job, since I didn’t set the exesev option to allow starting multiple parallel jobs.

And to finish it off, I found a link by looking around about Entropy Broker a software package designed to do more or less what I said above (handle multiple sources of entropy) and distribute them to multiple clients, with a secure transport whereas the Simtec-provided ekey-egd application allows sending data over an already secured channel. I’ll probably look into it, for my local network but more importantly for the vservers, which are very low on entropy, and for which I cannot use timer_entropyd (I don’t have privileges to add data to the pool). As it happens, it OpenSSL, starting version 0.9.7, tries accessing data through the EGD protocol before asking it to the kernel, and that library is the main consumer of entropy for my servers.

Unfortunately packaging it doesn’t look too easy, and thus will probably be delayed for the next few days, especially since my daily job is eating away a number of hours; Plus the package neither use standard build systems, lies about the language it is written in (“EntropyBroker is written in C”, when it’s written in C++), but most importantly, there are a number of daemon services for which init scripts need to be written…

At any rate, that’s a topic for a different post once I can get it to work.

Hardware signatures

If you read Planet Debian as well as this blog, you probably have noticed the number of Debian developers that changed their keys recently, after the shadows cast over the SHA-1 hash algorithm. It is debatable on whether this is an issue now or not, but that’s not what I want to discuss.

There are quite a few reasons why Debian developers are more interested in this than Gentoo developers; while we also sign manifests, there are quite a few things that don’t work that well in our security infrastructure, which we should probably pay more attention to (but I don’t want to digress now), so I don’t blame their consideration of tighter security.

I’m also considering the switch; while I have my key for quite a while, there are a few issues with it: it’s not signed by any Gentoo developer (I actually don’t think I have met anybody in person to be able to exchange documents and stuff), the Manifest signing key is not a subkey of my actual primary key (which by the way contains lots of data of my previous “personas” that don’t matter any longer), and so on so forth. Revoking this all and starting anew might be a good solution.

But, before proceeding, I want finally go get over with the thing and move to hardware cryptography if possible; I already expressed the interest before, but I never dug enough to find the important information, now I’m looking for that kind of information. And I want a solution that works in the broadest extension of cases:

  • I want it to work without SHA-1; I guess this starts already to be difficult; while it’s not clear whether SHA-1 is weak enough to be a vulnerability or not, being able to ignore the issue by using a different algorithm is certainly a desirable feature;
  • I want it to work with GnuPG and OpenSSH at least; if there is a way to get it to work with S/MIME it might also be a good idea;
  • I want it to work on both Linux and Mac OS X: I have two computers in my office: Yamato running Gentoo and Merrimac running OSX; I have to use both, and can’t do without either; I don’t care if I don’t have GnuPG working on OSX, I still need it to work with OpenSSH, since I would like to use it for remote access to my boxes;
  • as an extension to the previous point, I guess it has to be USB; not only I can switch it between the two systems (hopefully!), I’m also going to get a USB switch to use a single console between the two;

I guess the obvious solution would be a tabletop smartcard reader with one or more cards (and I could get my ID card to be a smartcard), but there is one extra point: one day I’m going to have a laptop again, what then? I was thinking about all-in-one tokens, but I have even less knowledge about those than I have about smartcards.

Can anybody suggest me a solution? I understand that the FSFE card only supports 1024 bit for the keys, which seems to be tied to weakness lately, no idea how much of that is true though, to be honest.

So, suggestions, very welcome!