Another personal postmortem: Heartbleed

So I promised I would explain why it took me that long to get the Heartbleed problem sorted out. So here’s what was going on.

Early last month, the company I worked for in Los Angeles have been vacating their cabinet at the Los Angeles hosting facility where Excelsior was also hosted. While they have been offering to continue hosting the server, the fact that I could not remotely log into it with the KVM was making it quite difficult for me to update the kernel and to keep updating the LXC ebuild.

I decided to bite the bullet, and enquired for hosting at the same (new) facility of them, Hurricane Electric. While they have very cheap half-rack hosting, I needed to get a full cabinet to host Excelsior on. At $600/mo is not cheap, but I can (barely) afford it right now, the positive side of being an absolute inept socially, and possibly I’ll be able to share it with someone else pretty soon.

The bright side with using Hurricane is that I can rely on nigh-infinite public addresses (IPv6), which is handy for a server like Excelsior that runs basically a farm of virtual machines. The problem is that you don’t only need a server, you also need a switch and a VPN endpoint if you were to put a KVM in there. That’s what I did, I bought a Netgear switch, a Netgear VPN router, and installed the whole thing while I was in the Bay Area for a conference (Percona Live Conference if you’re curious).

Unfortunately, Heartbleed was announced in-between the server being shut down at the previous DC and it being fully operational in the new one — in particular it was still in Los Angeles, but turned down. How does that matter? Well, the vservers where this blog, the xine Bugzilla and other websites run are not powerful enough to build packages for Gentoo, so what I’ve been doing instead is building packages from Excelsior and uploading them to the vservers. This meant that to update OpenSSL, I needed Excelsior running.

Now to get Excelsior running, I spent a full weekend, having to go to the datacenter twice: I couldn’t get the router to work, and after some time my colleague who was kindly driving me there figured out that somehow the switch did not like for port 0 (or 1, depending how you count) on that switch to be set on a VLAN that is not the default, so connecting the router to any other port made it work as expected. I’m still not sure why that is the case.

After that, I was able to update OpenSSL — but the problem was getting a new set of SSL certificates for all the servers. You probably don’t remember my other postmortem, but when my blog’s certificate expired, I was also in the USA, and I had no access to my StartSSL credentials, as they were only on the other laptop. The good news was that I had the same laptop I used that time with me, and I was able to log in and generate new certificates. While at it, I replaced the per-host SNI ones with a wildcard one.

The problem was with the xine certificate: the Class 2 certificate was already issued with the previous user, which I had no access to still (because I never thought I would have needed it), which meant I could not request a revocation of the certificate. Not only StartSSL were able to revoke it for me anyway, but they also did so free of charge (again, kudos to them!).

What is the takeaway from all of this? Well, for sure I need a backup build host for urgent package rebuilds; I think I may rent a more powerful vserver for that somewhere. Also I need a better way to handle my StartSSL credentials: I had my Zenbook with me only because I planned to do the datacenter work. I think I’ll order one of their USB smartcard tokens and use that instead.

I also ordered another a USB SIM-sized card reader, to use with a new OpenPGP card, so expect me advertising a second GPG key (and if I remember this time, I’ll print some business card with the fingerprints). This should make it easier for me to access my servers even if I don’t have a trusted computer with me.

Finally, I need to set up PFS but to do that I need to update Apache to version 2.4 and last time that was a problem with mod_perl. With a bit of luck I can make sure it works now and I can update. There is also the Bugzilla on xine-project that needs to be updated to version 4, hopefully I can do that tonight or tomorrow.

Heartbleed and SuperGenPass

After an older post of mine a colleague pointed out SuperGenPass to generate different passwords for each service out there from a single master password and the domain name in use. The idea was interesting, especially since it’s all client-side, which sounded very appealing to me.

Unfortunately, it didn’t take long for me to figure out a few limitations in this approach; the most obvious one is of course Amazon: while nowadays the login page even for Audible is hosted at the domain, the localized stores still log in on, e.g.,, but with the same password. Sure it’s easy to fix this, but it’s still a bit of a pain to change every time.

Also, at least the Chrome extension I’m using, makes it difficult to use different passwords for different services hosted at the same domain. You have an option to enable or disable the subdomain removal, so if you disable it, you’ll get different passwords for and (unlikely to be what you want) while if you enable it, you’ll get the same password for and (which is not what I want). Yes you can fix it on a per-service basis, but it adds to the problem above.

The last bother in the daily usage of the extension, has been with special characters. SuperGenPass does not, by default, use any special characters, just letter (mixed case) and numbers. Which is perfectly fine, unless you have a website that stupidly insists on requiring you to use symbols as well, or that requires you to use (less stupidly) longer or (insanely stupidly) shorter passwords. You then have to remember.

All three of these complains mean that you have to remember some metadata in addition to the master password: whether you have to change the domain used, whether you’re using subdomain removal or not for that particular service, and whether you have to change the length, or add special characters. It partly defeats the purpose of having a fully stateless hashing password generator.

There is also one more problem that worried me much more: while it makes it so that a leak from a single website would leak your base password for everything else, it does not entirely make it impossible. While there’s no real way to tell that someone is using SuperGenPass, if you’re targeting a single individual, it’s not impossible to tell; in particular, you now know I’ve been using SGP for a while, so if a password for an account named Flameeyes gets leaked, and it looks like an SGP password, it’s a good assumption that it is. Then, all you need to do is guess the domains that could be used to generate the password (with and without subdomain removal), and start generating passwords until you get to the master password used to generate that particular site password. Now you just need to have an educated guess to the domain you’re going to try login as me, and you’re done. And this is with me assuming that there is no weakness in the SGP algorithm — crypto is honestly too hard for me.

And now there is heartbleed — go change all your passwords, starting from xine. But how do you change your passwords when you have them generated? You have to change your master password. And now you have to remember if you changed the password for a given service already or not. And what happens if one of the services you’re using has been compromised before, such as Comixology? Now you have three different master passwords, if not more, and you’re back to square one, like SGP was never used.

So with all this considered, I’ve decided to say goodbye to SGP — I still have a few services that have not been migrated – but not those that I’ve named here, I’m not a moron – but I’m migrating them as I got. There are plenty of things I forgot I registered to at some point or another that have been mailing me to change their password. I decided to start using LastPass. The reason was mostly that they do a safety check for heartbleed vulnerabilities before you set up your passwords with them. I was skeptical about them (and any other online password storage) for a long time, but at this point I’m not sure I have any better option. My use of sgeps is not scalable, as I found out for myself, and the lack of 2FA in most major services (PayPal, seriously?) makes me consider LastPass as the lesser evil for my safety.

Heartbleed and xine-project

This blog comes in way too late, I know, but there has been extenuating circumstances around my delay on clearing this through. First of all, yes, this blog and every other websites I maintain were vulnerable to Heartbleed. Yes they are now completely fixed: new OpenSSL first, new certificates after. For most of the certificates, though, there is no revocation issued, as they are issued through StartSSL which means that they are free to issue, and expensive to revoke. The exception to this has been the certificate used by xine’s bugzilla which was revoked, free of charge, by StartSSL (huge thanks to the StartSSL people!)

If you have an account on xine’s Bugzilla, please change your passwords NOW. If somebody knows a way to automatically reset all passwords that were not changed before a given date in Bugzilla, please let me know. Also, if somebody knows whether Bugzilla has decent support for (optional) 2FA, I’d also be interested.

More posts on the topic will follow, this is just an announcement.