Protecting yourself from R-U-Dead-Yet attacks on Apache

Do you remember the infamous “slowloris” attack over HTTP webservers? Well, it turns out there is a new variant of the same technique, that rather than making the server wait for headers to arrive, makes the server wait for POST data before processing; it’s difficult to explain exactly how that works, so I’ll leave it to the expert explanation from ModSecurity

Thankfully, since there was a lot of work done to cover up the slowloris attack, there are easy protections to be put in place, the first of which would be the use of mod_reqtimeout… unfortunately, it isn’t currently enabled by the Gentoo configuration of Apache – see bug #347227 – so the first step is to workaround this limitation. Until the Gentoo Apache team appears again, you can do so simply by making use of the per-package environment hack, sort of what I’ve described in my previous nasty trick spost a few months ago.

# to be created as /etc/portage/env/www-servers/apache

export EXTRA_ECONF="${EXTRA_ECONF} --enable-reqtimeout=static"

*Do note that here I’m building it statically; this is because I’d suggest everybody to build all the modules as static; the overhead of having them as plugins is usually quite higher than what you’d have for loading a module that you don’t care about.*

Now that you got this set up, you should ensure to set a timeout for the requests; the mod_reqtimeout documentation is quite brief, but shows a number of possible configurations. I’d say that in most cases, what you want is simply the one shown in the ModSecurity examples. Please note that they made a mistake there, it’s not RequestReadyTimeout but RequestReadTimeout.

Additionally, when using ModSecurity you can stop the attack on its track after a few requests timed out, by blacklisting the IP and dropping its connections, allowing slots to be freed for other requests to arrive; this can be easily configured through this snipped, taken directly from the above-linked post:

RequestReadTimeout body=30

SecRule RESPONSE_STATUS "@streq 408" "phase:5,t:none,nolog,pass, setvar:ip.slow_dos_counter=+1,expirevar:ip.slow_dos_counter=60"
SecRule IP:SLOW_DOS_COUNTER "@gt 5" "phase:1,t:none,log,drop, msg:'Client Connection Dropped due to high # of slow DoS alerts'"

This should let you cover yourself up quite nicely, at least if you’re using hardened, with grsecurity enforcing per-user limits. But if you’re using hosting where you don’t have decision over the kernel – as I do – there is one further problem: the init script for apache does not respect the system limits at all — see bug #347301 .

The problem here is that when Apache is started during the standard system init, there are no limits set for the session is running from, and since it doesn’t use start-stop-daemon to launch the apache process itself, no limits are applied at all. This results in a quite easy DoS over the whole host as it will easily exhaust the system’s memory.

As I posted on the bug, there is a quick and dirty way to fix the situation by editing the init script itself, and change the way Apache is started up:

# Replace the following:
        ${APACHE2} ${APACHE2_OPTS} -k start

# With this

        start-stop-daemon --start --pidfile "${PIDFILE}" ${APACHE2} -- ${APACHE2_OPTS} -k start

This way at least the system generic limits are applied properly. Though, please note that start-stop-daemon limitations will not allow you to set per-user limits this way.

On a different note, I’d like to spend a few words on telling why this particular vulnerability is interesting to me: this attack relies on long-winded POST requests that might have a very low bandwidth, because just a few bytes are sent before the timeout is hit… it is not unlike the RTSP-in-HTTP tunnelling that I have designed and documented in feng during the past years.

This also means that application-level firewalls will start sooner or later filtering these long-winded requests, and that will likely put the final nail on the coffin of the RTSP-in-HTTP tunnelling. I guess it’s definitely time for feng to move on and implement real HTTP-based pseudo-streaming instead.

Even little differences count

If you ever find yourself relying for any reason on the behaviour of Microsoft Excel, make sure you never ever change its version. Seems like I had some problems with my job because the version of Microsoft Excel I bought is not the same as the one they are using at the office.

Also, if you want to shut up Valgrind’s warnings to make life easier for people developing against a given library, it’s time you hear about suppression files, rather than trying to change code you don’t know enough about.

Seems like a little difference in the patches applied by Debian (and Ubuntu) made OpenSSL vulnerable like it never was. Even my keys has to be considered compromised, so I had to change them on every service I use. This means Alioth, Gentoo’s Infra, GentooExperimental, SourceForge, Berlios, Savannah, Rubyforge, Launchpad (don’t ask), KDE, and of course my own server.

I think yesterday will be remembered for the years to come as a critical day. It’s like a city of the size of Milan starting to change the door locks for all the apartments at once. I’m surprised there hasn’t been more confusion.

Myself, I’m starting to consider some other kind of authentication, as the passphrases I use for the keys are quite long, and I’m tired of having to key it in every time I start the system. SmartCard and authentication tokens start to look quite nice, and it would be a nice test for Gentoo’s PAM infrastructure.

It’s unfortunate that Alon, who as far as I remembered managed these things, left Gentoo yesterday :/ Although I suppose that if I am to start looking into these things I might be able to lend a hand.

At the moment, I’m oriented toward an authentication token, as that would make it possible for me to have it around on the laptop too at any time. Strangely enough, I found an Italian producer, Eutronsec, and their CryptoIdentity token seems to be supported by pcsclite.

SmartCards are more likely useful in an organisation where you’d have one reader on a given system and many many cards around. Here I’d just have one reader and one smartcard if I did go that way. If the Italian ID card was a SmartCard I could have chosen that, but at the moment it doesn’t look like a convenient idea.

Unfortunately it seems like Eutronsec only sells to other companies, I’ve mailed them asking if they could sell to single entities, but I haven’t received an answer yet. I’m afraid I’ll have to start looking for alternatives if I want to proceed on this road.

On a totally different topic, but still related to “little differences”, I’m having difficulties finding cups that I can use with my espresso machine. Usual teacups that my family uses for coffee (yeah they are quite larger for coffee, we’re used to those though, a six servings Moka is good for three here) tend to be too thin at the bottom and large at the top, so the coffee spills a lot, especially when doing a hot boiling espresso. Mugs are quite better for the task, but all the ones I have here are little less than 9.5 centimetres (little more than three and a half inches), and require to be skewed to enter the espresso machine.

The idea would be an 8.5cm cup, but I can’t seem to find them in the shops around here, they are either 6 centimetres, which is way too short, or they are the “usual” mug size. In a Chinese shop around here I found some nice cups that are about 12cm actually, I was tempted to take one to hold the milk for cappuccino, actually, but they are quite too high for the espresso machine.