Does your webapp really need network access?

One of the interesting thing that I noticed after shellshock was the amount of probes for vulnerabilities that counted on webapp users to have direct network access. Not only ping to known addresses to just verify the vulnerability, or wget or curl with unique IDs, but even very rough nc or even /dev/tcp connections to give remote shells. The fact that probes are there makes it logical to me to expect that for at least some of the systems these actually worked.

The reason why this piqued my interest is because I realized that most people don’t do the one obvious step to mitigate this kind of problems by removing (or at least limiting) the access to the network of their web apps. So I decided it might be a worth idea to describe a moment why you should think of that. This is in part because I found out last year at LISA that not all sysadmins have enough training in development to immediately pick up how things work, and in part because I know that even if you’re a programmer it might be counterintuitive for you to think that web apps should not have access, well, to the web.

Indeed, if you think of your app in the abstract, it has to have access to the network to serve the response to the users, right? But what happens generally is that you have some division between the web server and the app itself. People who have looked into Java in the early nougthies probably have heard of the term Application Server, which usually is present in form of Apache Tomcat or IBM WebSphere, but here is essentially the same “actor” for Rails app in the form of Passenger, or for PHP with the php-fpm service. These “servers” are effectively self-contained environments for your app, that talk with the web server to receive user requests and serve them responses. This essentially mean that in the basic web interaction, there is no network access needed for the application service.

Things gets a bit more complicated in the Web 2.0 era though: OAuth2 requires your web app to talk, from the backend, with the authentication or data providers. Similarly even my blog needs to talk with some services, to either ping them to tell them that a new post is out, and to check with Akismet for blog comments that might or might not be spam. WordPress plugins that create thumbnails are known to exist and to have a bad history of security and they fetch external content, such as videos from YouTube and Vimeo, or images from Flickr and other hosting websites to process. So there is a good amount of network connectivity needed for web apps too. Which means that rather than just isolating apps from the network, what you need to implement is some sort of filter.

Now, there are plenty of ways to remove access to the network from your webapp: SElinux, GrSec RBAC, AppArmor, … but if you don’t want to set up a complex security system, you can do the trick even with the bare minimum of the Linux kernel, iptables and CONFIG_NETFILTER_XT_MATCH_OWNER. Essentially what this allows you to do is to match (and thus filter) connections based of the originating (or destination) user. This of course only works if you can isolate your webapps on a separate user, which is definitely what you should do, but not necessarily what people are doing. Especially with things like mod_perl or mod_php, separating webapps in users is difficult – they run in-process with the webserver, and negate the split with the application server – but at least php-fpm and Passenger allow for that quite easily. Running as separate users, by the way, has many more advantages than just network filtering, so start doing that now, no matter what.

Now depending on what webapp you have in front of you, you have different ways to achieve a near-perfect setup. In my case I have a few different applications running across my servers. My blog, a WordPress blog of a customer, phpMyAdmin for that database, and finally a webapp for an old customer which is essentially an ERP. These have different requirements so I’ll start from the one that has the lowest.

The ERP app was designed to be as simple as possible: it’s a basic Rails app that uses PostgreSQL to store data. The authentication is done by Apache via HTTP Basic Auth over HTTPS (no plaintext), so there is no OAuth2 or other backend interaction. The only expected connection is to the PostgreSQL server. Pretty similar the requirements for phpMyAdmin: it only has to interface with Apache and with the MySQL service it administers, and the authentication is also done on the HTTP side (also encrypted). For both these apps, your network policy is quite obvious: negate any outside connectivity. This becomes a matter of iptables -A OUTPUT -o eth0 -m owner --uid-owner phpmyadmin -j REJECT — and the same for the other user.

The situation for the other two apps is a bit more complex: my blog wants to at least announce that there are new blog posts, and it needs to reach Akismet; both actions use HTTP and HTTPS. WordPress is a bit more complex because I don’t have much control over it (it has a dedicated server, so I don’t have to care), but I assume it mostly is also HTTP and HTTPS. The obvious idea would be to allow ports 80, 443 and 53 (for resolution). But you can do something better. You can put a proxy on your localhost, and force the webapp to go through it, either as a transparent proxy or by using the environment variable http_proxy to convince the webapp to never connect directly to the web. Unfortunately that is not straight forward to implement as neither Passenger not php-fpm has a clean way to pass environment variables per users.

What I’ve done is for now is to hack the environment.rb file to set ENV['http_proxy'] = '' so that Ruby will at least respect it. I’m still out for a solution for PHP unfortunately. In the case of Typo, this actually showed me two things I did not know: when looking at the admin dashboard, it’ll make two main HTTP calls: one to Google Blog Search – which was shut down back in May – and one to Typo’s version file — which is now a 404 page since the move to the Publify name. I’ll be soon shutting down both implementations since I really don’t need it. Indeed the Publify development still seems to go toward the “let’s add all possible new features that other blogging sites have” without considering the actual scalability of the platform. I don’t expect me to go back to it any time soon.

The security snakeoil

I said I’m basically doing the job of a network and system administrator here in LA — although you could say I have been doing that for the past four years as well, with the main difference that in the past I would have only a few Unix systems to administer, and mostly I had to deal with way too many Windows boxes.

Anyway the interesting part of doing this job is having to deal with the strange concepts of security that customers (and previous administrators) have of security. I won’t enter into much details, but I’ll talk about a few of the common snakeoil that is being sold for security.

First let’s start with the mighty DROP — it seems to be common choice for most network administrators to set up their systems to DROP all the packets that are not to be delivered to their end destination. Why is this done, is up to debate; some insist that it’s to not let known to a scanner that there even is a host there — which only makes sense if there is no service at all on the system that needs to be accessible, in which case we’re talking about different issues. Another alternative story is that it makes it more difficult to discern between services that are available and those who aren’t — Peter Benie analyzed this better than I can do here.

So the only effective reason to use DROP is to reduce the amount of packets to process and send back during a DDoS — which might be something. On the other hand, if there is even one service that is open, they’ll just exploit that one for the DDoS, not multiple ports. On the other hand, using the DROP rule makes it harder to diagnose network problems for other administrators that are just trying to do their job. What is my solution? Rate-limit: you’ll start dropping packets after the host starts actually trying to flood you, this way the usual diagnostics just work correctly.

Then there is the idea that you can solve everything by putting it behind a firewall and a VPN. Somehow there is this silly idea floating around that the moment when you add a firewall in front of your cabinet, and then use a VPN to tunnel from your office, everything is, in one move, capable of being either trusted or not trusted. Bonus points when the firewall’s only task is to avoid “netscans” and otherwise just do port whitelisting for a bunch of Linux servers.

The problem starts appearing the moment you’re expecting to be able over 100Mbps … and you end up instead being bottle-necked by a 40Mbps firewall which is not even using packet inspection! And the snakeoil here is expecting that there is a huge difference between a server with only one whitelisted port on the firewall and the same server with the same port whitelisted by iptables — no there isn’t a policy on the outgoing connections, so don’t ask. Of course anything behind the VPN is then handled like a completely trusted network, with shallow password, if any at all, and no second-factor authentication.

Finally there is what I all hyper-security — the same kind of thing that enthuses OpenBSD users: lock everything down as tight as you might. “Hey it’s not difficult, you just spend time once!” I heard it so many times by now… Nobody is saying that it’s a good idea to ignore security issues and known vulnerabilities, but most of the time you have to come up with a good mediation between security and usability. If your security is so strong that you can’t do anything useful, then it’s bad security.

It’s not just about what you can do with the system — especially for servers it’s pretty trivial to lock it down to exactly just what you need and nothing more, so I can understand one willing to do so. The problem is that unless you can afford to re-validate all the checks every time you have to add something new, the chances are that with maintenance, you can easily slip up and forget about something, either breaking the services you run, or breaching security altogether.

Here’s the main catch: if it’s a system that only has one user, and will die with you, tightening it down is easy and can work, if you’re not an idiot — but if you feel smug because others accept compromises for the sake of not being a human single point of failure, stop it because you’re just the kind of scum that causes so many people to say “pfft” to security. You can actually have two very easy examples for it in Windows Vista and Fedora.

The former started asking you to confirm every single operation, to the point that users ended up click “Ok” without reading what it asked

  • Do you want to install the drivers for the printer you just bought? Ok.
  • Do you want to install the operating system updates? Ok.
  • Do you want to install Firefox? Ok.
  • Do you want to upgrade Firefox? Ok.
  • Do you want to install Microsoft Office? Ok.
  • Do you want to update Microsoft Office? Ok.
  • Do you want to install Adobe Reader? Ok.
  • Do you want to install this little game you just downloaded? Ok.
  • Do you want to install this driver that sniffs your network traffic? Ok.

I’ve seen malware actually requiring a confirmation, easy to actually notice if you look at the window, but usually just discarded by clicking “Ok” without looking.

For what concerns Fedora instead — you probably are familiar with the decision on RedHat’s part of enabling SELinux by default on their installs, since a very long time ago. And I tend to agree that it’s a good thing; the problem is that most people don’t understand how to work with SELinux, and lacking simple ways to do what they need to do in simple ways, they just decide to turn SELinux off altogether — this is the same problem with Nagios plugins and sudo where the documentation just tells you to give them sudo access to everything.

I could probably write a lot more about this kind of situations but I don’t think it’s worth my time. I just think that people insisting on hyper-security are detrimental to the overall security of the population, as they scare people away.