I said I’m basically doing the job of a network and system administrator here in LA — although you could say I have been doing that for the past four years as well, with the main difference that in the past I would have only a few Unix systems to administer, and mostly I had to deal with way too many Windows boxes.
Anyway the interesting part of doing this job is having to deal with the strange concepts of security that customers (and previous administrators) have of security. I won’t enter into much details, but I’ll talk about a few of the common snakeoil that is being sold for security.
First let’s start with the mighty DROP — it seems to be common choice for most network administrators to set up their systems to DROP all the packets that are not to be delivered to their end destination. Why is this done, is up to debate; some insist that it’s to not let known to a scanner that there even is a host there — which only makes sense if there is no service at all on the system that needs to be accessible, in which case we’re talking about different issues. Another alternative story is that it makes it more difficult to discern between services that are available and those who aren’t — Peter Benie analyzed this better than I can do here.
So the only effective reason to use DROP is to reduce the amount of packets to process and send back during a DDoS — which might be something. On the other hand, if there is even one service that is open, they’ll just exploit that one for the DDoS, not multiple ports. On the other hand, using the DROP rule makes it harder to diagnose network problems for other administrators that are just trying to do their job. What is my solution? Rate-limit: you’ll start dropping packets after the host starts actually trying to flood you, this way the usual diagnostics just work correctly.
Then there is the idea that you can solve everything by putting it behind a firewall and a VPN. Somehow there is this silly idea floating around that the moment when you add a firewall in front of your cabinet, and then use a VPN to tunnel from your office, everything is, in one move, capable of being either trusted or not trusted. Bonus points when the firewall’s only task is to avoid “netscans” and otherwise just do port whitelisting for a bunch of Linux servers.
The problem starts appearing the moment you’re expecting to be able over 100Mbps … and you end up instead being bottle-necked by a 40Mbps firewall which is not even using packet inspection! And the snakeoil here is expecting that there is a huge difference between a server with only one whitelisted port on the firewall and the same server with the same port whitelisted by iptables — no there isn’t a policy on the outgoing connections, so don’t ask. Of course anything behind the VPN is then handled like a completely trusted network, with shallow password, if any at all, and no second-factor authentication.
Finally there is what I all hyper-security — the same kind of thing that enthuses OpenBSD users: lock everything down as tight as you might. “Hey it’s not difficult, you just spend time once!” I heard it so many times by now… Nobody is saying that it’s a good idea to ignore security issues and known vulnerabilities, but most of the time you have to come up with a good mediation between security and usability. If your security is so strong that you can’t do anything useful, then it’s bad security.
It’s not just about what you can do with the system — especially for servers it’s pretty trivial to lock it down to exactly just what you need and nothing more, so I can understand one willing to do so. The problem is that unless you can afford to re-validate all the checks every time you have to add something new, the chances are that with maintenance, you can easily slip up and forget about something, either breaking the services you run, or breaching security altogether.
Here’s the main catch: if it’s a system that only has one user, and will die with you, tightening it down is easy and can work, if you’re not an idiot — but if you feel smug because others accept compromises for the sake of not being a human single point of failure, stop it because you’re just the kind of scum that causes so many people to say “pfft” to security. You can actually have two very easy examples for it in Windows Vista and Fedora.
The former started asking you to confirm every single operation, to the point that users ended up click “Ok” without reading what it asked
- Do you want to install the drivers for the printer you just bought? Ok.
- Do you want to install the operating system updates? Ok.
- Do you want to install Firefox? Ok.
- Do you want to upgrade Firefox? Ok.
- Do you want to install Microsoft Office? Ok.
- Do you want to update Microsoft Office? Ok.
- Do you want to install Adobe Reader? Ok.
- Do you want to install this little game you just downloaded? Ok.
- Do you want to install this driver that sniffs your network traffic? Ok.
I’ve seen malware actually requiring a confirmation, easy to actually notice if you look at the window, but usually just discarded by clicking “Ok” without looking.
For what concerns Fedora instead — you probably are familiar with the decision on RedHat’s part of enabling SELinux by default on their installs, since a very long time ago. And I tend to agree that it’s a good thing; the problem is that most people don’t understand how to work with SELinux, and lacking simple ways to do what they need to do in simple ways, they just decide to turn SELinux off altogether — this is the same problem with Nagios plugins and sudo where the documentation just tells you to give them sudo access to everything.
I could probably write a lot more about this kind of situations but I don’t think it’s worth my time. I just think that people insisting on hyper-security are detrimental to the overall security of the population, as they scare people away.
Well-designed setups always are a well-informed trade-off between convenience and security. If you have to do two-factor-authentication for every single thing done with sudo, people will start to keep a UID0 shell around so they don’t have to do it so often. Passwords become simple and are used between different sites because it’s hard to remember them individually etc.As for the first note on the almighty DROP: I’ve since forever used rejection chains, that are called lrd (log, reject, drop) and rd (reject, drop). They do rate-limited (logging and) rejection, then switch to just dropping once you go over the (very generous) ratelimit. This also gives you nice counters so you can monitor whether drop-rates are suddenly exploding.
> Finally there is what I call hyper-security — the same kind of thing that enthuses> OpenBSD users: lock everything down as tight as you might.OpenBSD doesn’t provide means to lock everything down. There’s no MAC/RBAC (systrace is flawed and unsupported), no POSIX capabilities (at all) or anything similar, setuid/gid executables are not restricted to special groups, no POSIX ACL, no syslog message filtering by sender’s uid/gid, no restrictions like PaX MPROTECT, no ASLR bruteforce deterring measures, etc. OpenBSD is about correctness, simplicity, consistency and *transparent* security that is *enabled by default*. Besides, not long ago their package manager didn’t even perform authenticity checks for the packages retrieved over plain HTTP/FTP – isn’t that not quite what one could expect from people who supposedly want to lock everything down as tight as they might? Please, check your facts before talking about something you probably didn’t even use, especially if you overgeneralize people just because they use the same operating system.> “Hey it’s not difficult, you just spend time once!” I heard it so many times by now…It’s very hard to talk with people who speak about paranoia, hyper-security and inconvenience caused by proactive security enhancements and use reductio ad absurdum, argumentum ad hominem, straw men, postulate false assumptions contrary to the modern best practices and don’t even use any logical or empirical evidence to prove their point.Let me elaborate (boring mode on). We have a strong empirical evidence that: 1. glibc may contain vulnerable code that runs before a setuid/setgid application have any opportunity to drop any privileges.2. Gentoo may not fix such vulnerabilities for more than two weeks for arguable reasons like waiting for the final upstream fix.3. Exploitation of some of the vulnerabilities may not be stopped or mitigated by the means of PaX, Grsecurity (excluding RBAC) or any security enhancement provided by Hardened Gentoo.4. Exploitation of some of these vulnerabilities was *trivial*.5. The impact of the vulnerabilities is unauthorized privilege escalation, up to the full root privileges.Additionally, Tavis Ormandy, the person who discovered the vulnerabilities, said (on IRC) that there still is some privileged code in glibc that looks like it may contain similar vulnerabilities:<taviso> my money is on LD_HWCAP_MASK breaking next, it’s just plain wrong.<taviso> Arach: if you’re sticking with the blacklist, LD_DEBUG, LD_ASSUME_KERNEL, LD_HWCAP_MASK, LD_BIND_NOW, LD_BIND_NOT sound like the scariestI conclude that the risks of full system compromise, service interruption, data loss and data leak are high, costly in case of the systems that actually contain something valuable to protect (and I am responsible for several such systems) and should be mitigated if it’s practically possible and unlikely to cause troubles (read: if it makes sense).What I do to mitigate the risks (and/or):1. Enforce *transparent* (permissive) RBAC policies to prevent exactly the unauthorized privilege escalation, but not to restrict regular activity in any interfering way (btw, this one is a very convenient and not interfering solution for binary distributions like Ubuntu).2. Deploy a patch for glibc that *transparently and completely* (if the LD_* variables are not whitelisted) avoids execution of the potentially vulnerable privileged code.3. Since it’s possible and makes sense in some of my use cases, remove setuid bit from every executable on the system. Make the best I can do to make sure there are no setgid executalbes that, in case of successful exploitation, could directly or indirectly allow dangerous privilege escalation. If there are practically irreplaceable executables that require to be setuid-root (read: sudo), allow them to be executed only by a limited group of trusted users. Btw, that’s what Openwall linux implements by default (read: snakeoil my @ss and good luck with ad hominem addressed to Solar Designer and co. ;). And it even prevents exploitation of vulnerabilities in the application logic (not just in some privileged parts of the runtime linker code).This eliminates the entire attack vector or at least makes the attack surface cardinally narrower. Is it practical and does it mitigate actual risks? Yes. Is it know and reviewed solution? Yes (spender and some other people on #grsecurity even reviewed the implementation). Is it interfering or caused any troubles? No. Was it expensive to implement or does it take a considerable effort to maintain? No.Now, please, tell me how is this a hyper-security? What things it may break and how, or what inconvenience it may cause?> Nobody is saying that it’s a good idea to ignore security issues and known vulnerabilities,Yeah. Some people are just doing it silently, while some other people are just ok with that. ;)> but most of the time you have to come up with a good mediation between security and usability.> If your security is so strong that you can’t do anything useful, then it’s bad security.Overgeneralization is so overgeneralized. Reductio ad absurdum FTW. Seriously, you just reject and call hyper-security (bad security) something that you don’t even care to understand, let alone to verify it actually doesn’t let you “do anything useful”.> It’s not just about what you can do with the system — especially for servers it’s> pretty trivial to lock it down to exactly just what you need and nothing more, soI hope I’ve explained above why sometimes it’s not necessary “to lock it down to exactly just what you need and nothing more” to mitigate some costly risks. You seem to be confused by MAC folks or alike who often don’t know better than locking everything down with SELinux, for example. But maybe if you start learning to understand things instead of rejecting them, you’ll know there are other options, including more lightweight, less intrusive and more effective ones.> I can understand one willing to do so. The problem is that unless you can afford> to re-validate all the checks every time you have to add something new, the> chances are that with maintenance, you can easily slip up and forget about> something, either breaking the services you run, or breaching security altogether.Maybe you seen people who don’t really understand Grsecurity (or PaX) or didn’t even try to use it. Then you know they think and say nonsense about it (probably repeat something they heard before), like how it must be difficult to use it on desktops or even at all, or that it is awfully unstable or takes so much effort to maintain. But you know that’s not true because you use and understand Grsecurity, right? Then maybe you should try a bit harder not to act like those people. ;)> Here’s the main catch: if it’s a system that only has one user, and will die with you,> tightening it down is easy and can work, if you’re not an idiot — but if you> feel smug because others accept compromises for the sake of not being a> human single point of failure, stop it because you’re just the kind of scum> that causes so many people to say “pfft” to security.What a brilliant main catch, thank you! 😉 Should I take that as I’m scum and one of the smug people because I grin and am being sarcastic sometimes when someone spreads – no doubt, very practical and useful – FUD about infosec? 😉 For the record: I act like that not because I feel smug, but because maybe I’d have to cry otherwise. There’s no other aspect of IT as fscked up and downplayed by “everyone” as infosec, and I happen to see it all the time, over and over again. Not that I care about a legion of fools being wrong on the Internet, but when someone as intelligent and capable as you spreads infosec FUD and diagnose paranoia remotely on the IRC channel where I am, commenting on what I said, I tend to disagree, to perceive that emotionally and take it personally. Sorry for the inconvenience, oh the righteous man, who runs away to resume the rant on his blog! ;)On a side note ;), I doubt that by choosing fcron (you know what I’m talking about) over existing full-blown alternatives like vixie cron you consciously accept a compromise (a little pun intended here 😉 either “for the sake of not being a human single point of failure” or any other reason. I bet you just don’t care to understand the risks. Besides, there are other cron daemons without setuid-user and setuid-root executables included, and choosing them doesn’t make one “a human single point of failure”. Don’t you agree? Btw, should I explain that compromising fcron’s setuid-user executables – sooner or later, but inevitably, unless the root user will never use fcron, or you purposefully did something – leads to a local root privilege escalation? And that’s a rhetoric question. ;)> You can actually have two very easy examples for it in Windows Vista and Fedora.> The former started asking you to confirm every single operation, to the point that> users ended up click “Ok” without reading what it askedYay! Let’s overgeneralize! Let’s say everything we don’t understand is probably as much annoying/ineffective/interfering in the end as this stuff, so we won’t give a sh!t! 😉 Maybe you don’t mean that. But you do use very particular extreme cases to prove a very (very-very) general point.(skipped the ranting about well-know SELinux deficiencies)> I could probably write a lot more about this kind of situations but I don’t think it’s> worth my time. I just think that people insisting on hyper-security are detrimental> to the overall security of the population, as they scare people away.You have no idea how much more *I* could write about *this* kind of situations. But I think I said enough for now. 😉 And I just think that people overgeneralizing and spreading FUD on info-security are detrimental to the overall security of the population, as they *don’t listen to empirical evidence and logic due to hardly measurable nature of infosec, recognize only their own experience and then, at some point of their career, fail to educate their younger colleagues and to keep other people’s valuable data safe and secure.*
For your information, 85% of this article was already written before you started behaving like an ass. Because you’re not the only ass out there. But don’t let that stop you from feeling under the spot, please.Also feel free to repeat to me what Tavis said, like I wasn’t one of the people who pushed for the fix sooner rather than later, like I didn’t (at the time) apply his patch locally as soon as I knew about it. Care to see who’s not understanding things out there?Real security does not lie on a handful of experts and that’s it, it’s a process and it requires people to actually follow it. And when you scare users away _exactly as you and your friends do_ the users don’t care. As Blackb|rd said, if you make it hard for users to use multiple (logged) sudo command, they’ll just use @sudo -i@ and be done with it, and then you have more trouble. If you really think that the problem with “infosec” (yet another term I honestly despise, because it’s 99% used just to feel cooler) is with people overgeneralizing, then I won’t even bother discuss with you. I maintain that the problem is the ivory tower approach of so many people who should know better.Let me repeat. *I maintain that the problem with security is that some professionals make it feel mystical and hard.* And not just because I feel this way, but because I have by now over six years of experience dealing with SMBs (which are by far the less secure targets you can find) who can’t even _start_ thinking about security because who they encountered in the past were people like you, who make them feel like security is so hard they can’t do anything at all, at which point they decide they don’t give a shit about security and just move on without considering it.But yes as I said feel free entitled to feel like you’re the messiah, it’s your choice.As for fcron — yes I choose it to avoid being a human single point of failure: especially when I see that whoever wrote the crontab for vixie before made a huge mess: the fcrontab syntax is much harder to screwup especially when you don’t want executions to overlap.I also told you that if you have an issue with it you’re welcome to open a bug, which you refused to do — I looked into it and I’m actually fixing the suidctl thing right now. Does it remove the need for setuid? Nope, it still uses that for PAM — which I’m pretty sure does not apply to all people out there, so yes, it would be possible to execute it without setuid root for the most part. For sure the ebuild is doing tons of stuff that’s much easier to do if you clean it up, from the legacy mess it is now, as I’m trying now.But again, you’re showing the behaviour of the kind of security person who cares not about helping as much as you care about sentencing. Keep it that way, and you probably will keep your status as messiah because nobody’s going to come near to security, of course.
> For your information, 85% of this article was already written before you> started behaving like an ass.Isn’t it convenient to entirely skip what I said and come up with more ad hominem?> Because you’re not the only ass out there. But don’t let that stop you> from feeling under the spot, please.Tell me more about what I feel, please.> Also feel free to repeat to me what Tavis said, like I wasn’t one of the> people who pushed for the fix sooner rather than later, like I didn’t (at> the time) apply his patch locally as soon as I knew about it.OK, I’ll repeat. He said it’s likely there are more vulnerabilities. What do you do about that in advance? Nothing. Instead you’re trying to insult and accuse me for what you did. Because it was you who called the simple, practical, known, reviewed and maintainable mitigations hyper-security.Let me repeat. *I maintain that you speak about things you didn’t care to understand and tell people that those things are mystical and hard (hyper-security), while in fact they are not.*> Care to see who’s not understanding things out there?Yes, enlighten me, please. Like I wasn’t one of the people who applied the same patch and proactively deployed RBAC policies (though not on every machine) that prevented the exploitation.> Real security does not lie on a handful of experts and that’s it,Like I said or acted like it does. Did I?> it’s a process and it requires people to actually follow it.Like I said or did otherwise. What’s interesting here: you don’t even follow (some of) the modern practices, calling them hyper-security. You don’t even care to understand them. Like there is some reasonable security, and everything above that is some trouble-making hyper-security.> And when you scare users away exactly as you and your friends do the users don’t care.I have some bad news for you. I don’t scare away my coworkers, my family and some people on IRC and mailing lists. Otherwise, why would someone thank me, follow my advises from time to time, understand and remember what I said and keep asking more security-related questions later?But wait, that just cannot be true. There shall remain no doubts, for I scared *The Diego* away! So don’t let that stop you from feeling under the spot, please.> As Blackb|rd said, if you make it hard for users to use multiple (logged) sudoYou know what’s the problem with your overgeneralizing lectures? I just don’t practice the things you’re talking about. In fact, I spoke against them. Care to read and verify some IRC logs, in case you don’t believe me? Just let me know. Or just keep overgeneralizing, if that’s your choice.> command, they’ll just use sudo -i and be done with it, and then you have more trouble.I think I would have more trouble the moment I allow sudo -i. ;)> If you really think that the problem with “infosec” (yet another term I honestly> despise, because it’s 99% used just to feel cooler)Now I’m the person who uses infosec to feel cooler? Care to suggest another word?> is with people overgeneralizing“Infosec” is having a huge set of problems. However, (hiphop mode on 😉 that fact doesn’t exclude from that set your mindset and the FUD you spread.> then I won’t even bother discuss with you.Is that your excuse now? For the record: you’re not discussing, you’re accusing and polemize (in quite a lame way though). Discussion was interrupted at the moment you decided to skip every technical point I expressed in my first comment and write a reply consisting of ad hominem and overgeneralizations.> I maintain that the problem is the ivory tower approach of so many people who> should know better.I’m not sure I get that “ivory tower” metaphor right.> Let me repeat. I maintain that the problem with security is that some professionals> make it feel mystical and hard.So turns out there is the problem with security! 🙂 Nice. It’s time for you to realize you’re among those professionals. I’m not sure about the mystical part, but the hard part – that’s for sure. That’s you, with all your hyper-security rhetoric and the plain FUD.But wait a minute. Could you, please, explain why do you think it’s *me* who is among them? What *exactly* I did or said?> And not just because I feel this way, but because I have by now over six> years of experience dealing with SMBs (which are by far the less secure> targets you can find) who can’t even start thinking about security because> who they encountered in the past were people like you, who make them> feel like security is so hard they can’t do anything at all, at which point they> decide they don’t give a shit about security and just move on without> considering it.Straw men is so straw. 😉 You don’t even know me and don’t understand my opinion, so much that you even reverse it. And obviously you have no idea what the people who “encountered” me feel and think about security, or what I told them and what I did.> But yes as I said feel free entitled to feel like you’re the messiah, it’s your choice.Are you trying to find yourself an excuse for ad hominem… by inroducing more ad hominem? Nice.> As for fcron — yes I choose it to avoid being a human single point of failure:So, not being a human single point of failure equals…> especially when I see that whoever wrote the crontab for vixie before made a> huge mess: the fcrontab syntax is much harder to screwup especially when you> don’t want executions to overlap….providing more convenience, either requested or not.But what if I want to prevent root from executing arbitrary code from any file as long as it’s owned by any other users? Should I disable TPE for the sake of abstract, unwanted convenience that wasn’t requested by anyone? I don’t think so. And unless someone ask me to install fcron for a reason, I’ll stick with more secure defaults. And this is the point where our opinions, or even mindsets, seem to differ very significantly.> I also told you that if you have an issue with it you’re welcome to open a bug,> which you refused to do — I looked into it and I’m actually fixing the suidctl> thing right now.Yes, I refused. Care to name any practical reason why I shouldn’t?> Does it remove the need for setuid? Nope, it still uses that for PAM — which> I’m pretty sure does not apply to all people out there, so yes, it would be possible> to execute it without setuid root for the most part.Care to make it run without setuid-user, which is even more important in this case (or else say goodbye to TPE)?> For sure the ebuild is doing tons of stuff that’s much easier to do if you clean it up,> from the legacy mess it is now, as I’m trying now.The problem is not with the ebuild, and you know it.> But again, you’re showing the behaviour of the kind of security person> who cares not about helping as much as you care about sentencing.Again? How about the python patches? How about the glibc patch? There’s no “again”. You just pretend that fixing the ebuild would fix the security flaws in fcron, and that I’m “showing the behavior” only because I refused to fix the ebuild that I think is useless. And you even knew it in the first place, because I told you about that on IRC.> Keep it that way, and you probably will keep your status as messiah> because nobody’s going to come near to security, of course.What way? What messiah? You see, I can’t read your mind, so I can’t know the important details of your imaginary world. You should tell me about them.
Not that it’s really possible to figure out much what you are talking about since at least I am missing about half the story (if you don’t care about anyone else reading you could just write a private mail and spare everyone else reading something they can’t really understand properly).But what you call> Isn’t it convenient to entirely skip what I said and come up with more ad hominem?is most likely a response to you accusing him of running “away to resume the rant on his blog!”
English talking assholes! 😉
> if you don’t care about anyone else reading you could justSome people might know what we’re talking about, since they were on IRC. Besides, I think I explained enough in my comments. Feel free to ask questions.> is most likely a response to you accusing him of running “away to> resume the rant on his blog!”That by far isn’t the only thing I said in my first comment. And since I expect from myself that in similar situation I would address the technical points instead of skipping them in favor of immature ad hominem, why shouldn’t I expect the same from Diego?
Arach: tl;dr