Thoughts on Amazon EC2 as of November 2010

I have written a bit about Amazon’s Elastic Computer Cloud before; mostly in bad even though some things are interesting and at least somewhat cool. Well, here is another post on the subject, that I’m using to clear my own ideas as to how to write a proper, objective and respectful guide to Gentoo on EC2. Besides, some of the roblems I faced with EC2 are giving me insights to improve Gentoo as a whole.

Metadata fetching: to pass details such as the hostname to use for the system or the SSH keys to authenticate against, Amazon provides the metadata through a webserver listening to an address in the APIPA range. At first I thought it was cumbersome because I expected it to be unable to talk outside of its range, and there is currently no way to have APIPA addresses assigned to the interfaces; luckily instead it works just as fine with the internal IPs that are assigned by the DHCP server, which means I can easily fetch the data after the interface is provided a real address.

To handle metadata fetching I have ready an init script that would just have to be executed, that will take care of changing /etc/conf.d/hostname to set the hostname properly, and creating an authorized_keys file for root to allow logging in on the system. It sounds easy but the problem here is that it has to run before hostname, but after network is brought up. Turns out that this is not possible with the OpenRC 0.6.3 version because of dependencies between the sysctl and network services, and hostname itself. This is fixed in the OpenRC GIT repository and will be has been released as part of 0.6.4.

Modules, keyboards, consoles: unfortunately that’s far from the only problem EC2 would face. Since the hardware on EC2 is pretty much frozen, and thus the options you have for the kernel are quite limited, in Gentoo there is no real reason to use modules and ramdisks (the fact that Amazon relies a lot on ramdisks has likely something to do that most other distributions rely on the ramdisks to load the correct modules for the harware at bootup, since they have to provide generic kernels). This would suggest you wouldn’t need the module-init-tools package, but that’s not the case, since it’s a dependency of OpenRC, as a number of init scripts and network modules depend on using modprobe themselves.

This actually made me think about our current situation with OpenRC: the openrc package itself provides init scripts for a few services, such as keyboard settings, timezone, “oldnet” support for things like ATM connections and so on so forth. Why shouldn’t we move these scripts to be installed by the single packages instead? So the modules init script would be installed by module-init-tools and keymaps by kbd? This way if I don’t use those I can simply not install the package on the target system. Unfortunately, this doesn’t seem to be in the plans of Mike, and I’m not sure if I can convince Jory to do this kind of work (since it would require changing a huge number of packages, especially packages that are mostly managed by base-system).

Architectures: maybe this should have gone before bu tI think it has lower importance on the matter; you might know that the performance of straight x86 on Xen are abysmally low; the reason is that there is a conflict between the way the Xen hypervisor hides itself and glibc implements Thread-Local Storage (TLS), which means that calls that are supposedly direct references are instead emulated, and thus slow the heck down of anything that uses TLS; which happens to be a lot of stuff nowadays since errno is implemented as TLS, and almost any function call in C uses errno to report errors back to the program.

A totally different thing is true for x86-64 though; so even if you were not to try hardened userland (which I think you should, but that’s a different problem) where it really makes a difference you might want to consider using a 64-bit kernel even with 32-bit userland, since that works without having to rebuild everything with strange CFLAGS. Unfortunately to do so, up to a few months ago you had to use the m2.large instances that are quite expensive to keep running.

Thankfully, the new t1.micro instances that they started providing as I said a few months ago allow both 32- and 64-bit guests. But they also have a catch: they only work with EBS-backed images, not with instance-store ones, and most distributions simply provide the instance-store AMIs. But that’s not an excessive trouble if you don’t factor in a few more problems related to EBS.

The volumes trouble: EBS is supposed to stand for Elastic Block Storage; it really has nothing elastic in it: once you give it a size, you cannot expand it (you have to create a new one starting from a snapshot, if you want to have a bigger volume). Also, it’s not elastic enough to be used directly as root file system. Sure, Amazon talks of EBS storage instances because what it runs on is a full-fledged volume, but to prepare them you have to use a snapshot, not a volume. I’m sure there are a number of good reasons why they decided to do it this way; on the other hand, this is quite upsetting because if you want to be able to terminate the instance and statefully restart it from time to time you have to take a snapshot of the current-volume before terminating it and then re-creating an AMI for it for the next time you start it up. Stopping a reserved instance is fine though, and require no further work, even though I guess it might be a tad more expensive.

You can have stateful EBS volumes without having to use snapshots; but those cannot be connected right at the start but rather need to be reconnected after each creation; so you can choose between creating the instance and then starting it, or simply wait till the instance is started, then connect the disks and start up the services using those volumes with Rudy, or with custom udev rules.

Custom kernels: might be of interest the fact that finally Amazon allows for custom kernels, which means we can use a non-Jurassic kernel with modern udev support (which is a godsend rootsend for what concerns attaching EBS volumes). This still requires a domU-capable kernel so in Gentoo you have to use xen-sources, not gentoo-sources to build your kernel, but still.

To implement this they use a hacked-up version of grub that will look for /boot/grub/menu.lst in either /dev/sda or /dev/sda1 (the “kernel” image to use changes between the two options — and between regions and bitness; don’t ask me why the bitness of the ”kernel“ differs, considering that we’re talking about grub).

Unfortunately since the filesystems supported by this grub is quite limited you have to jump through a few hops to have this working properly for filesystems like XFS and JFS that seem to have the best performances on EBS. And even more so if you want to use LVM. I’m still trying to overcome the problem of growing the partitions’ size properly when the original root EBS is not big enough. Maybe I should try with two volumes even though that wastes enough space on the /boot partition/volume.

Do note! If you are using a separate partition for /boot, for whatever reason you need the symlink /boot/boot -> . that is installed by the grub ebuild (without merging the ebuild); so just run ln -s . /boot/boot and be safe. Otherwise PVGRUB fail to find menu.lst and won’t let you start your instance.

Amazon EC2 and old concepts

On Friday I updated my Autotools Mythbuster guide to add support for 2.68 portability notes (all the releases between 2.65 and 2.67 have enough regressions to make them a very bad choice for generic use — for some of those we’ve applied patches that make projects build nonetheless, but those releases should really just disappear from the face of Earth). When I did so, I announced the change on my and then looked at the log, for a personal experiment of mine.

In a matter of a couple of hours, I could see a number of bots coming my way; some simply declared themselves outright (such as StatusNet that checked the link to produce the shortened version), while others tried more or less sophisticated ways to pass themselves for something else. On the other hand it is important to note that many times when a bot declares itself to be something like a browser, it’s simply to get served what the browser would see, for browser-specific hacks are still way too common, but that’s a digression I don’t care about here.

This little experiment of mine was actually aimed at refining my ModSecurity ruleset since I had some extra free time; the results of it are actually already available on the GitHub repository in form of updated blacklists and improved rules. But it made me think about a few more complex problems.

Amazon’s “Elastic Computer Cloud” (or EC2) is an interesting idea to make the best use of all the processing power of modern server hardware; this makes the phrase of a colleague of mine last year, sound even more true (“Recently we faced the return of clustered computing under the new brand of cloud computing, we faced the return of time sharing systems under the software as a service paradigm […]”) when you think of them introducing a “t1.micro” size for EBS-backed instance, for non-CPU-hungry tasks, that can be run with minimal CPU, but need more storage space.

But at the same time, the very design of the EC2 system gets troublesome in many ways; earlier this year I encountered troubles with hostnames when calling back between different EC2 instances, which ended up being resolved by using a dynamic hostname, like we were all used to use at the time of dynamic IP connections such as home ADSL (which for me has been basically till a couple of years ago). A very old technique, almost forgotten by many people, but pretty much necessary here.

It’s not the only thing that EC2 brought back from the time of ADSL though; any service based on it will lack a proper FcRDNS verification, which is very important to make sure that a bot request hasn’t been forged (that is until somebody creates a RobotKeys standard similar to DomainKeys standard), leaving it possible to non-legit bots to pass for legit ones, unless you can actually find a way to discern between the two with deep inspection of the requests. At the same time, it makes it very easy to pass for anything at all, since you can just judge by the User-Agent to find out who is making a request, as the IP address are dynamic and variable.

This situation lead to an obvious conclusion in the area of DNSBL (DNS-based black lists): all of the AWS network block is marked down as a spam source and is thus mostly unable to send email (or in the case of my blog, to post comments). Unfortunately this has a huge disadvantages: Amazon’s own internal network faces Internet from the same netblock, which means that Amazon employers can’t post comments on my blog either.

But the problem doesn’t stop there. As it was, my ruleset cached the result of robots analysis based on IP for a week. This covers the situation pretty nicely for most bots that are hosted on a “classic” system, but for those running on Amazon AWS, the situation is quite different: the same IP address can change “owner” in a matter of minutes, leading to false positives as well as using up an enormous amount of cache entries. To work around this problem, instead of hardcoding the expiration date of any given IP-bound test, I use a transaction variable, which defaults to the previous week, but gets changed to an hour in the case of AWS.

Unfortunately, it seems like EC2 is bringing us back in time, in the time of “real-time block lists” that need to list individual IPs rather than whole netblocks. What’s next, am I going to see again construction signs in websites “under construction”?

Avoid building (some) static libraries

Warning: the following trick is the kind of material that allows you to shoot yourself in the foot. On the other hand, since there isn’t enough push in Gentoo to solve the situation for good, I’ve decided to use this extensively myself.

Servers and embedded have two completely opposite conception of utility for libraries; in the latter case you most likely prefer static archives – when a single package use them at least – while in the former you prefer shared objects, even if it’s a single package, even better if it’s shared by multiple packages. This is mostly because of the difference in update policies: you don’t update an embedded system that is working unless you really have to; you update a server as soon as a security issue is found.

So, if shared objects are preferred for my own server, why should I have static archives laying around? Well I shouldn’t; especially on the server itself. Now, to remove static archives and include files on the server side, when using binary packages to upgrade the server, it’s actually quite easy: you just have to INSTALL_MASK them in the server’s configuration files (but not on the build chroot system you use to generate them). It solves the problem of using up space for no good reason but it still packages them in the the .tbz2 files that are sent to the server.

Now, for include files there is little you can do, since you do need on the build system, and you cannot really just INSTALL_MASK the static archives away entirely, because things like flex or libatomic_ops don’t install anything else but a static archive. What you can do is use the per-package environment support to remove the static archives (and, most of the time, the libtool files that are unneeded):

# /etc/portage/env/dev-libs/libxml2
export INSTALL_MASK="${INSTALL_MASK} /usr/lib*/*.a /usr/lib*/*.la"

This will be parsed at each phase, and will avoid merging in the static and libtool archives in the system and the tarballs. You just have to create a file like this for each of the packages that install unwanted static archives. But this will still compile two copies of every source file at build time, one with PIC and one without, to properly create static and shared objects. While solving this in a totally generic fashion is basically not possible, you can make use of some features of standard autoconf-based packages, starting from EXTRA_ECONF:

# /etc/portage/env/dev-libs/libxml2

# don't build static archive
export EXTRA_ECONF="${EXTRA_ECONF} --disable-static"

# get rid of the (useless) libtool archive
export INSTALL_MASK="${INSTALL_MASK} /usr/lib*/*.la"

This will pass the --disable-static option to econf, like most of the static-libs USE flags do. It only works for standard autoconf packages so it fails for stuff like OpenSSL and ncurses, for which you don’t have a proper escape route. On the other hand, it does its job for the most common packages. This trick gets also pretty useful when you have to deal with EC2, as the double-build is one heck of a killer.

Try not to abuse of EXTRA_ECONF, though. It actually lets you change ebuilds without having to hack eclasses or ebuilds altogether, but as I said, it’s one very good way to shoot yourself in the foot.

Amazon EC2: stable hostnames, for free, really

In my current line of work, dealing with Amazon’s “Elastic” Computer Cloud, I had to cope with the problem of getting a stable hostname for a “master” instance when dealing with “slaves”. I was appalled by the fact that Amazon does not provide a way to assign a “role” or “class name” to their instances, so to have them recognized from within their network.

When I was pointed out at multiple documentation showing how to achieve that feature, I knew that the problem wasn’t that Amazon failed to to think that through, but rather that they are most likely trying to get more money out of the users with these need. The common suggestion to solve the need to identify a particular instance, between executions, is to use their “Elastic IP” feature (which – like many other things they call “Elastic” – is not really elastic at all).

The idea behind this solution is that by assigning a public IPv4 (and keep in mind this is a precious resource nowadays) to the instance, you can pick up the reverse DNS for it (for instance will reverse-resolve to, and use that in your intra-EC2 requests… once using EC2’s own nameservers, will resolve to the (dynamic) internal IP address ( of the instance the IP was assigned to.

All fine and dandy, and this can be set up cleanly with Rudy as well. The problem here is… the IP is not free. Oh yes, the documentation will try to tell you that you won’t have to pay for it… as long as you use it. But it’s definitely not free, especially during the development phase of a project. As long as the address is assigned to an instance, you won’t pay for the Elastic IP service (but you will pay for the running instance); if you don’t have an assigned instance, you’ll end up paying $2.4 per day for the IP address to be reserved.

This is definitely one of the cases where Amazon’s limitation seem totally artificial to get you more money, as much money as they can get from you. D’oh!

What is my solution to this problem? It’s very simple: net-misc/ddclient. The IP address you’re looking for is the one, inside the instance, that is assigned to the eth0 interface. Just tell ddclient to fetch the address and use it on a classic DynDNS (or similar) hostname service. It doesn’t matter that you’re feeding them a local IP address (in the range); you won’t be able to use the same hostname for external and internal dealing, but it’ll definitely cost you less.

Ranting on about EC2

Yes, I’m still fighting with Amazon’s EC2 service for the very same job, and I’m still ranty about it. Maybe I’m too old-school, but I find using the good old virtual servers is much much easier to deal with. It’s not that I cannot see the usefulness of the AWS approach (you can easily try to get something going without sustaining a huge initial investment of capital to get the virtual servers, and you can scale it further on in the working), but I think more than half the interface is just an afterthought, rather than an actual design.

The whole software support for AWS is a bit strange: the original tools, that are available in Portage, are written in Java for the big part, but they don’t seem to be actively versioned and properly released by Amazon themselves, so you actually have to download the tools, then check the version from the directory inside the tarball to know the stable download URL for them (to package them in Gentoo, that is). You can find code to manage AWS services in many languages, including Ruby, for various pieces of it, but you cannot easily find an alternative console if not the ElasticFox extension for Firefox, which I have to say makes me doubt a lot (my Firefox is already slow enough). On the other hand, I actually found some promising command-line utilities in Rudy (which I packaged in Gentoo with a not indifferent effort), but beside some incompatibility with the latest version of the amazon-ec2 gem (which I fixed myself), there are other troubles with it (like not being straightforward how to handle multiple AMIs for different roles, or being impossible to handle snapshot/custom AMI creation through just it). Luckily, the upstream maintainer seems to be around and quite responsive.

Speaking about the libraries, it seems like one of the problems with the various Ruby-based libraries is that one of the most commonly used libraries (RightScale’s right_aws gem) is no longer maintained, or at least upstream has gone missing, and that causes obvious stir in the community. There is a fork for it, that forks the HTTP client library as well (right_http_connection, becoming http_connection — interestingly enough for a single, one line change that I’ve simply patched in on the Gentoo package). The problem is that the fork got worse than the original gem for what packaging is concerned: not only the gem is not providing the documentation, Rakefile, tests and so on, but they are not even tagged in the git repository last I check. Alas.

Luckily, it seems like amazon-ec2 is much better at this job; not that it was pain-free, but even here upstream is available, and fast to release a newer version; the same goes for litc, and the dependencies of the above-mentioned Rudy (see also this blog post from a couple of days ago). This actually make it so that the patches I’m applying, and adding to Gentoo, are deleted or don’t even enter the tree to begin with, which is good for the users who have to sync to keep the size of Portage down to acceptable levels.

Now, back to the EC2 support proper; I already ranted before about the lack of Gentoo support; turns out that there is more support if you go over the American regions, rather than the European one. And at the same time, the European zone seems to have problems: I spent a few days wondering why right_aws failed (and I thought it was because of the bugs that they forked it in the first place), but at the end I had to decide that the problem was with AWS itself: from time to time, a batch of my request fall into oblivion, with errors ranging from “not authorized“ to “instance does not exist” (for something I’m still SSH’d into, by the way). At the end, I decided to move to a different region, US/East, which is where my current customer is doing their tests already.

Now this is not easy either since there is no way to simply ask Amazon to transfer a volume from a given region (or zone) and copy it to another in their own systems (you can use snapshot to recreate a volume within a region on different availability zones, but that’s another problem). The official documentation suggests you to use out-of-band transmission (which, for big volumes, becomes expensive), and in particular the use of sync. Now this wouldn’t have to be too difficult, their suggestion is also to use rsync directly, which would be a good suggestion, if not for one particular. As far as I can tell, the only well-supported community distribution available, with a decently recent kernel (one that works with modern udev, for instance) is Ubuntu; in Ubuntu, you cannot access the root user directly as you all probably well know, and EC2 is no exception (indeed, the copiable command that they give you to connect to your instances is wrong for the Ubuntu case, they explicitly tell you to use the root user, when you have, instead, to use the ubuntu user, but I digress); this also means that you cannot use the root user as either origin or destination of an rsync command (you can sudo -i to get a root session from one or the other side, but not on both, and you need it on both to be able to rsync over the privileged files); okay the solution is definitely easy to find, you just need to tar up the tree you want to transfer, and then scp that over, but it really strikes odd to me that their suggested approach does not work with the only distribution that seems to be updated and supported on their platform.

Now, after the move to the US/East region, problems seems to have disappeared and all commands finally succeeded every time, yuppie! I finally was able to work properly on the code for my project, rather than having to fight with deployment problems (this is why my work is in development and not system administration); after such an ordeal, writing custom queries in PostgreSQL was definitely more fun (no Rails, no ActiveRecord, just pure good old PostgreSQL — okay I’m no DBA either, and sometimes I might have difficulties getting big queries to perform properly, as demonstrated by my work on the collision checker but some simpler and more rational scheme I can deal with pretty nicely). Until I had to make a change to the Gentoo image I was working with, and decided to shut it down, restart Ubuntu, and make the changes to create a new AMI; then hell broke loose.

Turns out that for whatever reason, for all the day yesterday (Wednesday 17th February), after starting Ubuntu instances, with both my usual keypair and a couple of newly-created ones (to exclude a problem with my local setup), the instance would refuse SSH access, claiming “too many authentication failures”. Not sure on the cause, I’ll have to try again tonight and hope that it works as I’m late on delivery already. Interestingly enough, the system log (which only appears one out of ten requests for it from the Amazon console) shows everything as okay, with the sole exception of the Plymouth software that crashes with segmentation fault (code 11) just after the kernel loaded.

So all in all, I think that as soon as this project is completed, and with the exception of eventual future work on this, I will not turn back to Amazon’s EC2 anytime soon; I’ll keep getting normal vservers, with proper Gentoo on them, without hourly fees, with permanent storage and so on so forth (I’ll stick with my current provider as well, even though I’m considering adding a fallback mirror somewhere else to be on the safe side; while my blog’s not that interesting, I have a couple of sites on the vserver that might require me to have higher uptime, but that’s a completely off-topic matter right now).

Ruby-NG: The Ruby’s eye (or, sometimes it’s a positive day)

I have ranted and ranted and ranted about Ruby packages not being good enough for packaging, I also have ranted about upstream developers not even getting their own testsuite cleared up, or being difficult to work with. I have complained about GitHub because of the way it allows to “fork” packages too easily. I’m not going to retract those notes, but… sometimes things do turn out pretty well.

In the past days I’ve been working toward adding Rudy in tree — as I don’t want to keep on building my own slow, hard, and boring scripts to deal with EC2, and I’m spending more time understanding how to get EC2 working than writing the code I’m paid to write. As I wrote before, this is another of those compound projects that is split in a high number of small projects (some literally one source files per gem!). It worried me to begin with, but on the other hand, the result is altogether not bad.

Not only the first fixes I had to apply to amazon-ec2 were applied, and a new version released, the very night I sent them upstream (and added them to Gentoo, now gone already), but also Delano (author of Rudy – and thus of lots of its dependencies) applied quickly most of my changes to get rid of the mandatory requirement for hanna, even on some packages I didn’t send them for yet, and released them again. Of course the job is far from finished, as I haven’t reached Rudy itself yet, but the outcome start to look much nicer.

I also have good words for GitHub right now: since it makes it very easy and quick to take the code from another project, patch it up and send it to the original author to be merged (and re-released hopefully). This also works fine with patches coming from other contributors, like Thomas Enebo from JRuby who sent me a fix (or “workaround” if you prefer, but it’s still a way to achieve the wanted result in a compatible way) to make newer matchy work properly with JRuby. On the whole, I have to say I’m getting quite positive about GitHub, but I’d very much like they allowed me to reply to the messages I receive by mail, rather than having to log-in on the system. I positively hate multiple mail systems, Facebook’s as well as GitHub’s, as well as most forums’.

And for the shameless plug and trivia time, I have more repositories in my GitHub page than items in my wishlist…

Anyway, back to work now!

My first experiences with with Amazon EC2

It really shouldn’t be a surprise for those reading this post that I’ve been tinkering with Amazon EC2 in the past few days, the reason for that is that you can find it out by either looking at my stream or at my commit feed and noticing how I ranted about EC2 and bumped the tools’ packages in Gentoo.

As a first experience with EC2, I have to say it does not really come out very nice… Now, the whole idea of EC2, doesn’t look half as bad. And on the whole I do think the implementation is also not too bad. What is a problem is the Gentoo EC2 guest support: while Amazon “boasts” support for Gentoo as a guest operating system, there is no real support for that out of the box.

*Incidentally, there is another problem: the AWS Web Console that Amazon make available killed my Firefox 3.6 (ground to a halt). I ended up installing Chromium even though it stinks (it stinks less than Opera, at least). It seems pretty much faster, but it’s lacking things like the Delicious sidebar, still. Seems like my previous post wasn’t totally far off. Sure there are extensions now, but as far as I can tell, the only one available for Delicious does not allow you to use Delicious as it was your bookmarks.*

The first problem you might have to affront is finding an image (AMI) to use Gentoo.. I could only find one (at least in the European availability zone), which is … a stage3. Yes a standard Gentoo stage3, without configured network, without SSH, without passwords, … Of course it won’t start. I spent the best part of three hours last night trying to get it to work, and at the end I was told that the only way to work that around is to install using Ubuntu as it was a live CD installing Gentoo on a real system. Fun.

So start up Ubuntu, create an EBS (stable storage) for the Gentoo install, install it as it was a normal chroot, create the snapshot and… here is one strange behaviour of Amazon: when you connect a new EBS volume to an instance, it is created as a block device (say, sdb). When you use a snapshot of that volume to register a machine (AMI), it becomes a partition (sda1). If, like me, you didn’t consider this when setting it up for install, and partitioned it normally, you’ll end up with an unbootable snapshot. Fun ensures.

By the way, to be able to register the machine, you have to do that through recent API tools, more recent than those that were available in Portage today. Hoping that Caleb won’t mind, I bumped them, and also made a couple of changes to the new API/AMI tools ebuilds: they now don’t require you to re-source the environment every time there’s an upgrade, and the avoid polluting /usr/bin full of symlinks.

So you finally complete the install and re-create the AMI, start an instance and… how the heck is it supposed to know your public key? That’s definitely a good question: right now there in Gentoo there is no way for the settings coming from Amazon to e picked up by Gentoo. It’s not difficult, and it seems to be documented as well, but as it is it’s not possible to do that. As I don’t currently need the ability to generate base images, I haven’t gone further pursuing that objective, on the other hand, I have some code that I might commit soonish.

Anyway, if you got interest in having better Gentoo experience on EC2, I might start looking into that more, in the future, as part of my general involvement, so let your voice be heard!