Linux Resource Containers and Gentoo, another step

I currently am tackling a new job, details of which I’m not going to disclose, but one of the steps on my “pre-work setup” task was to create a new LXC-based guest, reason for which is quite easy to guess: I wanted a clean slate to work on rather than my messed up live workstation.

I could have gone to use Fedora or Ubuntu in a KVM, or even a Gentoo install in KVM, but I still like a lot the idea of using LXC, it’s a very nice “virtualisation” technology that does not consume a huge amount of resources, nor it requires special kernel support (it’s working mostly fine with vanilla kernels). And as I’m one of the maintainers for it in Gentoo, I also wanted to check if I could improve the situation.

Thanks to Adrian Nord, I’ve been able to learn quite a bit of things about LXC even without having to follow the upstream mailing lists. Unfortunately, the documentation about using LXC is scattered and sometimes messed up, so it took me a while to get this to work as intended. So here a few notes that might come useful to other people wanting to use it:

I’ve currently not packaged lxc-0.6.5 that was released earlier this week; my reason to avoid it is that I cannot get it to work, at least I couldn’t when it was released; once I can find what the heck is wrong with it, I’ll make sure to add it; unfortunately I cannot look into it as long as I’m using the lxc guests to do real work. On a similar, and maybe related note, I couldn’t get 0.6.4 to work on the 2.6.33 release candidates, so it might be a problem of version-specificity.

You cannot rely on udev, neither to have /dev filled in, nor to have a working detection of hardware; I guess that might be obvious given that we’re talking about virtual environments, but there is a trick: by default if you just bind-mount /dev (like I was used to do on chroots) you end up with the guest having full access to the devices in the host. You need to set up an access control list on the device nodes that are allowed to be accessed for that to work.

The easiest way to handle the problem with /dev, as Adrian pointed me to, is to use a static /dev; this does not mean using the static-dev package as that installs a huge amount of extra stuff we will probably never need, and at the same time it does not install some other devices, such as random on my system. My solution? This tarball – not even compressed as it’s just metadata! – creates the subset of devices that seem to actually be used by this stuff.

You don’t want to bind-mount /dev or /dev/pts, but you want instead for LXC to take care of the latter for you: not only it’ll mount a new instance of the PTS filesystem, but it’ll also bind the tty* devices to pseudo-terminals, allowing you to access the various virtual consoles through the lxc-console command. To do that you need a bit of configuration in the file:

lxc.tty = 12
lxc.pts = 128

Note the numbers there: the second one, referring to lxc.pts is, as of 0.6.4, unused, and just needs to be non-zero so that the new /dev/pts instance is created properly. The former instead is important: that’s the number of TTYs that LXC will be wrapping around to pseudo-terminals. You want that to be at least 12 for Gentoo-based guests. The reason is that consolefonts and keymaps scripts will be accessing tty1 through tty12 during the “start-up” and will then mess up you system badly if they are not wrapped. There are two extra catches: the first is that if the devices don’t exist, the init scripts will create regular files with those names (and that might sound quite strange when you go debugging you problems), the second is that you need to have the files for the given number of ttys around: LXC will silently fail if it cannot find the file to bind at the device path. Which also takes a while to debug.

I still haven’t finished making sure that the cgroup device access functions work as intended, so for now I won’t post anything about those just yet. But you might want to look into lxc.cgroup.devices.access to whitelist the device nodes that I have in the tarball, with the exclusion of the 4:* range (which is the real TTYs.

Now, maybe I’ll be able to add an init script with the next release of LXC I’m going to package!