Linux Resource Containers and Gentoo, another step

I currently am tackling a new job, details of which I’m not going to disclose, but one of the steps on my “pre-work setup” task was to create a new LXC-based guest, reason for which is quite easy to guess: I wanted a clean slate to work on rather than my messed up live workstation.

I could have gone to use Fedora or Ubuntu in a KVM, or even a Gentoo install in KVM, but I still like a lot the idea of using LXC, it’s a very nice “virtualisation” technology that does not consume a huge amount of resources, nor it requires special kernel support (it’s working mostly fine with vanilla kernels). And as I’m one of the maintainers for it in Gentoo, I also wanted to check if I could improve the situation.

Thanks to Adrian Nord, I’ve been able to learn quite a bit of things about LXC even without having to follow the upstream mailing lists. Unfortunately, the documentation about using LXC is scattered and sometimes messed up, so it took me a while to get this to work as intended. So here a few notes that might come useful to other people wanting to use it:

I’ve currently not packaged lxc-0.6.5 that was released earlier this week; my reason to avoid it is that I cannot get it to work, at least I couldn’t when it was released; once I can find what the heck is wrong with it, I’ll make sure to add it; unfortunately I cannot look into it as long as I’m using the lxc guests to do real work. On a similar, and maybe related note, I couldn’t get 0.6.4 to work on the 2.6.33 release candidates, so it might be a problem of version-specificity.

You cannot rely on udev, neither to have /dev filled in, nor to have a working detection of hardware; I guess that might be obvious given that we’re talking about virtual environments, but there is a trick: by default if you just bind-mount /dev (like I was used to do on chroots) you end up with the guest having full access to the devices in the host. You need to set up an access control list on the device nodes that are allowed to be accessed for that to work.

The easiest way to handle the problem with /dev, as Adrian pointed me to, is to use a static /dev; this does not mean using the static-dev package as that installs a huge amount of extra stuff we will probably never need, and at the same time it does not install some other devices, such as random on my system. My solution? This tarball – not even compressed as it’s just metadata! – creates the subset of devices that seem to actually be used by this stuff.

You don’t want to bind-mount /dev or /dev/pts, but you want instead for LXC to take care of the latter for you: not only it’ll mount a new instance of the PTS filesystem, but it’ll also bind the tty* devices to pseudo-terminals, allowing you to access the various virtual consoles through the lxc-console command. To do that you need a bit of configuration in the file:

lxc.tty = 12
lxc.pts = 128

Note the numbers there: the second one, referring to lxc.pts is, as of 0.6.4, unused, and just needs to be non-zero so that the new /dev/pts instance is created properly. The former instead is important: that’s the number of TTYs that LXC will be wrapping around to pseudo-terminals. You want that to be at least 12 for Gentoo-based guests. The reason is that consolefonts and keymaps scripts will be accessing tty1 through tty12 during the “start-up” and will then mess up you system badly if they are not wrapped. There are two extra catches: the first is that if the devices don’t exist, the init scripts will create regular files with those names (and that might sound quite strange when you go debugging you problems), the second is that you need to have the files for the given number of ttys around: LXC will silently fail if it cannot find the file to bind at the device path. Which also takes a while to debug.

I still haven’t finished making sure that the cgroup device access functions work as intended, so for now I won’t post anything about those just yet. But you might want to look into lxc.cgroup.devices.access to whitelist the device nodes that I have in the tarball, with the exclusion of the 4:* range (which is the real TTYs.

Now, maybe I’ll be able to add an init script with the next release of LXC I’m going to package!

5 thoughts on “Linux Resource Containers and Gentoo, another step

  1. Hi Diego, will you package in your findings along with the init script? I.e. the static dev or the tty configuration?

    Like

  2. Hi, Diego. Do you use lxc.mount.entry? Doeslxc.mount.entry = /lib /root/myrootfs/lib none ro,bind 0 0work?

    Like

  3. @Pavel of course I’ll be documenting the configuration sample and the static dev with LXC proper, once I can get 0.6.5 to behave. Or 2.6.33 to behave, or whatever the magical combination is there. I would also like to have a guest profile for LXC that disables udev entirely, but I’m afraid that might go a bit too far in what I can do with the time I have.@Vadim no I’m still currently using an fstab as it failed quiiiite badly last time I tried using the @lxc.mount.entry@ options. I should try it again with .5, if I can get it to work, but in theory your line _should_ work.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s