Virtualization updates

Seems like that one way or another a common “column” on my blog is reserved to virtualisation issues. I blogged a lot about VirtualBox (before dissing it finally), and then I moved on to KVM and QEmu.

Last time I blogged about it, I was still unable to get NetBSD to detect any network card with KVM, while I had OpenSolaris, FreeBSD and Ubuntu working fine. I also had some problems with Gentoo/FreeBSD and the KVM video emulation. But since then, stuff changed, in particular, QEmu now supports KVM technologies natively (and it’s not yet updated to the latest version). Let’s see if this changed something.

Thanks to aperez I now know how to get NetBSD to identify the network card: disabling ACPI. Unfortunately disabling ACPI with KVM freezes the boot. And I want to use VDE for networking since I already have Yamato configured as a router and file server for the whole network, which seems to fail when using NetBSD with QEmu: while dhcpd receives the requests, the replies never reach NetBSD, and I’m stuck for now. I’m going to try again with the newer QEmu version. Also, out of all the cards I tried in QEmu, the Intel E1000 fails because it cannot find the EEPROM.

The Gentoo/FreeBSD video problem that stopped me from using vim during the configuration phase on the minimal CD does not happen when using QEmu; on the other hand since the SDL output is tremendously slow, I’m using the VNC support; quite nice if it wasn’t that Vinagre does not seem to support VNC over Unix sockets, which would make the whole configuration much nicer, without consuming precious network ports. I have to see if I just missed something, and if I didn’t, I should either request for it to be added, or write the support myself (even better). I guess that the underlying code supports the Unix socket since I expect the virt-manager to use that to communicate with the VM.

Speaking of which, I haven’t looked at virt-manager or anything in quite a while; I should see if they still insist on not giving me the choice of just using VDE for networking instead of dnsmasq and similar; for now the whole configuration is done manually with a series of aliases in my ~/.shrc file, with (manually) sequential MAC addresses hardcoded, as well as VNC ports, LVM volumes (used for the virtual disks, seem to be quite faster than using a file over VFS), and hostnames (in /etc/hosts beside for Ubuntu that has Avahi working).

I have to admit, though, that I have some doubts about the performances of QEmu/KVM versus the usual KVM, at least it’s taking quite a long time to unpack the tarball with the stage3 of Gentoo/FreeBSD 7.1. I hope I/O is the bottleneck here.

Speaking of I/O as bottleneck, I was finally able to get a gigabit switch for the office, the next step is to buy some many metres of cable so I can actually wire up my bedroom with the office, passing through a few other rooms of the house so that I can actually have a fast enough network for all the computers in their standard setup (and use wireless only when strictly needed). Although I do have some doubts about this since I really want to move out.

In the mean time, Enterprise is soon going to be re-used as a backup box, I just need to find an easy way to send a WOL packet, wait for the box to come up, backup everything, and shut down, once a week. I have the last unused 500GB disk on that box so it should be easy. But I’d like to have an mtree of the data that has been backed up, which I’m still unsure on how to get.

Virtualisation woes, again

I know this starts to get old, with my ranting about virtualisation software, but since I’m trying my best to optimise the power of Yamato to software testing, I’m still working on getting virtualised systems to properly work for me.

In a long series of blog posts ranting about VirtualBox, QEmu, KVM and so on, there was exactly one system that worked quite fine up to now: Windows XP (SP3) under VirtualBox. With the latest release, though, this was broken too: network started up then came crashing down, with a striking resemblance to an old Solaris problem .

Since I was in need to have my Windows XP virtual machine working for a job, I tried porting it to Parallels on my iMac, with the Parallels demo (since my license was only valid for 3.x series). After waiting for the 64GB image file to convert, it turns out that there is no hope in getting it to start: the VirtualBox additions drivers crash with a blue screen of death at boot when they are executed outside of a VirtualBox instance; the Windows Recovery console does not allow to remove the drivers from loading, and trying to delete the drivers to avoid them from loading was not an option, since they get installed in the program files directory (that the recovery console cannot access).

At the end, given the absolute unreliability of VirtualBox on every operating system at this point, I simply gave up and paid for the upgrade of my license to Parallels 4, which is now providing as my only Windows XP instance (which I’m still unfortunately tied to for work), and deleted VirtualBox from my system. Why, you’d ask, since networking not working is far from the biggest problem out there? Well the biggest problem, and the final straw that broke the camel’s back, was that while trying to figure out why Samba was not working, VirtualBox’s network filter module crashed the kernel. So what? Well, VirtualBox decided that rather than using the quite well-tested mixed kernel/userland TUN/TAP networking system, or the userland virtual network (with tap to interfacing it with the rest) provided by VDE, they had to provide a kernel module instead. For performance reasons, or, quite most likely, so that they can have the same interface to the network internals between different operating systems. Do I have to make it explicit how this is a problem?

Interestingly, while writing this I noticed that there are problems downloading VirtualBox and the thing also reminded me of how many time they messed up the ebuilds by changing the tarballs…

But it doesn’t stop here. Remember the NetBSD trouble with the networking I reported about one month ago? Well, I wanted to see if something changed with the new NetBSD 5.0 release (I actually wanted to make sure that feng detected the newly-added POSIX Message Queue support properly), but still no luck, I don’t see any network card with whatever model I provide to KVM, included the e1000 that I’d expect NetBSD to support at least.

On the other hand I was at least able to get Ubuntu (9.04) working on KVM, next step is Fedora 11, so I can actually test feng on other distributions as well as Gentoo.

Virtual machine, real problems

Since I bought Yamato I have been trying my best to make use of the AMD-V support in the Opterons, this included continuing the fight with VirtualBox to get it to work properly, with Solaris and Fedora, then trying RedHat’s virt-manager and now, after the failure the other day QEmu 0.10 (under Luca’s insistence).

The summary of my current opinions is something like the following:

  • VirtualBox is a nice concept but the limitation in the “Open Source Edition” are quite nasty, plus it has huge problems (at least, in the OSE version) with networking under Solaris (which is mind boggling for me since both products are developed by Sun), making it unusable for almost anything in my book; replacing the previously used Linux tun/tap code with its own network modules wasn’t very nice because it reminded me of VMware, and it didn’t solve much in my case;
  • RedHat’s virt-manager is a nice idea but it has limitations that are (quite understandably from one point of view) tied with the hardcoding of RedHat style systems; I say quite understandably because I’m not even dreaming to ask RedHat to support other operating systems before they feel their code is absolutely prime-time ready; on the other hand it would be nice if there was better support for Gentoo, maybe with an active branch for it;
  • I still don’t like the kqemu approach, so I’m hoping for the support to Linux’s own KVM interface in the next kernel release (2.6.29), but it should be feasible; on the other hand, setting up QEmu (or kvm manually) is quite a task the first time.

So while I’m using VirtualBox to virtualise a Windows XP install (which, alas, I have to use for some work tasks and to offer Windows support to my “customers”) I decided to try QEmu for a FreeBSD (vanilla) virtual system; I needed a vanilla FreeBSD to try a couple of things out, so that was a good choice to start. I was actually impressed by the sheer speed of FreeBSD install in the virtual system even without kqemu or KVM, it indeed took less than on my old test systems. I don’t know if the I/O difference between QEmu and VirtualBox was because VirtualBox uses more complex virtual disk images (with recovery data I expect), or because I set QEmu to work straight on a block device (lvm logical volume); I had, though, to curse a bit to get networking working.

A side on networking; since what I wanted was to be able to interface the virtual machines with the rest of my network transparently, I decided to give a try to net-misc/vde; unfortunately getting the thing working with that has been more troublesome than expected. For once, if you don’t set up the TAP device explicitly with OpenRC, vde will try to do so for you, but on my system, it put udev in a state that continuously took up and down the interface, quite unusable. Secondly, I had some problem with dhcpd: even if I set the DHCPD_IFACE variable in /etc/conf.d/dhcpd, the init script does not produce proper service dependencies, I have to explicitly set RC_NEED. In both those case the answer would be “dynamic” dependencies of the scripts, calculating the needed network services based on the settings in the conf.d files. I guess I should open bugs for those.

Once I finally had the networking support working properly, I set up SSH, connected and started the usual task of basic customisation. The first step for me is always to get zsh as shell. Not bash because I don’t like bash as a project, I find zsh more useful too. But once it started building m4, and in particular to test for strstr() time linearity, the virtual machine was frozen solid; qemu strted taking 100% CPU constantly, and even after half an hour it never moved from there. I aborted the VM and tried again, hoping it was just a glitch, but it’s perfectly reproducible. I don’t know what the problem is with that.

So I decided to give a try to installing Solaris, I created a new logical volume, started up again qemu and .. it is frozen solid during boot from the 2008.11 DVD.

In truth, I’m disappointed because the FreeBSD install really looked promising: fast, nice, not overloading more than a single core (I have eight, I can spare one or two for constantly-running VMs), it also worked fine when running as unprivileged user (my user) after giving it access to the kqemu device and the block device with the virtual disk; it didn’t work as nice with the tun/tap support in qemu itself in this setup since it required root to access the tap device, but at least with vde it reduced the amount of code running unprivileged.

On the other hand, since the KVM and QEmu configuration is basically identical (beside the fact that they emulate different network cards), I just tried again kvm, using the manual configuration I used for QEmu and vde for networking (networking configuration was what made me hope to use virt-manager last time, but now I know I don’t need it); it does seem faster, and it also passed the strstr() test before. So I guess the winner this round is KVM, and I’ll wait for the next Linux release to test the QEmu+Linux KVM support.

Post Scriptum: even KVM is unable to start the OpenSolaris LiveDVD though, so I wonder if it’s a problem with Solaris itself; I’d like to try the free as in soda version of Solaris 10, but the “Sun Download Manager” does not seem to work with IcedTea6 and downloading that 4GB image with Firefox is masochistic.