Virtual machine, real problems

Since I bought Yamato I have been trying my best to make use of the AMD-V support in the Opterons, this included continuing the fight with VirtualBox to get it to work properly, with Solaris and Fedora, then trying RedHat’s virt-manager and now, after the failure the other day QEmu 0.10 (under Luca’s insistence).

The summary of my current opinions is something like the following:

VirtualBox is a nice concept but the limitation in the “Open Source Edition” are quite nasty, plus it has huge problems (at least, in the OSE version) with networking under Solaris (which is mind boggling for me since both products are developed by Sun), making it unusable for almost anything in my book; replacing the previously used Linux tun/tap code with its own network modules wasn’t very nice because it reminded me of VMware, and it didn’t solve much in my case;
RedHat’s virt-manager is a nice idea but it has limitations that are (quite understandably from one point of view) tied with the hardcoding of RedHat style systems; I say quite understandably because I’m not even dreaming to ask RedHat to support other operating systems before they feel their code is absolutely prime-time ready; on the other hand it would be nice if there was better support for Gentoo, maybe with an active branch for it;
I still don’t like the kqemu approach, so I’m hoping for the support to Linux’s own KVM interface in the next kernel release (2.6.29), but it should be feasible; on the other hand, setting up QEmu (or kvm manually) is quite a task the first time.

So while I’m using VirtualBox to virtualise a Windows XP install (which, alas, I have to use for some work tasks and to offer Windows support to my “customers”) I decided to try QEmu for a FreeBSD (vanilla) virtual system; I needed a vanilla FreeBSD to try a couple of things out, so that was a good choice to start. I was actually impressed by the sheer speed of FreeBSD install in the virtual system even without kqemu or KVM, it indeed took less than on my old test systems. I don’t know if the I/O difference between QEmu and VirtualBox was because VirtualBox uses more complex virtual disk images (with recovery data I expect), or because I set QEmu to work straight on a block device (lvm logical volume); I had, though, to curse a bit to get networking working.

A side on networking; since what I wanted was to be able to interface the virtual machines with the rest of my network transparently, I decided to give a try to net-misc/vde; unfortunately getting the thing working with that has been more troublesome than expected. For once, if you don’t set up the TAP device explicitly with OpenRC, vde will try to do so for you, but on my system, it put udev in a state that continuously took up and down the interface, quite unusable. Secondly, I had some problem with dhcpd: even if I set the DHCPD_IFACE variable in /etc/conf.d/dhcpd, the init script does not produce proper service dependencies, I have to explicitly set RC_NEED. In both those case the answer would be “dynamic” dependencies of the scripts, calculating the needed network services based on the settings in the conf.d files. I guess I should open bugs for those.

Once I finally had the networking support working properly, I set up SSH, connected and started the usual task of basic customisation. The first step for me is always to get zsh as shell. Not bash because I don’t like bash as a project, I find zsh more useful too. But once it started building m4, and in particular to test for strstr() time linearity, the virtual machine was frozen solid; qemu strted taking 100% CPU constantly, and even after half an hour it never moved from there. I aborted the VM and tried again, hoping it was just a glitch, but it’s perfectly reproducible. I don’t know what the problem is with that.

So I decided to give a try to installing Solaris, I created a new logical volume, started up again qemu and .. it is frozen solid during boot from the 2008.11 DVD.

In truth, I’m disappointed because the FreeBSD install really looked promising: fast, nice, not overloading more than a single core (I have eight, I can spare one or two for constantly-running VMs), it also worked fine when running as unprivileged user (my user) after giving it access to the kqemu device and the block device with the virtual disk; it didn’t work as nice with the tun/tap support in qemu itself in this setup since it required root to access the tap device, but at least with vde it reduced the amount of code running unprivileged.

On the other hand, since the KVM and QEmu configuration is basically identical (beside the fact that they emulate different network cards), I just tried again kvm, using the manual configuration I used for QEmu and vde for networking (networking configuration was what made me hope to use virt-manager last time, but now I know I don’t need it); it does seem faster, and it also passed the strstr() test before. So I guess the winner this round is KVM, and I’ll wait for the next Linux release to test the QEmu+Linux KVM support.

Post Scriptum: even KVM is unable to start the OpenSolaris LiveDVD though, so I wonder if it’s a problem with Solaris itself; I’d like to try the free as in soda version of Solaris 10, but the “Sun Download Manager” does not seem to work with IcedTea6 and downloading that 4GB image with Firefox is masochistic.

Free virtualisation – not working yet

If you’ve been following my blog for a while you probably remember how much I fought with VirtualBox once it was released to get it to work, so that I could use OpenSolaris. Nowadays, even with some quirks, VirtualBox Open Source Edition is fairly usable, and I’m using it not only for OpenSolaris but also for a Fedora system (which I use for comparing issues with Gentoo), a Windows install (that I use for my job), and a Slackware install that I use for kernel hacking.

Obviously, the problem is that the free version of VirtualBox come with some disadvantages, like not being able to forward USB devices, having limited type of hardware to virtualise and so on. This is not much of a problem for my use, but of course it would have been nicer if they just open sourced the whole lot. I guess the most obnoxious problem with VirtualBox (OSE at least, not sure about the proprietary version) is the inability to use a true block device as virtual disk, but rather having to deal with the custom image format that is really slow at times, and needs to pass through the VFS.

For these reasons Luca suggested me many times to try out kvm instead, but I have to say one nice thing of VirtualBox is that it has a quite easy to use interface which allows me to set up new virtual machines in just a few clicks. And since nowadays it also supports VT-x and similar, it’s not so bad at all.

But anyway, I wanted to try kvm, and tonight I finally decided to install it, together with the virt-manager frontend; although there are lots of hopes for this, it’s not yet good enough, and it really isn’t usable for me at all. I guess I might actually get to hack at it, but maybe this is a bit too soon yet.

Continue reading on my blog for the reasoning, if you’re interested.
One thing I really dislike of the newer versions of VirtualBox is that it went the VMware route and decided to create its own kernel module to handle networking. They said it’s for performance reasons but I’d expect the main reason is that it would allow them to have a single interface between Linux, Windows, Solaris and so on. The TUN/TAP interface is Linux-specific so supporting that together with whatever they have been doing on the other operating systems is likely a cost for them. Although I can understand their reasoning, it’s not that I like it at all. More kernel code means more code that can crash your system, especially when not using the Open Source Edition.

Interestingly enough, the RedHat’s Virtual Machine Manager is instead doing its best to avoid creating new network management software, and uses very common pieces of software: dnsmasq as DHCP server, the Linux kernel bridging support, and so on. This is very good, but it also poses a few problems: first of all, my system already runs a DHCP server, why do I have to install another? But it’s not just that; instead of creating an userspace networking bridging software, like VirtualBox does, it tries to use what the kernel provides already, in the form of the bridging support and iptables to forward requests and create NAT zones. This is all fine and dandy, as it reduces the feature duplication in software, but obviously requires more options to be enabled in kernels that might not otherwise have it enabled at all.

As it happens, my system does have bridging support installed, but not iptables nor masquerade targets and similar. Up to now I never used them so it had no reason to be there. I also sincerely can’t understand why it does need it if I don’t want a NAT, but it doesn’t seem to allow me to set up the network myself. Which would include me being able to just tell it to create a new interface and add it to a bridge I manage myself, leaving to me all the details like dhcp, and thus not requiring iptables at all.

Also, even though there is a way to configure an LVM-based storage pool, it does not seem to allow me to choose directly one of the volumes of that pool when it asks me what to use as a virtual disk. Which is kinda silly, to me.

But these are just minor annoyances, there are much bigger problems: if the modules for kvm (kvm and kvm-amd in my system) are not loaded when virt-manager is loaded, it obviously lack a way to choose KVM as hypervisor. This is nice, but it also fails to see that there is no qemu in the system, and tries to use a default path, that is not only specific to RedHat, but also very not existing here (/usr/lib64/xen/bin/qemu-dm, on x86-64, just checking the uname!), returning an error that the particular command couldn’t be found. At least it should probably have said that it couldn’t find qemu, rather than the particular path. It would have also been nice to still allow choosing kvm but then telling that the device was missing (and suggested loading the modules; VirtualBox does something like that already).

To that you have to add that I haven’t been able to finish the wizard to create a new virtual machine yet. This because it does not check the permission to create the virtual machine before proposing you to create one, so it let you spend time to fill in the settings in the wizard and then fails telling you don’t have permission to access the read/write socket. Which by default is accessible only by root.

Even though it’s obvious by the 0-prefix that the software is still not production-level, there are also a few notes on the way the interfaces are designed. While it’s a good idea to use Python to write the interface, since it allows a much faster test of the code and so on, and speed is not a critical issue here, as the work is carried out by C code in the background, every error is reported with a Python traceback, one of the most obnoxious things to do for users. In particular for the permission problem I just listed, the message is a generic error message: “Unable to complete install: ‘unable to connect to ‘/var/run/libvirt/libvirt-sock’: Permission denied’”; what is the user supposed to know here? Of course virtualisation is not something the user at the first tries with Linux is going to use, but still, even a power user might be scared by errors that appear this way and have attached a traceback (that most users would probably rationally link with “the program crashed” like bug-buddy messages, rather than “something wasn’t as the software expected”, which is what happened.

On a very puny level, instead, I have to notice that just pressing Enter to proceed to the next page on the wizard fails, and using ESC to close the windows too. Which is very unlike any other commonly-used software.

So to cut the post short now, what is the outcome of my test? Well, I didn’t get to test the actual running performance of KVM against VirtualBox, so I cannot say much about that technology. I can say that there is a long road ahead for the Virtual Machine Manager software to become a frontend half as good as VirtualBox’s or VMware’s. This does not mean that the software was badly written, at all. The software by design is not bad at all; of course it’s very focused on RedHat and less on other distributions, which explains lots of the problems I’ve noticed, but in general I think it’s going the right way. A more advanced setup for advanced users would probably be welcome by me, as well as an ISO images management like the one VirtualBox has (even better if you can assign an ISO image to be of a specific operating system, so that when I choose “Linux”, “Fedora” it would list me the various ISO for Fedora 8, 9, 10 and then have a “Other” button to take a different ISO to install, if desired.

I’ll keep waiting to see how it evolves.

Solaris on VMware: another one lost

So after the problem with VirtualBox, even VMware makes it difficult for me to run Solaris Express.

To summarise the problem with VirtualBox, well, the network doesn’t work, it’s a known problem probably due to the PCnet card not completely emulated.

With VMware the problem is more subtle: the graphic emulation is slower, the mouse movement clumsier, at least without the VMware tools installed in the guest operating system, so I installed them.

Since I installed them and configured xorg.conf, the size of the window increased to be bigger than the size of my monitors. Even if the xorg.conf is set to only allow 1152×900, the size taken by the X server is way higher, although I’m not sure what it really is.

Why running Solaris in a virtual machine is so difficult? I won’t use QEmu anytime soon, I had my past bad experience with kqemu (lockup of the box) and even if it’s GPL’d now, I don’t really trust it too much; I could use the modified QEmu with KVM, if I had a KVM-capable CPU, but the only one I have is the one from the MacBookPro, but I don’t really want to burn it out by having to do heavy work with the temperature I have here; beside it’s not as easy to handle as a two-screens workstation.

So right now my efforts to port xine-lib-1.2 to work fine on Solaris, and the work to be done to finish the audio conversion branch, are stalled and I have no clue when they’ll be resurrected; I’d need a new spare box (which is not going to be easy for me to take anyway, for space constrains), or a new main box (to run KVM); neither are likely to happen in the near future.

And now the big question: why should you care if you don’t use Solaris? If I can’t port the audio out plugin to the new branch, I can’t merge the branch into 1.2, and 1.2 series will not likely be released until at least that part of the code is merged back into it. There are already enough things that needs to be fixed or postponed, but I don’t really want to ignore the audio conversion improvement for 1.2 series.