How to improve releases quality: working on PulseAudio 0.9.16

Just when I said that I was resuming my work as a PulseAudio maintainer in Gentoo, Lennart released a 0.9.16-test1 tarball. This was my cue to enter the scene upstream: the first test at packaging this in Gentoo failed, for a series of different reasons, some of which are internal (we don’t have the latest version of udev available yet, I hope we will by the time PulseAudio 0.9.16 final is releasd), but most are due to upstream changes that didn’t take into consideration some corner cases that Gentoo, as usual, gets to deal with.

So you won’t see the test1 (rc1) ebuild in the tree at all, you’ll probably have to wait for test2, and even that will require some work. For now I’ve fixed all the build- and run-time issues I’ve seen in the released tarball and git repository; plus I’ve been able to get it to properly build fully on both (Gentoo/)FreeBSD and OpenSolaris (with Prefix). I haven’t been able to experiment with actually having it playing yet, but it’ll come there at one point.

Unfortunately there are still a few shady details that I or someone else has to take care of. For instance, the tests still fail consistently: last time I tried them I got two failures on Yamato, one related to IPv6 enabled in PulseAudio build, but not enabled for the kernel, resulting in the IP ACL test asseting out (now I’ve fixed it, by warning of the case, and ignoring it as a failure); the other is the mixing test, which fails for everybody because it doesn’t know anything about the 24-bit and 24-bit-in-32-bit sample types; this I extended to support 24-bit, but was unable to do anything about the 24-bit-in-32-bit because I couldn’t grok it properly.

On non-Linux operating systems (FreeBSD and OpenSolaris), I had to work on a few more issues, like implicit declarations (there still is one in OpenSolaris), shadowed names, and of course there is some slight porting to be done, which I have nowhere near finished yet: the shm (Shared Memory) support in FreeBSD is imperfect, and for neither operating systems I’ve implemented the “get process name” function.

Okay I’m not able to provide a 100% porting to all the operating systems out there, but I still think I can do a bit to help out by making sure that PulseAudio won’t need to be extensively patched by all the porters out there. And until Lennart actually gets around merging my patches, you can find all them at gitorious so you can test them.

Update (2017-04-22): as you may know, Gitorious was acquired by GitLab in 2015 and turned down the service. This means you can’t access those patches anymore.

Random bits

Since I haven’t had much time to write in the past few days (lots of stuff going on in my life, both personal and professional), I’m just going to draft up a few quick, random bits that might be of interest:

  • I’ve written some notes about automake 1.11 on the Axant blogs; as I said before, you might want to take a look to that blog too since I’ll post something there from time to time; I’m also working on some extension to the Autotools Mythbusters guide which I’ll post later;
  • staying in topic with the guide, it is no longer donation-based (for the same reason why I cannot accept money donations any longer); instead I’ll work on it on a free-time basis; you can still send me a gift if you wish for me to write about a particular topic about Autotools, or you can hire me if you need it for sure;
  • for what concerns my return to Gentoo/FreeBSD I still haven’t finished porting PulseAudio; but I was able to tackle a few more problems, included unused and missing USE flags in the system ebuilds, now it should look nicer;
  • I’m currently on hold when it comes to feng but I count on getting back on track quite soon;
  • no luck with NetBSD yet, even with the latest version of QEmu (0.10.4);
  • while I have scanned over 400 sheets of paper , there are more documents, bills, and packaging slips that I haven’t scanned; and I haven’t even found the time to actually scan the house’s bills (which wouldn’t be required for me to move out but would still be nice); scanning the stuff by hand with the flatbad scanner is not too nice though, so if somebody has a suggestion for a cheap (less than €300) scanner with ADF properly supported by sane, it’d be nice;
  • the Typo install on the blog has been modified a bit to improve the tagging functionalities, but I haven’t merged duplicated tags yet, nor added proper descriptions where it’s needed, for this reason I’ve temporarily removed indexing of tags for all the search engines; once the tags are cleaned up I’ll see to put it back on; I should also hide tags with just one post each, for safety;
  • I need some personal time to handle some things with the change of season, starting with discarding old clothes that I can’t wear any longer (the Summer Of Code 2006 T-Shirt is one of those unfortunately): with the whole thing about hospital, I went from an XXL to a M size..

And for those who actually wonder what the heck I’ve been doing lately, there’s always identi.ca (and Twitter, but I prefer identi.ca).

Virtualization updates

Seems like that one way or another a common “column” on my blog is reserved to virtualisation issues. I blogged a lot about VirtualBox (before dissing it finally), and then I moved on to KVM and QEmu.

Last time I blogged about it, I was still unable to get NetBSD to detect any network card with KVM, while I had OpenSolaris, FreeBSD and Ubuntu working fine. I also had some problems with Gentoo/FreeBSD and the KVM video emulation. But since then, stuff changed, in particular, QEmu now supports KVM technologies natively (and it’s not yet updated to the latest version). Let’s see if this changed something.

Thanks to aperez I now know how to get NetBSD to identify the network card: disabling ACPI. Unfortunately disabling ACPI with KVM freezes the boot. And I want to use VDE for networking since I already have Yamato configured as a router and file server for the whole network, which seems to fail when using NetBSD with QEmu: while dhcpd receives the requests, the replies never reach NetBSD, and I’m stuck for now. I’m going to try again with the newer QEmu version. Also, out of all the cards I tried in QEmu, the Intel E1000 fails because it cannot find the EEPROM.

The Gentoo/FreeBSD video problem that stopped me from using vim during the configuration phase on the minimal CD does not happen when using QEmu; on the other hand since the SDL output is tremendously slow, I’m using the VNC support; quite nice if it wasn’t that Vinagre does not seem to support VNC over Unix sockets, which would make the whole configuration much nicer, without consuming precious network ports. I have to see if I just missed something, and if I didn’t, I should either request for it to be added, or write the support myself (even better). I guess that the underlying code supports the Unix socket since I expect the virt-manager to use that to communicate with the VM.

Speaking of which, I haven’t looked at virt-manager or anything in quite a while; I should see if they still insist on not giving me the choice of just using VDE for networking instead of dnsmasq and similar; for now the whole configuration is done manually with a series of aliases in my ~/.shrc file, with (manually) sequential MAC addresses hardcoded, as well as VNC ports, LVM volumes (used for the virtual disks, seem to be quite faster than using a file over VFS), and hostnames (in /etc/hosts beside for Ubuntu that has Avahi working).

I have to admit, though, that I have some doubts about the performances of QEmu/KVM versus the usual KVM, at least it’s taking quite a long time to unpack the tarball with the stage3 of Gentoo/FreeBSD 7.1. I hope I/O is the bottleneck here.

Speaking of I/O as bottleneck, I was finally able to get a gigabit switch for the office, the next step is to buy some many metres of cable so I can actually wire up my bedroom with the office, passing through a few other rooms of the house so that I can actually have a fast enough network for all the computers in their standard setup (and use wireless only when strictly needed). Although I do have some doubts about this since I really want to move out.

In the mean time, Enterprise is soon going to be re-used as a backup box, I just need to find an easy way to send a WOL packet, wait for the box to come up, backup everything, and shut down, once a week. I have the last unused 500GB disk on that box so it should be easy. But I’d like to have an mtree of the data that has been backed up, which I’m still unsure on how to get.

Debian, Gentoo, FreeBSD, GNU/kFreeBSD

To shed some light and get around the confusion that seems to have taken quite a bit of people who came to ask me what I think about Debian adding GNU/kFreeBSD to the main archive, I’d like to point out, once again, that Gentoo/FreeBSD has never been the same class of project as Debian’s GNU/kFreeBSD port. Interestingly enough, I already said this before more than three years ago.

Debian’s GNU/kFreeBSD uses the FreeBSD kernel but keeps the GNU userland, which means the GNU C Library (glibc), the GNU utilities (coreutils) and so on so forth; on the other hand, Gentoo/FreeBSD uses both the vanilla FreeBSD kernel, and mostly vanilla userland. With mostly I mean that some parts of the standard FreeBSD userland are replaced, with either compatible, selectable or updated packages. For instance instead of shipping sendmail or the ISC dhcp packages as part of the base system, Gentoo/FreeBSD leaves them to be installed as extra packages, just like you’d do with Gentoo. And you can choose whichever cron software you’d like instead of using the single default provided by the system.

But, if a software is designed to build on FreeBSD, it usually builds just as fine on Gentoo/FreeBSD; rarely there are troubles, and most of the time the trouble are with different GCC versions. On the other hand, GNU/kFreeBSD require most of the system-dependant code to be ported, xine already has undergone this at least a couple of time for instance.

I sincerely am glad to see that Debian finally came to the point of accepting GNU/kFreeBSD into main; on the other hand, I have no big interest on it beside as a proof of concept; there are things that are not currently supported by glibc even on Linux, like SCTP, which on FreeBSD are provided by the standard C library; I’m not sure if they are going to port the Linux SCTP library to kFreeBSD or if they decided to implement the interface inside glibc. If that last one is the case, though, I’d be glad because it would finally mean that the code wouldn’t be left as stale.

So please, don’t mix in Gentoo/FreeBSD with Debian’s GNU/kFreeBSD. And don’t even try to call it Gentoo GNU/FreeBSD like the Wikipedia people tried to do.

More tinderboxing, more analysis, more disk space

Even though I had a cold I’ve kept busy in the past few days, which was especially good because today was most certainly Monday. For the sake of mental sanity, I’ve decided a few months ago that the weekend is off work for me, and Monday is dedicated at summing up what I’m going to do during the rest of the week, sort of a planning day. Which usually turns out to mean a lot of reading and very little action and writing.

Since I cannot sleep right now (I’ll have to write a bit about that too), I decided to start with the writing to make sure the plans I figured out will be enacted this week. Whih is especially considerate to do considering I also had to spend some time labelling, as usual this time of the year. Yes I’m still doing that, at least until I can get a decent stable job. It works and helps paying the bills at least a bit.

So anyway, you might have read Serkan’s post regarding the java-dep-check package and the issus that it found once run on the tinderbox packages. This is probably one of the most interesting uses of the tinderbox: large-scale testing of packages that would otherwise keep such a low profile that they would never come out. To make more of a point, the tinderbox is now running with the JAVA_PKG_STRICT variable set so that the Java packages will have extra checks and would be much more safely tested on the tree.

I also wanted to add further checks for bashisms in the configure script. This sprouted from the fact that, on FreeBSD 7.0, the autoconf-generated configure script does not discard the /bin/sh shell any longer. Previously, the FreeBSD implementation was discarded because of a bug, and thus the script re-executed itself using bash instead. This was bad (because bash, as we should really well know, is slow) but also good (because then all the scripts were executed with the same shell on both Linux and FreeBSD). Since the bug is now fixed, the original shell is used, which is faster (and thus good); the problem is that some projects (unieject included!) use bashisms that will fail. Javier spent some time trying to debug the issue.

To check for bashisms, I’ve used the script that Debian makes available. Unfortunately the script is far from perfect. First of all it does not really have an easy way to just scan a subtree for actual sh scripts (using egrep is not totally fie since autoconf m4 fragments often have the #!/bin/sh string in them). Which forced me to write a stupid, long and quite faulty script to scan the configure files.

But even worse, the script is full of false positives: instead of actually parsing its semantics, it only scans for substrings. For instance it identified the strange help output in gnumeric as a bash-specific brace expansion, when it was in an HEREDOC string. Instead of this method, I’d probably take a special parameter in bash that tells the interpreter to output warnings about bash-specific features, maybe I should write it myself.

But I think that there are some things that should be addressed in a much different way than the tinderbox itself. Like I have written before, there are many tests that should actually be executed on source code, like static analysis of the source code, and analysis of configure scripts so to fix issues like canonical targets when they are not needed, or misaligned ./configure --help output, and os on so forth. This kind of scans should not be applied only to released code, but more importantly on the code still in the repositories, so that the issues can be killed before the released code.

I had this idea when I went to look for different conditions on Lennart’s repositories (which are as usually available on my own repositories with changes and fixes and improvements on the buildsystem – a huge thanks to Lennart for allowing me to be his autotools-meister). By build-checking his repositories before he makes release I can ensure the released code works for Gentoo just fine, instead of having to patch it afterwards and queue the patch for the following release. It’s the step beyond upstreaming the patches.

Unfortunately this kind of work is not only difficult because it’s hard to write static analysis software that gets good results; US DHS-founded Coverity Scan, although lauded by people like Andrew Morton, had tremendously bad results in my opinion with xine-lib analysis: lots of issues were never reported, and the ones reported were often enough either false positives or were inside the FFmpeg code (which xine-lib used to import); and the code was almost never updated. If it didn’t pick up the change to the Mercurial repository, that would have been understandable, I don’t pretend them to follow the repository moves of all the projects they analyse, but the problem was there since way before the move. And it also reported every and each day the same exact problems, repeated over and over; for a while I tried to keep track of them and marked hte ones we already dealt with or which were false positives or were parts of FFmpeg (which may even have been fixed already).

So one thing to address is to have an easy way to keep track of various repositories and their branches, which is not so easy since all SCM programs have different ways to access the data. Ohloh Open Hub has probably lots of experience with that, so I guess that might be a start; it has to be considered, though, that Open Hub only supports the three “major” SCM products, GIT, Subversion and the good old CVS, which means that extending it to any repository at all is going to take a lot more work, and it had quite a bit of problems with accessing Gentoo repositories which means that it’s certainly not fault-proof. And even if I was able to hook up a similar interface on my system. it would probably require much more disk space that I’m able to have right now.

For sure now the first step is to actually write the analysis script that first checks the build logs (since anyway that would already allow to have some results, once hooked up with the tinderbox), and then find a way to identify some of the problems we most care about in Gentoo from static analysis of the source code. Not an easy task or something that can be done in spare time so if you got something to contribute, please do, it would be really nice to get the pieces of the puzzle up.

Between Mono and Java

Some time ago I expressed my feelings about C# ; to sum them up, I think it’s a nice language, by itself. It’s near enough C to be understandable by most developers who ever worked with that or C++ and it’s much saner than C++ in my opinion.

But I haven’t said much about Mono, even though I’ve been running GNOME for a while now and of course I’ve been using F-spot and, as Tante suggested, gnome-do.

I’ve been thinking about writing something about this since he also posted about Mono, but I think today is the best day of all, as there has been some interesting news in Java land.

While I do see that Mono has improved hugely since I last tried it (for Beagle), I do still have some reserves against Mono/.NET when compared with Java.

The reason for this is not that I think Mono cannot improve or that Java is technically superior, it’s more that I’m glad Sun finally covered the Java crap. OpenJDK was a very good step further, as it opened most of the important parts of the source code for others. But it also became more interesting in the last few days.

First, Sun accepted the FreeBSD port of their JDK into OpenJDK (which is a very good thing for the Gentoo/FreeBSD project!), and then a Darwin port was merged in OpenJDK. Lovely, Sun is taking the right steps to come out of the crap)

In particular, the porters project is something I would have liked to get involved in, if it wasn’t for last year’s health disaster.

In general, I think Java has now much more chances to become the true high-level multiplatform language and environment, over C# and Mono. This because the main implementation is open, rather than having one (or more) open implementations trying to track down the first and main implementation.

But I’d be seriously interested on a C# compiler that didn’t need Mono runtime, kinda like Vala.

Blast from the recent past; two years ago

Today I was feeling somewhat blue, mostly because I’m demotivated to do most stuff, and I wanted to see what it was like to work in Gentoo two years ago.

One thing I read is that a little shy of exactly two years ago, ICQ broke Kopete, just like they did yesterday. Interestingly enough, even though a workaround has been found for Kopete 0.12 (the one shipped with KDE 3.5), there is no bump I see in the tree this time. Sign that the KDE support in Gentoo has changed, most likely.

There is also the whole thing with ALSA problems, which span so many posts that it’s not worth listing all them. The current ALSA maintainer simply gave up on providing something that, at least for some users, ended up being quite important.

And all the work on Gentoo/FreeBSD! Although Javier is doing a huge work now to support FreeBSD 7.0, he’s not prone to blog about it, and you can see that Gentoo/FreeBSD is easily ending up in the “historical” memory, rather than being discussed and tried out by users daily.

What didn’t change at all is my insomnia, it’s almost 2AM and I’m still up. And this time I don’t even have Antiques Roadshow to watch. I’m currently working on xine, just like two years ago.

In general, I think a lot of areas in Gentoo did go downhill from two years ago, rather than improving. While Portage is certainly improved, thanks to Zac, Genone and the rest of the team, and we can see that in the new extended repoman checks, that also helps QA. But the general user support seems, to me, lacking.

This is a direct consequence, in my opinion, of leaving open doors for people who are just driving Gentoo’s energy away, by taking over projects to make them stall, by discussing details over and over and over, by repeating the same request even when people reject it as it stands, and so on.

I hope things will improve in the next months, thanks to a new council that can finally grow some balls, straightening up the situation, but if this does not happen, I’m already preparing for my plan B…

Sometimes you do need Linux-PAM, but mostly…

… you don’t.

I’m not referring to PAM-less systems, I sincerely don’t care much about those. I’m referring to the dependencies of packages, which sometimes require sys-libs/pam even though they work fine with any PAM implementation.

My idea at the moment is to check for packages that can be ported to virtual/pam without code changes. I already found at least two, and there probably are others. The ones not working, are usually looking for the misc_conv symbol, which is provided by the libpam_misc library “proprietary” to Linux-PAM.

This falls into two lines of work for me, the first is that I’m trying to clean the area for Seraphim Mellos, who’s working on Summer of Code to support OpenPAM, the second is for Gentoo/FreeBSD, as there Linux-PAM is not used. Additionally, the less references directly to sys-libs/pam I have in the tree, the easier it will be when I decide to move it to sys-auth/Linux-PAM as it owe to be called.

Again, I wonder why am I stuck with this task as I usually don’t need PAM at all myself. I hope that getting a token changes something, so that I’d just have to key in the PIN of the token and be done with login too, but even that it’s a bit of a stretch for a standard user.

Anyway if somebody would like to help out, it would be nice if the packages using misc_conv were ported not to need it, most software doesn’t need it anyway. At the moment I know that sys-apps/qingy, sys-apps/busybox and app-misc/away don’t support OpenPAM as they require pam_misc.

Reducing the drift between Linux and FreeBSD (in Gentoo)

One of my projects is now unleashed in the tree: pambase is now being keyworded by all arch teams so that all ~arch users can make use of it.

I’m also preparing some documentation about it, but it might take a bit more time before it is available, unfortunately the time is what it is, and tomorrow I have to go to the hospital again.

One thing that pambase is supposed to do is to make it easier for me to handle the different PAM configuration files for Linux and FreeBSD, as I can just make stuff conditional in a single configuration file rather than in multiple ones sparse on the tree.

Today I hit the first two issues where this is really important.

The first was with pam_nologin: the Linux-PAM implementation supports running it in the account chain (as needed by OpenSSH’s sshd), the OpenPAM one supports it only on auth chain. So it has to be conditional. To have the conditional inside net-misc/openssh it would be an overhead.

The second was to finally add pam_passwdqc to the default system auth stack. As it is now, by enabling the passwdqc USE flag, you add to the default passwd chain the pam_passwdqc module developed by OpenWall, which is a different analysis module from the one that you most likely got by default (pam_cracklib). It is used by default in FreeBSD, but it wasn’t present in Gentoo/FreeBSD up to now. With pambase, it will soon be present by default in Gentoo/FreeBSD, reducing a bit the difference between FreeBSD and Gentoo/FreeBSD.

You should get used to a lot of USE flags in pambase as that is what I thought it for, you can tweak most of the default configuration simply by switching around the USE flags. More will follow in the future.

Talking about PAM, I also added one more possible project to the SoC page (as well as adding a column for the contact). The new project is:

Gentoo supports already two PAM implementations, but they are tied to one operating system only: Linux-PAM (sys-libs/pam) on Linux and OpenPAM (sy-auth/openpam, plus sys-freebsd/freebsd-pam-modules) on FreeBSD. Being able to choose between either implementation on either operating system is one of the long term goals of the PAM team. The first step would be to produce a set of single PAM modules compatible with both Linux and OpenPAM that could replace Linux-PAM and its default set of PAM modules.

As it says, I’d like to be able to provide users with a choice, on whether to go with Linux-PAM or OpenPAM. OpenPAM is more lightweight. And by splitting the modules up in a number of packages, it should be more flexible for different configurations.

Other than adding modules, it would be nice to implement libpam_misc in a OpenPAM-compatible way, so that all the software that use libpam_misc could finally be used without Linux-PAM. An alternative would be to port the software that uses it not to use libpam_misc.

So if you’re interested in learning how Linux (and other Unix) authentication support works, and you’d like to be part of something entirely new, feel free to contact me to flesh out details for your application with SoC!

Why people insist on using /boot

You might or might not know that for a while now I’ve passed most of my time idling and chatting in #gentoo-it, trying to offer support whenever I can (when the user asking support deserves it at least). One of the strangely quite common type of support request involved to some extent the standalone /boot partition.

But why people insist on using a standalone /boot partition?

The /boot partition, where you add not only the grub configuration (with its stages), but also the kernels (you might, and probably should, have multiple copies), with their System.map, and optionally their configuration files, the eventual splashscreen for grub and some other stuff, was classically used to allow grub to access the kernel even on systems with a BIOS unable to allow access over the 1024 sector of an hard drive (grub can’t obviously have drivers for all the controllers, so it only uses the BIOS to access the disk). As a partition that would cross that boundary wouldn’t be properly readable by the BIOS, and thus by grub, the common solution was to put a small /boot before that boundary, and then leave the root partition to cross it, as once the kernel booted, the limitation could be ignored safely.

There are of course other cases where a standalone /boot partition could be useful, one case can be to have a way for grub to start and load the kernel, which in turn can boot with a rootfs stored on a device that the BIOS wouldn’t have been able to see (like a software raid1 or a PCI controller that couldn’t be detected); this is my reason to use a /boot on a CF memory card for Klothos: OpenBOOT doesn’t recognize the Promise SATA controller (I just have a SATA disk for that box), and thus I need to boot the kernel from an EIDE-compatible storage (in this case, the CF through an adapter). Please note that Klothos runs FreeBSD; more on that later.

Other cases where having /boot standalone can help is for half-thin clients where the kernel is stored locally, and then the rootfs is mounted via NFS: you can use a simple storage, like the CF I use, to keep /boot, and then load the rest, if the network card doesn’t support proper network boot.

But for the average user, does /boot provide any advantage? Maybe the only one is to avoid the user from deleting the kernel with rm -rf / but that’s almost useless: you would have screwed your system anyway at that point. I find it actually has a big disadvantage: if the user forgets to put it in fstab, he’d have to always mount it before running a make install for the kernel, and that’s something easily forgot.

Also, the use of a different partition for /boot confuses the hell out of some users, who don’t really understand the difference between Linux’s root filesystem’s partition and grub’s root. When I get a support request about installing grub, and I understand the user is confusing the root= parameter to the kernel and the (root hdX,Y) parameter for grub, my suggestion is to just get rid of the standalone /boot.

Not only this, it’s also difficult to decide the size for such a partition: a lot of people would use a size too small, or too big and then waste space.

Now about FreeBSD, well, it also uses a /boot directory, although it contains not only the kernel but also all its modules, and it makes it way harder to move it on a standalone partition. The FreeBSD documentation doesn’t really cover that option, and even looking around you’ll see a lot of people telling you it can’t be done, that FreeBSD ain’t Linux and that /boot is not something to move to its own partition. The truth is that sometimes you just need to do it, and you can, it’s just something much harder to do than in Linux. I had my own trouble, but then solved it.

So, while I can’t say I like FreeBSD idea of hiding the information that shouldn’t be used by the average user, I think that they are cutting out a lot of possible problems this way, and I think that Linux documentation should actively discourage average users of modern system from using a standalone /boot partition.

So my suggestion is: if you can’t name the reason why you’re using /boot as a standalone partition, then don’t use it.