There is one interesting differnce between Linux and full operating system projects like FreeBSD, the other BSDs and OpenSolaris: Linux historically didn’t have much coordination between kernel and userland.
This becomes a problem for instance when udev and the kernel disagree on how to handle something, or on when you end up with a tool trying to use some old kernel interface.
It looks tremendously bad when you see that even strace fail (which on FreeBSD does not seem to happen, as ktrace is part of the single project). And I don’t know of any strace replacement.. and I relay a lot on that tool!
This gets further interesting when you add in the USB access through /proc (usbfs), that has been deprecated a long time ago, but that most people are probably still enabling in their kernel. The new interface, using /dev is available for quite a while, and libusb is supporting it very well. But as it turns out, VMware does not use libusb for accessing the USB devices, and it does not suppor the new interface.
I wonder how many projects have this problem. I remember net-tools being worked on, iproute2 replacing ifconfig and so on… but how many tools are actually always in sync between kernel and userland, as of now?
ALSA is also a very common problem with this as the drivers in the past often ended up out of sync between kernel and driver, causing subtle and obnoxious problems.
And even counting software tools that are well in sync between the two, how many of these tools are being audited for, for instance, performance improvements? I wonder.
I’m afraid this is a blog post without solution, but I’d like to make people think about this, maybe someone can help finding solutions 😉