Tag: System

How down can you strip a Gentoo system?

In my previous post I noted I’m working on a project here in Los Angeles, of which I don’t wan to get much into the details of. On the other hand, what I’m going to tell you about this, for now, is that it’s a device and it’s running Gentoo as part of its firmware.

You can also guess, if you know us, that since both me and Luca are involved, there is some kind of multimedia work going on.

I came here to strip down as much as possible the production firmware so that the image could be as small as possible, and still allow all the features we need on the device. This is not a new task for me, as I’ve done my best to strip down my own router so that it would require the least amount of space as possible, and I’ve also done some embedded firmwares based on Gentoo before.

The first obstacle you have if you want to reduce the size of Gentoo is almost certainly the set of init scripts that come with OpenRC; for a number of reasons, the init scripts for things like loadkeys and hwclock are not installed by the package that install the commands (respectively sys-apps/kbd and sys-apps/util-linux) but rather are installed by OpenRC itself. They are also both enabled by default, which is okay for a general desktop system but fails badly on embedded systems, especially when they don’t even have a clock.

Then you have to deal with the insane amount of packages that form our base system; without going into the details of having man, tar and so on as part of the base untouchable system (which luckily is much easier to override with the new Portage 2.2, even if it insists bothering you about an overridden system set), and focusing on what you’re going to use to boot the system, OpenRC is currently requiring you a mixture of packages including coreutils, which (a single command that lies in its own package … for ESR’s sake, why was it not implemented within coreutils yet?), grep, findutils, gawk and sed (seriously, four packages for these? I mean I know they are more widely used than coreutils, as they are often used on non-Linux operating systems, but do they really deserve their own package, each of them?).

The most irritating part nowadays for me I guess is the psmisc vs procps battle: with the latter now maintained by Debian, I wonder why they haven’t merged the former yet. Given that they are implementing utilities for the same areas of the system… of course one could wonder why they are not all part of util-linux anyway — yes I know that Debian is also supporting GNU/kFreeBSD on their package. At any rate there is another consideration to be made: only the newer procps allows you to drop support for the ncurses library, earlier depended on it forcefully, and the same is still true for psmisc.

For what it’s worth, what I decided to do was to replace as much as possible with just busybox, including the troublesome psmisc, so that I could drop ncurses from our firmware altogether — interestingly enough, OpenRC depends explicitly on psmisc even though it is not bringing in most of the rest of its dependencies.

Public Service Announcement: if you’re writing an init script and you’re tempted to use which, please replace it with type -p use command -v instead … depending on an extra program when sh already has its built-in is bad, ‘mkay?

Edit: people made me notice that type -p is not in POSIX so it does not work in Dash. I’m afraid that my only trials to run OpenRC without bash before have used BusyBox, which supports it quite fine; the option to use command -v is valid though, thanks Tim.

Oh right, of course to replace coreutils usage with BusyBox you have to be able to drop it out of the dependency tree. Sounds easy, doesn’t it? Well, the problem is that even if you’re not deal with PulseAudio (which we are), which brings in eselect-esd, as of yesterday at least every package that could use Python would bring eselect-python in! Even when you were setting USE=-python.

Fortunately, after I bitched a bit about it to Luca, he made a change which at least solves the issue at hand until the stupid eclass is gone out of the tree. And yes I’m no longer even trying to consider it sane, the code is just so hairy and so encrypted that you can’t make heads or tails of it.

There are more issues with a project like this that I can discuss, outside of those part that are “trade secret” (or rather business logic), so expect to hear a bit more about it before it’s announced full out. And many of these have to do with how easy (or not) is to use Gentoo as a basis for devices’ firmwares.

Proposal: reduce the size of system packages set

This is not an official proposal. I repeat, this is not an official proposal. It’s just an idea I had and which I should propose officially one day on gentoo-dev, but as I’m not subscribed right now and I didn’t want to yet, I’ll wait for another day to do that.

I already ranted about the fact that the dependency tree of our ebuilds is vastly incomplete, as many lack dependency on zlib; trying to get this fixed was impossible, as Donnie and other insisted that as zlib was in system, we shouldn’t depend on it at all. I disagree, and I would like to know why we can’t depend on a system package, but whatever.

Anyway, as having a complete dependency tree is almost impossible because of that, I have an alternative proposal: reducing the size of the system package set.

Right now system contains stuff like ncurses, readline, zlib, autoconf, automake and m4, perl, gnuconfig, and so on. Those are packages that certainly would be part of any base Gentoo system, but are those actual part of the system set of packages? I sincerely doubt it.

The reason of the existence of the system package set is related first and foremost to breaking circular dependencies: for instance if any package that used the C compiler would depend on gcc, then the packages that gcc depends upon would create a circular dependency between gcc and itself. Also, specifying libc in almost any ebuild would be quite pointless, as well as coreutils (or freebsd-bin/ubin) for cp, mv, install, …

But why autoconf and automake? Well the easy answer is that those are often used without making sure they are depended upon explicitly… or at least this was the case till me and Martin added autotools.eclass to the tree; nowadays all the ebuilds using autotools should have proper autoconf/automake dependency already, and if they don’t they are broken anyway. So why leaving them in system? And what about m4? m4 is not part of a common Unix system, it’s just an autoconf dependency, why isn’t it just an autoconf dependency?

For what concern the three main libraries, there aren’t that many packages using zlib directly nowadays, this is especially easy to spot on a system built with --as-needed, as without that you actually do see zlib used in every other binary, for indirect dependencies. Nor there aren’t tons and tons of packages using readline, or ncurses. Actually in my own vserver’s chroot I only found four packages using readline, none of them part of system: ruby with the readline extension (uhm I wonder if I should ask for this to become an USE flag, I certainly don’t need it and I’d rather get rid of it), psql from postgresql (which maybe it’s still good to have with readline compiled in, but I could easily get rid of), bc (which is just an e2fsprogs build-dep, and I could build without readline just fine), and mysql.

A little bit different the status of ncurses, which is used by screen, gettext (only a build-dep, not needed for runtime on Linux anyway), procps, psmisc and util-linux (and I wonder why we don’t have a switch to turn it off), texinfo (wonder why we can’t remove it entirely actually) and yet again ruby. Still, I wonder why ncurses is in system rather than being properly on the dependencies list of those packages.

As for perl, that’s probably a bit more justified, there are tons of packages using perl directly or indirectly, especially in build systems. But I would like those to depend on perl properly rather than having perl in system, as there are cases where perl is not really needed at runtime at least.

And the only users of gnuconfig are the packages making use of the old and deprecated gnuconfig.eclass, or portage’s econf. Why can’t it be a dependency of portage then?

There are probably other packages that should, in my opinion, be removed from system, but these are certainly some of the most common. Unfortunately there’s a recursive problem here: to remove the packages from system without breaking stuff you’d need a proper deptree, and to get a proper deptree you need to remove the packages from system. This is what actually stops me from proposing this right away…