Rails is not for fire-and-forget hosting!

in my position of Gentoo Ruby team member (and thus Ruby packager), I’d like to give a couple more insights regarding what Jürgen wrote about on the Diaspora topic. I guess I should be considered weighting slightly more than him not only because I dislike Python myself (which seems to be the base criticism moved to Jürgen) but also because I did work with Rails, and this very blog runs on Rails!

I have some personal grudges with Rails, but in general I don’t dislike the MVC approach they take. Unfortunately, while the theory is often pretty solid, the implementation leaves to be desired a lot. A framework that suggests doing agile “Test Driven Development” and fails its own tests is not something nice to look at. But that’s not the problem at this point.

The first problem sits at the way the Diaspora developers worked on the project; someone else already dissected some of the security issues (some, not all of them, do note); and it shows that whoever wrote the code wasn’t a huge expert of Rails to begin with. Beside the number of mistakes in security ideas, working on the older Rails 2 framework is, by itself, a bad idea; considering they are targeting a “later” release, working on the newer Rails 3 branch would have reduced the possible upgrade headaches in the future.

Here comes the biggest problem of all though: Rails is far from a fire-and-forget hosting framework; while PHP has had a long history of incompatibilities between versions I don’t think it ever reached the amount of incompatible versions between one release and the next. Basically, when you write an application with Rails, unless you rely only on the very basic architecture, there is an almost 100% chance that the application will only work with the current version of Rails; full version of it, so if you write it on 2.3.5 it might not work on 2.3.8. And Typo, that I use for this blog, still only work on 2.3.8, not 2.3.9!

To complicate the matter, the single-version-compatibility problem extends not only to Rails itself (which include its own helper gems, such as activerecord) but also to their dependencies, such as Rack, and to the other gems that the application might require. This is one thing that makes maintaining Ruby and Rails packages in a distribution such a hell. To “solve” this problem, Rails 3 mandates the use of Bundler, which, well, creates bundled libraries in every Rails application. Indeed this “solves” (or to be precise, hides) the problem of different versions of packages for different applications, but at the price of possibly leaving around older libraries for older applications.

One thing has to be told about Bundler at least: it should make it much easier to update an application running on Rails, as it takes care of keeping the dependencies at the right version, without causing all the dependency hell that was happening before. Unfortunately, the dependency hell is not the only thing that makes the upgrade of a Rails application (or, for what matters, any other web application) complicated; the other is migrating the database. Rails supposedly provides tools for migrating the databases in a clean way, but often enough they only work with a subset of the database drivers Rails comes with. This is supposed not to be a problem with the “NoSQL” databases, but I’m sincerely quite sceptic about those. Finally, there is the problem of customisations, even if it’s just themes, since they tend to embed enough Ruby logic to require full porting.

This brings us to two points: updating the dependencies of a Rails app only happens when updating the app and updating a Rails app is far from trivial. This easily translates to “updating a Rails app often enough happens only when the administrator is forced to”; newer version of the interpreter breaking the older application, new hosting, new protocol versions and so on so forth.

And if you am to tell me that Rails is pretty secure a framework, please note that there are at least six bugs in the GLSA Archive since 2006, which means an average of slightly over once an year. I can’t think of many security notices even coming for third party gems, and that makes it very difficult to assess their security status.

Up to now, I don’t think I have seen much of an interest for security-evaluating third party gems, as most of them don’t have so much known use (as far as I can tell, most of the Rails applications out there are developed as closed-source, proprietary web applications, not Free applications). Those who are actually used by more widely used applications such as Typo and Redmine probably undergo more scrutiny, but even those can be considered small fishes (WordPress, Trac, Drupal definitely look much meatier targets). With the coming of Diaspora, especially with its idea of “distributed social network” (see the related post for more details), these gems are likely to become an interesting target as well. Especially the eventually older versions that the Diaspora releases will be looking into.

Do I think a different language/framework would have worked better? Not sure sincerely; I don’t like PHP for many reasons, but I have been told that newer versions have much better frameworks than what we had around version 4.2 which is the last version I actively used. Since PHP is (even more than Ruby) designed to be used for web applications, one should suppose it has gained more and more features to make it difficult to make mistakes, even when that makes it more difficult to use it for general purposes. For what I’m concerned, I doubt Python would get much better results either.

Rails is, in my opinion, a pretty good framework if you actually maintain a web application; it’s not good if you write an application for somebody to use and leave it there. By its own definition, it’s “agile” development, but “agile development” require that you actually follow it. Do you think that the average “Diaspora enthusiast” is going to follow the Agile development cycle, or will just set up an instance and stop caring about updating it when it becomes too difficult?

It’s not a matter of language or framework, it’s a matter of architecture; in a project whose basic idea was to allow each user to maintain its own instance, the choice should have gone toward software that made it difficult to misuse, mismanage or keep out-of-date instances running. In this, they seem definitely to have failed.

Humoristic note of the day: if it proves that my concerns come true, and Diaspora become a vessel for more spam, I’ll be renaming that DiaSPAMora…

Depend on what you use

New Note 20

To this day, we still get --as-needed failures for packages in Gentoo; both for new packages and bumps. To this day, checking the list of reverse dependencies of libpng is not enough to ensure that all the packages build fine with libpng-1.4 (as Samuli found out the hard way). One common problem in both is represented by missing dependencies, which in a big part are caused by transitional transitive dependencies.

Transitional Transitive dependencies are those caused by indirect linking; since I don’t want to bore you all repeating myself you can read about it in this post and this one and finally another one — yes I wrote a lot about the matter.

How do transitional transitive, indirect dependencies, cause trouble with both --as-needed and with upgrade verification? Well it depends on a number of factors actually:

  • the build might work on the developers’ systems because the libraries linked against indirectly bring in the actually needed libraries, either by DT_NEEDED or by libtool archives, but the former libraries aren’t used directly, thus --as-needed breaks the link — misguided link
  • the build might work properly because some of the used (and linked to) libraries optionally use (and link to) the otherwise missing libraries; this work just as long as they are not built without that support; for instance you might use OpenSSL and Curl, separately, then link to Curl only, expecting it to bring in OpenSSL… but Curl might be built against GnuTLS or NSS, or neither;
  • the build might work depending on the versions of libraries used, because one of the linked libraries might replace one library for another, dropping a needed library from the final link.

The same rules generally apply to the DEPEND and RDEPEND variables; relying on another package to bring in your own dependencies is a bad idea; even if you use GTK+ it doesn’t mean that you can avoid listing libpng as a package used, if you use it directly. On the other hand, listing libpng because it’s present in the final link (especially when not using --as-needed) is a bad idea which you definitely should avoid.

By ignoring transitional transitive dependencies, you invalidate the dependency-tree, which means we cannot rely on it when we’re trying to avoid huge fuckups if an important package changed API and ABI. This is why I have (wrongly) snapped back at Samuli for closing the libpng-1.4 tracker before I had the chance of running it through the tinderbox.

Bottomline, please always depend on what you use directly, both in linking and in ebuilds. Please!

Thanks to Odi for letting me know that I used (consistently) the wrong word in the article. This goes to show that I either should stop writing drafts at 3am or I should proofread them before posting.

Ruby-NG: The Dependency Society (or, why multi-package projects worry me)

I’m still working hard on my free time on the conversion of Ruby packages to the new eclasses as well as improving JRuby support over the place – for this reason I might do another shameless plug – and in particular, between yesterday and today, I ended up working on a set of packages for packaging bones in Gentoo.

Why is that a set of packages? Well, I had to commit together four different packages: bones, bones-extra, loquacious and little-plugger; a subset of them wasn’t possible at all. There are two interesting things to say about these, if you care about Ruby development. The first is that I wouldn’t actually count bad for it, as what it does is mandating an interface for Gems going a step further than Hoe, Echoe and Jeweler as it encompasses testing, documentation and packaging tasks. The second thing is that it comes from the codeforpeople project which consists of what can only be called a fuckload of different and almost unrelated packages, and as far as I know it used to be manned by a single person. Not even the most friendly of the blokes, to be honest: when I asked him a couple of questions about another project – session – the one-line reply I got was a link to a (newly created) GitHub repository.

So what is the problem here? Well, check out the depgraph on this page. I spoke about four packages before, but it lists four, one is not packaged and I’ll probably just package it if I’ll end up using Bones myself. Of the four packages, one is the “interface” one and is Bones itself, two are dependencies, and are generic libraries, the other two are “plugins” (extras and git) that provide optional functionalities; or at least sort-of-extra functionalities. Indeed, the extras plugin provides RSpec testing and RubyForge publishing support, which should probably be considered optional functionalities… on the other hand, if your package uses Bones, and defines the configuration keys for either, it will fail if the extras package is not installed. So much for the extras, then.

The runtime dependencies are actually quite linear: Bones needs the two libraries, and the two plugins need Bones, which is quite logical and very easy to deal with. But when you start considering the build-time dependencies you are way out of luck. All the packages including Bones itself would depend on Bones, and on the extras package (because all of them define the RubyForge project, and almost all use RSpec), for both documentation and testing purposes. Obviously, if you try to install Bones with USE=doc or FEATURES=test, Portage will bail out because of the cyclic dependencies. And I don’t think I can fix it in any way for now. To be honest, this would have been much easier if Bones itself contained what is in the extras, as at that point you would have everything in the same package, which wouldn’t then require itself.

Or would it? To be honest if you look at the development dependencies as listed within the gems, all of the Bones-managed gems will require Bones, and that is true… for Bones as well! Indeed, in the image above, the dotted lines represent the officially-listed dependencies which we’re not following (for some strange, but welcome, reason, the bones-git package is truly optional).

Somehow, it seems like many multi-package projects decided to go with similar ways of handling dependencies. For instance, take prawn, a PDF-writing gem: when I added it to Portage (as dependency of a webapp I worked on last summer), it was a single gem, depending on another gem by the same author for its testing. Now, it’s actually a simple wrapper gem for three split gems… and you most likely will need all of them for doing most non-trivial stuff. Or a much more common gem: Rails! The activerecord, actionpack, activesupport and actionmailer gems are so interdependent that I’m not surprised most projects just bundle Rails in their own sources (albeit this is very bad from the quality point of view).

Ruby Gems Problems: missing dependencies

As I said before, I’m doing a lot of work to port Ruby packages in Gentoo to the new eclasses so that the gems can integrate better with Portage. Integrating better in this case means respecting the build phases we have in ebuilds (unpack, prepare, compile, test, install, merge to live filesystem), respecting the dependency tree as provided by Gentoo (including split dependencies for Ruby 1.8, 1.9 and JRuby) and the doc USE flag to build or not the API documentation.

Unfortunately, this is not really such an easy task sometimes, for at least two main reasons. The first is the lack of ebuilds for the packages needed for the tests to work; the second is the lack of proper dependency information in gems.

Missing ebuilds are due to the fact that, earlier, we never ran tests so we never had the need for some of the test-specific dependencies. At the same time we never cared much about documentation, so also the dependencies needed for documentation building were never worked on. There is quite a cross-over between the two of these classes of dependencies, and those are the dependencies for the Rakefile directly (both tests and documentation generation are handled by rake for most packages).

The other problem is that the gems themselves, while having a syntax to provide the dependency information (also split between runtime and development dependencies, the equivalent of our RDEPEND and DEPEND), they are rarely complete, at least for what concerns development dependencies (which is what we care about for documentation and, even more importantly, tests). Without this kind of information, making sure that tests work is often a matter of try-and-see: you run the tests and add the dependencies if they are not installed.

Unfortunately while usually adding a new ebuild does not lead to extreme trouble, Ruby gems make it quite a hassle, especially for the problems mentioned above. You cannot rely on the specification-provided information, since as I said is often incomplete. And at the same time, the sheer amount of packages that are possibly used for tests is definitely high (there are some different documentation tools as well but not near as many), which sometimes mean that to fix the tests for one single package, you end up having to package about a dozen, sometimes with interleaved, looping dependencies.

Now, the reason why I’m blogging about this right now s that I have to ask you, the users, to help me out with this. Please try to run tests on the Ruby ebuilds, and report if the any dependency is missing. Having now dozens of installed Ruby gems, both via fakegem and the old-style ebuilds, I don’t “feel” the dependencies as much; while I do tend to check the require directives, they often are confused enough that I might not notice entirely.

So pretty please, help me help you, and check for missing dependencies together ith me and the rest of the developers, thanks!

More router improvements

My router project is the idea of running Gentoo Linux as main home gateway, on an embedded-like system (not really embedded since it’s a Celeron 2.80GHz), without Portage installed in the runtime system, and without some part of the so-called system (that I still think should be reduced).

While there are still lots of minor details I haven’t even started looking into yet, there are a few things that already started being fixed up, for instance last week I cracked down on packages that don’t set RDEPEND/DEPEND properly (hostapd was in that list and I needed it).

Today one more little fix entered the tree that was required by my router: glibc no longer rutnime-depend on binutils; or rather it does no longer need to. Previously the nscd (name service cache daemon) init script used the strings command to find out the pid file to use for each start and stop request. Since the file does not change on disk after the build, at least 2.10.1 now checks for the pidfile at install time and then replace it in the init script. Dropping the dependency.

Now I got to say that the router is working mostly fine, so I don’t think I’ll be tinkering with it for a while, at least until I get the entropy key and I’ll start packaging the ekeyd daemon. This is also due to the fact that I have to reduce the time employed in that to return to work and other more important Gentoo things. This does not mean I’ll abandon the idea of fixing the system set so that it can be vastly reduced.

Hopefully I’ll be able to entangle enough between my normal Gentoo work and the router-specific work in the future. In the mean time, I’m happy to accept gifts (especially useful stuff like the testing strips — the Italian health service only pass me 50 strips every three months, which is less than one test a day) and kudos to prod me on continuing pursuing all free software work I have scheduled.

RDEPEND safety

I’m hoping this post is going to be useful for all the devs and devs to be that want to be sure their ebuilds have proper runtime dependencies. It has sprouted by the fact it seems at least a few developers were oblivious of the implications of what I’m going to describe (which I described briefly on gentoo-core a few days ago, without any response).

First of all, I have to put my hands forwards and say that I’m going to focus on just the binary ELF packages, and this is far from a complete check for proper runtime dependencies. Scripting code is much more difficult to check, while Java is at least somewhat simpler thanks to the Java team’s script.

So you got a simple software that installs ELF executable fils or shared libraries, and you want to make sure all the needed dependencies are listed. The most common mistake there is to check the link chain with ldd (which is just a special way to invoke the loader, dumping out the called libraries). This would most likely show you a huge amount of false positives:

yamato ~ # ldd /usr/bin/mplayer
    linux-gate.so.1 =>  (0xf7f8d000)
    libXext.so.6 => /usr/lib/libXext.so.6 (0xf7eec000)
    libX11.so.6 => /usr/lib/libX11.so.6 (0xf7dfd000)
    libpthread.so.0 => /lib/libpthread.so.0 (0xf7de5000)
    libXss.so.1 => /usr/lib/libXss.so.1 (0xf7de1000)
    libXv.so.1 => /usr/lib/libXv.so.1 (0xf7ddb000)
    libXxf86vm.so.1 => /usr/lib/libXxf86vm.so.1 (0xf7dd4000)
    libvga.so.1 => /usr/lib/libvga.so.1 (0xf7d52000)
    libfaac.so.0 => /usr/lib/libfaac.so.0 (0xf7d40000)
    libx264.so.65 => /usr/lib/libx264.so.65 (0xf7cae000)
    libmp3lame.so.0 => /usr/lib/libmp3lame.so.0 (0xf7c37000)
    libncurses.so.5 => /lib/libncurses.so.5 (0xf7bf3000)
    libpng12.so.0 => /usr/lib/libpng12.so.0 (0xf7bcd000)
    libz.so.1 => /lib/libz.so.1 (0xf7bb9000)
    libmng.so.1 => /usr/lib/libmng.so.1 (0xf7b52000)
    libasound.so.2 => /usr/lib/libasound.so.2 (0xf7a9a000)
    libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0xf7a13000)
    libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0xf79e6000)
    libmad.so.0 => /usr/lib/libmad.so.0 (0xf79cd000)
    libtheora.so.0 => /usr/lib/libtheora.so.0 (0xf799b000)
    libm.so.6 => /lib/libm.so.6 (0xf7975000)
    libc.so.6 => /lib/libc.so.6 (0xf7832000)
    libxcb-xlib.so.0 => /usr/lib/libxcb-xlib.so.0 (0xf782f000)
    libxcb.so.1 => /usr/lib/libxcb.so.1 (0xf7815000)
    libdl.so.2 => /lib/libdl.so.2 (0xf7810000)
    /lib/ld-linux.so.2 (0xf7f71000)
    libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0xf77ef000)
    librt.so.1 => /lib/librt.so.1 (0xf77e6000)
    libexpat.so.1 => /usr/lib/libexpat.so.1 (0xf77bf000)
    libogg.so.0 => /usr/lib/libogg.so.0 (0xf77b9000)
    libXau.so.6 => /usr/lib/libXau.so.6 (0xf77b4000)
    libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0xf77ae000)

In this output, for instance, you can see listed the XCB libraries, and Expat, so you could assume that MPlayer depends on those. On the other hand, it really doesn’t, and they are just indirect dependencies, that the loader will have to load anyway. To avoid being fooled by that the solution would be to check the file itself for the DT_NEEDED entries in the .dynamic section of the ELF file. This can be achieved by checking the output of readelf -d or much more quickly by using scanelf -n:

yamato ~ # scanelf -n /usr/bin/mplayer
 TYPE   NEEDED FILE 
ET_EXEC libXext.so.6,libX11.so.6,libpthread.so.0,libXss.so.1,libXv.so.1,libXxf86vm.so.1,libvga.so.1,libfaac.so.0,libx264.so.65,libmp3lame.so.0,libncurses.so.5,libpng12.so.0,libz.so.1,libmng.so.1,libasound.so.2,libfreetype.so.6,libfontconfig.so.1,libmad.so.0,libtheora.so.0,libm.so.6,libc.so.6 /usr/bin/mplayer 

As you can see here MPlayer does not use either of those libraries, which means that they should not be in MPlayer’s RDEPEND. There is, though, another common mistake here. If you don’t use --as-needed (especially not forcing it), you’re going to get indirect and misguided dependencies . So you can only trust DT_NEEDED when the system has been built with --as-needed from the start. This is not always the case and thus you can get polluted dependencies. And thanks to the fact that now the linker silently ignores --as-needed on broken libraries this is likely to create a bit of stir.

One of the entries in my ever so long TODO list (explicit requests for tasks during donation helps, just so you know) is to write a ruby-elf based script that can check the dependencies without requiring the whole system to be built with --as-needed. It would probably be a lot like the script that Serkan pointed me at for Java, but for ELF files.

After you got the required dependencies are seen by the loader right, though, your task is not complete yet. A program has more dependencies that it might appear to have, since it might require data files to be opened, like icon themes and similar, but also more important dependencies in form of other programs or libraries. And that is not always too obvious. While you can check if the software is using the dlopen() interface to load dynamically further libraries, again using scanelf, that is not going to tell you much and you have to check the source code. Also the program can call another through way of the exec family of functions, or through system(). And even if your program does not call any of these functions you cannot be sure that you got the complete dependencies right without opening it

This is because libraries adds indirection to these things too. The gmodule interface in glib allows for dynamically loading plugins, and can actually load plugins you don’ t see and check, and Qt (used to) provide a QProcess class that allows to execute other software.

All in all, even for non-scripting programs, you really need to pay attention to the sources to be safe that you got your dependencies right and you should never ever rely purely on the output of a script. Which is another reason why I think that most work in Gentoo cannot be fully automated, not just yet at least. At any rate, I’m hoping to provide developers with an usable script one day soonish, at least it’ll be a step closer than it is now.

Blurring the separation between RDEPEND and PDEPEND

A few months ago I’ve written about DEPEND and RDEPEND; as Josh noticed in the comments, I’ve outright ignored PDEPEND in that instance, since I didn’t really want to open a can of worms relating to that. Now I think it would be a nice time to open that can of worms.

Portage already started blurring the separation between RDEPEND and PDEPEND; PDEPEND may be brought in before the package it’s reported in, to help breaking dependencies; which is, after all, what PDEPEND was born for, as both Petteri (in that post) and I think Brian (some time ago) repeated. I think this would be a nice starting point for trying to blurr and finally remove the separation between RDEPEND and PDEPEND.

I got htis idea while doing my tree-build for testing, the idea is that -B should work; but it really doesn’t a lot of times, since developers try to see if their software builds in general; rarely they test if it builds on a clean slate chroot; even more rarely they check it builds without its RDEPEND installed. I’ve been fixing a few of these problems in the tree, without even asking to be honest, as this usually comes down to bitrot that predates EAPI ideas (empty DEPEND meant “use RDEPEND” in the past), or to typos (REDEPEND or DEPEND=$DEPEND), so they are very trivial to fix. But this is far from optimal since I’m testing one package after the other and it’s not really so easy.

An option I proposed to Zac was to make portage have a developers-only option that installed packages treating RDEPEND as PDEPEND, so that the build of the package is triggered without having the runtime dependency installed. This was what made me think about it.

If every package was stating DEPEND and RDEPEND properly, PDEPEND wouldn’t be needed at all; the package manager would be allowed to treat packages in RDEPEND only so that installing them would be indifferent before or after the package that brought them in. Now of course this has a bit of a problem: if a dependency failed to install and was installed after the package you’d be having a non-working installed package, so I guess it’ll have to be evaluated quite more deeply, but it at least seems an idea to think about.

Of course making this change will require a new EAPI, so the proposal should be prepared as soon as possible so that it will fill in the tracks for EAPI=3, but that’s beside the point right now, I just wanted to put in writing an option that I think would be very nice indeed to simplify ebuild writing (dropping a whole class of dependencies!) and at the same time allow for proper dependency cycles breaking.

Now of course, before this can be even considered, it’d be nice if the tree was checked properly for dependencies…

Parallel emerge versus parallel make

Since I now have a true SMP system, it’s obvious that I’m expected to run parallel make to make use of this. Indeed, I set in my make.conf to use -j8 where I was using -j1 before. This has a few problems in general and it’s going to take some more work to be properly supported.

But before I start to get to those problems, I’d like to provide a public, extended answer to a friend of mine who asked me earlier today why I’m not using Portage 2.2’s parallel emerge feature.

Well, first of all, parallel emerge is helpful on SMP system during a first install, a world rebuild (which is actually what I’m doing now) or in a long update after some time spent offline; it is of little help when doing daily upgrades, or when installing a new package.

The reason is that you can effectively only merge in parallel packages that are independent of each other. And this is not so easy to ensure, to avoid breaking stuff, I’m sure portage is taking the safe route and rather serialise instead of risking brokenness. But even this, expects the dependency tree to be complete. You won’t find it complete because packages building with GCC are not going to depend on it. The system package set is not going to be put in the DEPEND variables of each ebuilds, as it is, and this opens the proverbial vase to a huge amount of problems, in my view. (Now you can also look up an earlier proposal of mine, and see if it had sense then already).

When doing a world rebuild, or a long-due update, you’re most likely going to find long chains that can be put in parallel, which I sincerely find risky, but they don’t have to be. When installing a new package, on a system that is already well installed and worked on for a few weeks even, you’ll be lucky (or unlucky) to find two or three chains at all. If you’re doing daily updates, finding parallel chains is also unlikely, as the big updates (gnome, kde, …) are usually interdependent.

Although it’s a nice feature to have, I don’t find it’s going to help a lot on the long run, I think parallel make is the thing that is going to make a difference in the medium term.

Okay, so what are the problems with using -j8 for ebuilds then?

  • we express the preference in term of (GNU) make parameters, but not all packages in Portage are built with make, let alone GNU’s;
  • ebuilds that use a non-make-compatible build system will try to parse the MAKEOPTS variable to find out the number of parallel jobs to execute; this does not always work right because there can be other options, like -s (which I use) that might make parsing difficult;
  • even -s option can be useful to some non-make-compatible build systems, but having to translate every option is tremendously pointless and boring;
  • some people use a high number of jobs because they have multiple box building as a cluster, using distcc or icecream; these won’t help with linking though, or with non-compile jobs; forcing non-compile tasks to a single job is going to discontent people using SMP systems, using a job count based on network hosts for non-compile tasks is going to slow down people with single-cpu and multi-host setups;
  • some tasks are being serialised by ebuilds when the could be ran in parallel;

And this is not yet taking into consideration buildsystem-specific problems!

What should be doing then? Well, I think the first point to solve is the way we express the preferences. Instead of expressing it in term of raw parameters to make, we should express it in term of number of jobs, and of features. For instance, a future version of portage might have a make.conf like this:

BUILD_LOCAL_JOBS="8"
BUILD_NETWORK_JOBS="12"

BUILD_OPTIONS="ccache icecream silent"

And then an ebuild would call, rather than simply emake, two new scripts: elmake and enmake (local and network), which would expand to the right number of jobs, for make-compatible buildsystems, that is. For other build systems eclasses could deal with that by getting the number of jobs and the features from there.

More options might be translated this way without having to parse the make syntax in each ebuild, or in each eclass. The ebuilds could also declare a global or per-phase limit to jobs, or a RESTRICT="parallel-make", that would make Portage use a single job.

The last point is probably the most complex one. Robin already dealt with a similar issues in the .bdf to .pcf translation of fonts, and solved it by having a new package provide a Makefile with the translation rules, the conversion could then be parallelised by make, instead of being serialised by the ebuild. I think we should do something like this in quite a few cases; the first one I can think of is the elisp compile for emacs extensions, and I don’t know whether Python serialises or execute in parallel the bytecode generation when providing multiple files to py_compile. And this is just looking at two eclasses I know doing something similar to this. But also Portage’s stripping and compressing of files should probably be parallelised, where there are enough resources to do so locally.

I guess I have found yet another task I’ll spend my time on, especially once I’m back from the hospital.

When you should use RDEPEND and when you should use DEPEND

Maybe it’s somehow not clear in the documentation, as I see this kind of mistakes happens quite a lot. The obvious reason is that sometimes it happens you oversee something, but as a lot of these problems happen on packages maintained by the same person or group, I start to think this is not clear enough. In particular I’m afraid the reasoning why you should make sure that the two variables are filled in properly is not clear enough.

First of all, at the moment we unfortunately have only two dependency variables, DEPEND and RDEPEND. Cross-compiling and proper multilib support would need two more, splitting both in executed and linked-against packages, as the first just need to be available in some executable form, while the latter need to have the same ABI as the program being built. But that’s a story for another day.

So we’re back to DEPEND and RDEPEND. The first is the set of packages that has to be available during build, as this is what we have, it has to have both tools used to build, and library linked against. RDEPEND instead is the set of packages that has to be available during execution of the program. This means it has to have the libraries linked against, if shared – which is what we want with pretty rare exceptions – and the programs that are ran by the program at runtime.

Programs like unzip and other tools used to extract the source archive as DEPEND and not RDEPEND. Unless they are executed by the software itself, like Amarok does (alas), in which case they are RDEPEND and not DEPEND. Build tools, like pmake or build systems like imake, scons or cmake, generators like flex and bison, rust or swig are DEPEND only, as they are used during source build. Software used for the tests is DEPEND only and subject to the test USE flag.

Fonts, when used by a graphical application, are most likely useful only at runtime. Rarely tools require them at buildtime, but it happens, for instance when you use gd to generate some images for the documentation. By the way, Doxygen is a build-only dependency, and should always be subject to the doc USE flag!

Most, but not all, packages for interpreted languages like Perl, Python and Ruby extensions have little or no build dependency (they are not built, most of the times) but may have lots of runtime dependencies. And might have build-time dependencies under test USE flag for the tests.

When a package install no shared library, it’s usually just one of the two between DEPEND and RDEPEND, as rarely the software checks at build-time what it will execute at runtime, which is quite good for us, as we’ll see. If the configure script for a package just checks if the tools are available, but doesn’t use them, it’s a good idea to either fool it through cache values, or change it so that the tests are not done. Again, we’ll see why.

So okay, why should we spend time to make sure the dependencies are right? Shouldn’t people install the packages anyway for both build and runtime? Well, sort of. When you prepare a filesystem image for another system through ROOT=, you usually just want the packages needed at runtime, not at buildtime. This means you don’t want autotools, you don’t want pkg-config, you don’t want lzma and so on. Often this is done through cross-compile, and this saves you cross-compiling for instance Perl – which, if you didn’t know, is not cross-compilable at this stage – but it might also apply to systems that are maintained through binary packages, as you don’t want to have to distribute extra packages, especially if the target machines don’t have as much space as you’d like them to be.

But having proper DEPEND/RDEPEND also helps testing. For instance, if I’m to check whether a given package builds after a flex bump, I don’t want to have to install all its runtime dependencies, as that would stop me from using the -B option of emerge to try just the build.

This also means that if you know for sure that the package calls the linker with a given library at build-time, but the library is never ever used (and for instance --as-needed drops it), you should be patching the package not to try to use it at build-time, or use it only conditionally, so that you don’t depend on a package that is not useful at all.

Please, when you write the dependencies of an ebuild, think a lot how to get them right, make sure that datafiles are only runtime dependencies unless they are used for build stuff too, make sure that build tools are not leaked at runtime. Basically, look at the package from every angle and make sure that you actually get the thing how it is supposed to be.

Today’s mass filings

Our bugzilla today really had an hard day, I’ve started filing bugs for build tools added to runtime-depend of packages, the list is awfully long, and “My Bugs” search now lists over 300 bugs.

Why did I do this? Well, we really need to keep the tree clean on these things so that --depclean works as expected. But mostly my interest is for final system deployment. On my vserver, as well as on a final target for a cross-compiled embedded system, you don’t want packages like flex, bison or swig, and this is quite possible if the packages are not in RDEPEND, when using ROOT= or binary packages.

Now, the work today was boring, long and a huge pour of time. I did it gladly without being forced to, I’m fine with it and I’m not expecting anything in particular for what I did. If I didn’t want to do it, I could have just reported the issue, and let someone else doing the thing. This is what volunteer-based work is about.

Why do I say this? Well there seems to be people to think that even volunteer work should apply the same rules as businesses. While sometimes you might make use of strategies designed for businesses, like marketing your project better and similar, volunteer works and businesses have vast differences.

Sure there are free software projects ran as businesses, but that usually involves people paid to do their job as the core rather than full volunteer-based approaches. So please if you think to run a volunteer project as a business, well think twice or expect serious trouble from the volunteers.

So please remember that there are lots of people who are fine to work on a volunteer project that would probably start to be quite nervous if you run it as a business, okay?