This Time Self-Hosted
dark mode light mode Search

Parallel emerge versus parallel make

Since I now have a true SMP system, it’s obvious that I’m expected to run parallel make to make use of this. Indeed, I set in my make.conf to use -j8 where I was using -j1 before. This has a few problems in general and it’s going to take some more work to be properly supported.

But before I start to get to those problems, I’d like to provide a public, extended answer to a friend of mine who asked me earlier today why I’m not using Portage 2.2’s parallel emerge feature.

Well, first of all, parallel emerge is helpful on SMP system during a first install, a world rebuild (which is actually what I’m doing now) or in a long update after some time spent offline; it is of little help when doing daily upgrades, or when installing a new package.

The reason is that you can effectively only merge in parallel packages that are independent of each other. And this is not so easy to ensure, to avoid breaking stuff, I’m sure portage is taking the safe route and rather serialise instead of risking brokenness. But even this, expects the dependency tree to be complete. You won’t find it complete because packages building with GCC are not going to depend on it. The system package set is not going to be put in the DEPEND variables of each ebuilds, as it is, and this opens the proverbial vase to a huge amount of problems, in my view. (Now you can also look up an earlier proposal of mine, and see if it had sense then already).

When doing a world rebuild, or a long-due update, you’re most likely going to find long chains that can be put in parallel, which I sincerely find risky, but they don’t have to be. When installing a new package, on a system that is already well installed and worked on for a few weeks even, you’ll be lucky (or unlucky) to find two or three chains at all. If you’re doing daily updates, finding parallel chains is also unlikely, as the big updates (gnome, kde, …) are usually interdependent.

Although it’s a nice feature to have, I don’t find it’s going to help a lot on the long run, I think parallel make is the thing that is going to make a difference in the medium term.

Okay, so what are the problems with using -j8 for ebuilds then?

  • we express the preference in term of (GNU) make parameters, but not all packages in Portage are built with make, let alone GNU’s;
  • ebuilds that use a non-make-compatible build system will try to parse the MAKEOPTS variable to find out the number of parallel jobs to execute; this does not always work right because there can be other options, like -s (which I use) that might make parsing difficult;
  • even -s option can be useful to some non-make-compatible build systems, but having to translate every option is tremendously pointless and boring;
  • some people use a high number of jobs because they have multiple box building as a cluster, using distcc or icecream; these won’t help with linking though, or with non-compile jobs; forcing non-compile tasks to a single job is going to discontent people using SMP systems, using a job count based on network hosts for non-compile tasks is going to slow down people with single-cpu and multi-host setups;
  • some tasks are being serialised by ebuilds when the could be ran in parallel;

And this is not yet taking into consideration buildsystem-specific problems!

What should be doing then? Well, I think the first point to solve is the way we express the preferences. Instead of expressing it in term of raw parameters to make, we should express it in term of number of jobs, and of features. For instance, a future version of portage might have a make.conf like this:

BUILD_LOCAL_JOBS="8"
BUILD_NETWORK_JOBS="12"

BUILD_OPTIONS="ccache icecream silent"

And then an ebuild would call, rather than simply emake, two new scripts: elmake and enmake (local and network), which would expand to the right number of jobs, for make-compatible buildsystems, that is. For other build systems eclasses could deal with that by getting the number of jobs and the features from there.

More options might be translated this way without having to parse the make syntax in each ebuild, or in each eclass. The ebuilds could also declare a global or per-phase limit to jobs, or a RESTRICT="parallel-make", that would make Portage use a single job.

The last point is probably the most complex one. Robin already dealt with a similar issues in the .bdf to .pcf translation of fonts, and solved it by having a new package provide a Makefile with the translation rules, the conversion could then be parallelised by make, instead of being serialised by the ebuild. I think we should do something like this in quite a few cases; the first one I can think of is the elisp compile for emacs extensions, and I don’t know whether Python serialises or execute in parallel the bytecode generation when providing multiple files to py_compile. And this is just looking at two eclasses I know doing something similar to this. But also Portage’s stripping and compressing of files should probably be parallelised, where there are enough resources to do so locally.

I guess I have found yet another task I’ll spend my time on, especially once I’m back from the hospital.

Comments 7
  1. The biggest problem with parallel make here is the configures, which tend to be single-thread and with parallelized ccached compiles can take as much time as the compiles.The problem I found with parallel emerge is that in an emerge –emptytree world, the first hundred or so packages are more or less parallel, but then it goes 100% single-job for the other 500 or so. =:^(BTW, on my 2x dual-core Opteron here, 8 gig memory (which your machine pretty much doubles, to 2x quad-core and 16 gigs, IIRC), after pointing PORTAGE_TMPDIR at tmpfs so I’m not I/O wait-queue bound and with the kernel set to per-user scheduling and PORTAGE_NICENESS=19 (thereby getting the idle class batch scheduling), I find I’m not CPU job limited at all — I can go a couple hundred jobs without unduly negatively influencing responsiveness. Rather, I’m much more limited by memory. As long as I don’t go more than a gig or so into swap (which is striped on 4 physical spindles), responsiveness is fine. Get over that, and responsiveness does suffer some, during the swapping periods. I’ve found I can reliably run 16 jobs or so, without overly stressing memory. Naturally I can often run way more than that (a couple hundred jobs on a kernel compile, for instance), if they don’t overly stress memory, but with packages like kmail where a single job takes over a gig of memory at one point, I keep it to 16.With IIRC 16 gigs memory, you should be much less limited, and can probably easily run 32-jobs without issue.BTW2, I’ve found occasional apparent race issues with the job-count. If I set jobs, I’ll occasionally get builds erroring out with job tokens should be X, but were (X+1) type errors. Therefore I’ve found that using the -l switch works better. I use -j -l16 or there abouts, so unlimited jobs, but don’t start new ones if the load average is above 16 (which from my observation is about what it takes to get full CPU utilization, due to the load average being one-minute based). That way the job token count doesn’t get screwed up and abort the build! =8^)Maybe if you start hitting those job token errors, tho, you can dive into the code and figure out where that race condition is and fix it. That’d be nice. =8^)And while you are at it tho it’s a kernel thing I’m sure, a minimum 1-minute load average might have been appropriate back when CPU clocks were running in the kHz, tolerable with them in the MHz, but now in the multiple GHz, having something like a 6 or 10 second load average minimum would be quite useful. If there was some way to get and hook into that, both for display and for stuff like make’s -l parameter, it’d be very useful. I hate seeing CPUs graphing idle even when the load average is high enough to avoid kicking in additional jobs!

  2. For what concerns @./configure@ calls, that’s a problem with most autotools as the are used by packages. The ideal here would be if every maintainer tried to contact upstream and get them to improve their autotools-fu, but I know this is hard to reach :/

  3. “The reason is that you can effectively only merge in parallel packages that are independent of each other. And this is not so easy to ensure, to avoid breaking stuff, I’m sure portage is taking the safe route and rather serialise instead of risking brokenness. But even this, expects the dependency tree to be complete. You won’t find it complete because packages building with GCC are not going to depend on it.”Absolutely, but that’s not any better if you run without parallelism, I’ve done emerge -Dup world and before and noticed wine building before gcc, which of course doesn’t make much sense.Fact is the only way you can solve that is a emerge -Dup system first, and even then you might not solve it for higher level compilers, that are not part of the system set.Maybe it’s time for a “compiler” keyword? That’s always updated before everything else in an emerge update?

  4. Check again the other post I linked, IMHO the proper solution is to start having a more detailed deptree and reduce the size of the system set. That will start solving issues quite a bit.

  5. Maybe time for at [TRACKER] for ebuilds with workarounds (i.e. “emake -j1″, or MAKEOPTS=”-j1″ set)? Many of them are simple fixed if you know your autotools (Makefiles using wrong hooks) and some of them are not even required anymore since they are fixed upstream (true at least for some GNOME-packages from time to time), but the workaround has followed the ebuild bumping.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.