This is a particularly Gentoo-oriented instance of the For A Parallel World series, please don’t look away too much because I’m not trying to explain how to improve software in general, this time, at least not directly.
My tinderbox, of which I’ve been writing a lot almost daily, is running on an 8-way system, Yamato, a dual quad Opteron (partially funded by users last year); by running on such a system, I can usually cut down the time to build packages thanks to parallel make, but this is not always possible.
There are instances of packages that serialise build either because they are bugged and would break when building in parallel, or because they simply are bugged and disable parallel make without understanding it. This is the reason why I started this series in the first place. Now, for ebuilds that do use serial make, calling emake -j1, I’ve already asked that bugs are kept open, so that either I or somebody else can take care of reproducing and fixing the various issues, instead of going around trying to get it to work. While this works just partially, it’s still an improvement over the previous state of “if it works in serial, close the bug”.
But there are a couple extra problems: before I joined, most ebuilds that wanted to avoid parallel make used make rather than emake -j1; while the latter is usually caught by repoman which warns about an upstream workaround, the former is not. It also makes it difficult to understand whether the non-parallel make is requested on purpose, or it was just overlooked, since rarely there are comments about that.
Thanks to Kevin Pyle, I’ve been tracking down these rogue ebuilds; he showed me a very simple snippet of code that can track down the callers of direct make in ebuilds:
make() {
eqawarn "/etc/portage/bashrc QA notice: 'make' called by ${FUNCNAME[1]}"
emake "$@"
}
Thanks to this snippet I’ve been able to identify already a few packages that called make but builds fine in parallel, and a couple that require serial make or fail to build or install (bugs opened and ebuild worked around). Hopefully, on the long term this check won’t hit any longer and ebuilds will work properly in parallel. It would really be a good thing, because processors these days are increasing the number of cores faster than they achieve extra speed, and being able to build stuff in parallel, as well as execute it in parallel, is the key to reduce the time to install of Gentoo.