For A Parallel World. Improvements: make call checks

This is a particularly Gentoo-oriented instance of the For A Parallel World series, please don’t look away too much because I’m not trying to explain how to improve software in general, this time, at least not directly.

My tinderbox, of which I’ve been writing a lot almost daily, is running on an 8-way system, Yamato, a dual quad Opteron (partially funded by users last year); by running on such a system, I can usually cut down the time to build packages thanks to parallel make, but this is not always possible.

There are instances of packages that serialise build either because they are bugged and would break when building in parallel, or because they simply are bugged and disable parallel make without understanding it. This is the reason why I started this series in the first place. Now, for ebuilds that do use serial make, calling emake -j1, I’ve already asked that bugs are kept open, so that either I or somebody else can take care of reproducing and fixing the various issues, instead of going around trying to get it to work. While this works just partially, it’s still an improvement over the previous state of “if it works in serial, close the bug”.

But there are a couple extra problems: before I joined, most ebuilds that wanted to avoid parallel make used make rather than emake -j1; while the latter is usually caught by repoman which warns about an upstream workaround, the former is not. It also makes it difficult to understand whether the non-parallel make is requested on purpose, or it was just overlooked, since rarely there are comments about that.

Thanks to Kevin Pyle, I’ve been tracking down these rogue ebuilds; he showed me a very simple snippet of code that can track down the callers of direct make in ebuilds:

make() {
        eqawarn "/etc/portage/bashrc QA notice: 'make' called by ${FUNCNAME[1]}"
        emake "$@"
}

Thanks to this snippet I’ve been able to identify already a few packages that called make but builds fine in parallel, and a couple that require serial make or fail to build or install (bugs opened and ebuild worked around). Hopefully, on the long term this check won’t hit any longer and ebuilds will work properly in parallel. It would really be a good thing, because processors these days are increasing the number of cores faster than they achieve extra speed, and being able to build stuff in parallel, as well as execute it in parallel, is the key to reduce the time to install of Gentoo.

3 thoughts on “For A Parallel World. Improvements: make call checks

  1. When you blog and say you opened bugs on the Gentoo bugzilla regarding the issue you discuss, could you please link to a way to find those bugs? (a keyword search or a meta tracker bug or something else). It would be good to see specifics of the issue you explain and also (since I know the number of bugs is sometimes staggering) increase both the appreciation for the amount of work you do and maybe the eagerness to participate and help out with some of them (especially if it’s a package the reader actually knows/uses).

    Like

  2. I agree with Zeev Tarantov as it’s quite hard to dig into the idiotic Bugzilla’s search engine without lose mental sanity and your bugs are spreaded all over different trackers, making impossible to follow them with a simple search: http://tinyurl.com/mo3oxw

    Like

  3. I’m actually worried on the load on Bugzilla to write in the post itself the URL, since it takes quite a while for it to generate the right page; but the correct list would be http://tinyurl.com/l84hwn rather than what you have above.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s