For A Parallel World: ebuild writing tip: faster optional builds

Today lurking on #gentoo-hardened I came to look at an ebuild written particularly badly, that exasperated one very bad construct for what concerns parallel builds (which are a very good thing with modern multi-core multi-thread CPUs):

src_compile() {
  if use foo; then
     emake foo || die
  fi

  if use bar; then
    emake bar || die
  fi

  if use doc; then
    emake doc || die
  fi
}

This situation wastes a lot of processing power: the three targets with all their dependencies will be taken into consideration serially, not in parallel; if you requested 12 jobs, but each of foo and bar only have three object files as dependencies, they should have been built at the same time, not in two different invocations.

I admit I made this mistake before, and even so recently, mostly related to documentation building, so how does one solve this problem? Well there are many options, my favourite being something along these lines:

src_compile() {
  emake 
    $(use foo && echo foo) 
    $(use bar && echo bar) 
    $(use doc && echo doc) 
    || die "emake failed"
}

Of course this has one problem in the fact that I don’t have a general target so it should rather be something more like this:

src_compile() {
  local maketargets=""

  if use bar ; then
    maketargets="${maketargets} bar"
  else
    maketargets="${maketargets} foo"
  fi

  emake ${maketargets} 
    $(use doc && echo doc) 
    || die "emake failed"
}

This will make sure that all the targets will be considered at once, and will leave make to take care of dependency resolution.

I tried this approach out in the latest revision of the Drizzle ebuild that I proxy-maintain for Pavel; the result is quite impressive because doxygen, instead of taking its dear time after the build completed, runs for about half of the build process (using up only one slot of the twelve jobs I allocate for builds on Yamato).

Obviously, this won’t make any difference if the package is broken with respect to parallel build (using emake -j1) and won’t make a difference when you’re not building in parallel, but why not doing it right, while we’re at it?

5 thoughts on “For A Parallel World: ebuild writing tip: faster optional builds

  1. In the common case where the use flag matches the name of the make target, there is a shortcut available to simplify the command even more:

    src_compile() {    emake all $(use doc && echo doc) || die}

    can be replaced with

    src_compile() {    emake all $(usev doc) || die}

    The usev() command has been in portage since about the dawn of time, so you don’t even need to worry about the EAPI.

    Like

  2. Diego,Wonder what will happen if neither of the 3 use flags are enabled. Will the ‘make’ degenerate into ‘make all’? If not, which mechanism prevents that?Also perhaps I am missing something obvious, but what is the difference with your last example, except that foo and bar order is changed?Ok, maybe the ‘general taget’ is a term that would explain my confsion, but both google and ‘info man’ fail to shed light on this…

    Like

  3. Ah thanks Jonathan I entirely forgot about @usev@.Drybone, the last example ensures that foo is enabled if bar is not. In general, you should always have at least a target enabled when you start @emake@, so you might have to simply _default_ to one if the USE flags would tell you to have no target at all.

    Like

  4. Ah, so in that example, unlike the previous ones, you can not have foo and bar both enabled simultaneously. Not the most usual thing for USE flags, which is a point of my confusion. And I afraid checking if default target should be added is not going to be the most elegant piece of code… well, maybe clumping the targets together into a string first, and then checking if it is still empty could work – still sounds hackish.So things would go much easier if each software package contains a target like ‘all’ or ‘bin’ which could be given to emake unconditionally. If that is not possible, then perhaps the package is a good candidate to be split into several more specific independent packages. If I’ve got the world model right at this time. :)

    Like

  5. src_compile() {  local maketargets=$(usev doc)  if use bar ; then    # use foo && ewarn "bar && foo both enabled, bar kill foo" # managed in src_prepare?    emake bar ${maketargets} || die "emake failed"  else    emake foo ${maketargets} || die "emake failed"  fi}

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s