Again the crazy overlays idea

Daniel seems to come up again with the idea of splitting the tree in multiple overlays. I’m not sure why people still think of this from a point of view for which there are only advantages in breaking up the tree, when it’s quite easy to come up with a lot of reasons why the tree shouldn’t be shattered up. It’s not the first time I try to show the downsides of breaking up the tree, you can find a post from last october if you want to read it, or look at my answer to Seemant that was about a month before that.

But maybe, some better real life example will show the problem more easily. As you might or might not know, this month I tackled over a quite old problem, the use of debug.eclass and its policy violations; that eclass was used by about forty packages in the main tree, which were fixed in just a night by me, Alec and Ryan; not a big deal, it was just matter of removing it from the inherit line, and add the debug useflag if it was needed but not already in IUSE.

Still, the effect of my “whining mode” on the eclass is being heard again and again because many overlays had packages which inherited debug.eclass, sometimes even eclasses like XFCE’s. With a single main important tree, fixing the packages in main portage was enough; if we were having a ton of overlays or repositories with all the important stuff, to introduce a single change I’d have to pass over them one by one and make sure that there are no ebuilds that needs to be fixed.

And taking sci-* categories as an example, as Daniel did, when I started my work to make sure that gnuconfig_update use was limited to the exceptional cases where it has to be used, it was important for me to find that at least one of the scientific packages is a peculiar case when it comes to configure scripts, as that made me assess better the need for completing autoepatch.

One more: I’ve been looking at the whole tree in the past days to make ebuilds use mirror:// whenever possible rather than the url of GNU’s or Debian’s FTP – this is useful for developers during version bumps, and to users when they either don’t have a fast enough Gentoo mirror at hand or when they are downloading the sources of an old ebuild no more available on our mirrors – it’s a feasible task to do with a single tree, it’s a pain in the ass to do on a high number of overlays.

Amarok’s live Subversion ebuild is another good example: amarok-svn ebuild is referred in a lot of documents, forum posts and so on, and it’s in a number of different overlays; all of them got out of sync with my official release ebuild sooner or later, to the result that those ebuilds does not work if you’re lucky, or will mess up your system if you’re unlucky; having the live Subversion ebuild just to the side of the release ebuild make it wasy easier to keep them in sync, and they won’t screw up with the user again.

The behavioural changes I mentioned in my past blog posts can be also seen applied to a real situation: when I bumped media-plugins/live earlier this month, I made it change entirely the way it is installed on the final system: instead of creating a built tree with the headers and the static (PIC) libraries, I’ve made it install in the usual libraries’ and headers’ directories, with shared versions of the libraries, as well as static (non-PIC) ones. As the change was pretty big, I had to check the reverse dependencies of live: all three of them (of the ones in the tree at least) needed to be changed somehow, VLC and Mplayer just needed changes to the ebuilds, while the other one I forgot the name of (sorry) required changes to the Makefile too.

If we had more than one tree with multimedia software, I would have to check all of them.

Similarly, I’m going to check most of the tree against my experimental alsa-lib soon… I don’t want to think how it would be to check it on something like twenty different overlays.

Eclasses changes are a problem too, even when they are mostly compatible. For instance when I introduced the ARTS_REQUIRED variable as a parameter to kde.eclass, I looked up the tree for which packages were dying, and converted them to the new method; when I decided to introduce the ability to update the admin directory semi-automatically, I also looked up the tree for which packages could make use of that feature.

And, there are still the other problems I wrote about, like the clashes between overlays for the same package (think if both Amarok and Banshee required the same library, but they also both need two different non-exclusive options to be enabled.. if they are on different overlays, the maintainers would likely just put on both one version of the library with the required option enabled… but then the users couldn’t have both Amarok and Banshee on the same system), the possible obstructionism from maintainers of shared packages that couldn’t care less about packages from other overlays because they don’t like their topic, the interdependencies of the overlays (where does Amarok go? Multimedia overlay or KDE overlay? Or would we need a KDE Multimedia overlay, a KDE Development overlay, a KDE Games overlay).

Yes, the tree size is critical, and I’m all for removing dead, unused or unusable packages, and I agree that 63 ebuilds for one single e-zine is a broken way around the problem, but I disagree with breaking the tree up to fix the problem of the size.

That problem is fixed in different ways, for instance cleaning up old versions of the packages (thing that I always try to do), avoid just bumping a package and leaving all the older minor versions there if there are no visible regressions, using patchset tarballs rather than sticking 40KiB of patches on $FILEDIR, and considering when the packages should be conglomerated.

I disagree that Portage is no place for documentation or eyecandy packages, it vastly depends on what the packages are: sets of icons, themes, wallpapers, where a single package carries a load of data, is something that would be fine as long as there are people willing to maintain them. Of course adding one ebuild to install one wallpaper is a bad idea.

Sure, overlays are great if you want to add a patch to a package that you know will not be merged in Portage (because upstream disagrees for instance, or because the patch is massive and will just be present on the next release – say “Hi”, Amarok ebuild from my overlay), or for developing new packages that need to be tested and can change quickly (think of the way Java team used their overlays), but they are no universal solution.

Beside, even if I’m an user of just a couple of sci-* packages (sci-electronics, for what it’s worth saying), I don’t really find that a proper example of category that could be moved to overlays, at least not without citing other things like Games, or Perl, Python and Ruby extensions, or webapps, … I wouldn’t point fingers, because usually that happens toward those packages you don’t like or you’re not interested in, but that is not always shared among a lot of people.