Metadata

Mailbox: about the debug USE flag

I’m back discussing the backtrace guide that I wrote some time ago. Fredrik asks by mail:

[…] after reading the bt-article I’m starting to think that the debug USE, more or less simply enables the developers own debugging stuff (like noisy extra output during run-time, assertions (which, afaik, are removed by -DNDEBUG which isn’t that uncommon in “release mode” cf. “debug mode”), etc).

So, my question is simply: if I want debugging symbols I should stay away from the debug USE and only add “-ggdb” to my CFLAGS/CXXFLAGS and put splitdebug in my FEATURES?

The answer is not as simple as I’d like it to be. Fredrik is definitely right into assessing that the debug USE flag, at least in most cases, is used to enable “debug mode” builds. Assertions, debug verbosity, extra safety checks, … all this kind of stuff, that is useful for developers during debug session, and totally useless (or even harmful) for production builds, can be tied to the debug USE flag. But, since we’re Gentoo we cannot do things this cleanly. Unfortunately.

While the QA team, as far as I know, agrees on the meaning of debug USE flag, not all developers seem to accept that, so you may well have multiple meanings of it right now:

enable the debug code, like I stated above: logs, assertions and so on;
enable debug symbols generation appending -g or -ggdb to the build flags;
disable optimisation by ignoring the CFLAGS variable.

Or most likely, a combination of the above. Note that the official meaning has been, for a long time, the following:

debug – Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see http://www.gentoo.org/proj/en/qa/backtraces.xml

The reason why the QA team would like for the USE flag to only handle the first case is that the other two make little to no sense in the context of Gentoo. Even though you ask the build system to emit debug symbols with -ggdb, without setting the proper features into portage, they’ll be stripped off before merge to live filesystem (and don’t even think of using RESTRICT=strip!); toying with CFLAGS is also a very bad idea because you’re second-guessing the user (and we have enough ways to handle per-ebuild CFLAGS to deal with that). Enabling the debug code is instead something that only an USE flag can do cleanly.

Unfortunately it’s not just the opinion of the QA team or of the maintainers that are on the table here: some upstream developers, like the Drizzle developers, seem not to understand that backtraces can be valid even when using optimisation (at least up to -O2), or that you can have useful information out of a backtrace even without enabling assertions and similar problems. Actually, to find out why something crashed in production, you want the backtrace to be produced without the debug code, otherwise you’re debugging two different programs!

Now, I sincerely think that the description for the debug USE flag is definitely too generic; but you cannot give more details about it without becoming specific to one package or one set of packages; fortunately me and Doug solved that some time ago – over two and a half years ago to be precise – by allowing per-package extended USE flags descriptions in metadata.xml files. And indeed there are quite a few packages that do document it. In one case (app-portage/eix) even admitting they are doing it wrong.

I could write a whole post about how I would expect app-portage packages to be a notch over the rest of the tree, given that are packages written specifically for Gentoo, and should know and comply with the policies we set out for packages. The fact that at least two fails to do that, is a bad sign.

It is important to write some notes about the -g flags here, before people get confused. You can neither always disable them nor always enable them in Gentoo itself. The latter is a common assumption I read people thinking on: “since Gentoo already strips the debug information, why not always emitting it and leave the user just to tinker with the stripping features?”, well the answer is actually easy to understand; emitting the debug information is not a trivial task, as the compiler needs to write more data to the assembler that needs to compile it into the DWARF code (and Arnaldo from the dwarves/perf projects has shown me how messed up that is, trust me, it is a lot). It might sound very easy when you look at very simple source files, but it becomes tremendously complex when you have thousand-lines source files or, worse, C++ source files with templates. Not only it’s a lot of work for the assembler (and linker) but it also increases the size of the temporary object files (the products of compile+assembly) and of the final shared objects and executable files, before stripping; if you build in RAM, this means you’ll need more memory to store the build directory; if you build on disk, it increases the amount of stuff that gets written to and read from the disk, slowing down the process quite a bit in some cases.

At the same time, we cannot use (or suggest the use of) -g0 to disable the whole debug information generation. While the idea looks nice, and I also thought about it a bit, I wouldn’t be surprised if some packages actually made use of basic debug information; of course in those cases you’d also have RESTRICT=strip (and there are a number of those packages in the tree already). So you might want to try it by yourself but.. you’re on your own if you shoot yourself in the foot with that.

One could ask here why the debug USE flag is so package-specific, and why one shouldn’t just use -DNDEBUG to disable all the asserts to begin with. The answer is that, unfortunately, a huge amount of software fails to work when assertions are disabled, because they consider their asserts as the main error handling feature, while asserts should really just be used to verify situations that will not happen. Unfortunately, once you disable those asserts, they have no graceful handling of errors, so the software might crash badly or – much worse – corrupt the data before crashing. So before deciding whether you can add the debug USE flag, you got to find out whether the package supports doing so.

Also, you might have to note that not all the assertions are disabled by defining NDEBUG; for instance, g_assert() from glib is disabled via another macro (which if I recall correctly should be G_ASSERT_DISABLE or something along those lines); but even that might not be a good idea to disable, unless you know what you’re doing).

I sincerely try my best to have both the upstream packaging and the ebuild abiding by this rule: --enable-debug in feng adds some further debug messages, and adds function that fully free the resources before exiting (useful to remove false positives from valgrind), while --disable-debug removes all of that and all the assertions.

At any rate, the bottom line is that, as usual, Gentoo lacks consistency in its design and decisions; we really should start applying this kind of consistency to new packages as they are added, but I fear most reviewers, including the Sunrise reviewers, will forget about these details. I’m not asking perfection here, but at least having a clue?

Gentoo packages and co-maintainership

One of the problems that QA often face is making sure that there are maintainers for packages; this is easier said than done, because just having someone to bump the packages or look at the bugs most of the times is not enough; you need somebody who run the code, and makes sure it behaves as intended.

Indeed, especially given the bad track that testsuites have the (unfortunately common) “it compiles, ship it!” paradigm is a very bad one to follow for what concerns quality. Even if the testsuite might take care of ensuring the basic functionality of the software is good, it doesn’t help to know whether the integration of the code is correct.

This gets even hairier when the packages interact with many other components, such as network systems, hardware, or third-party software (which is the case both for code generation and most libraries). The testing we apply in ebuilds cover just a fraction of all those cases, and you can see how bad that can be with libpng or berkdb both of which lately have caused quite a stir, one for all users, and the other just for the tinderbox, luckily. And neither actually relied on having and using special hardware!

Now, in this case, often enough, you end up having the packages being maintained directly by one person, that either has the hardware, or knows how to handle the package, by making use of the library, generator, service, or whatever the heck the package needs to have. This is good from one side because bothering a team with a package that requires specific hardware is not going to be any helpful, but on the other hand introduces a single point of failure: if the maintainer is not available, you get stuck with a package that cannot be tested, and thus properly maintained. And sometimes, that’s not strictly necessary.

Unfortunately, we get somewhat “possesive” and territorial on our packages, many many times, and the result is that most packages have at most one maintainer, or one maintainer and a proxy. I have, as of late, instead decided to make it more viable to have multiple maintainers, two, or even three or four, for the same package. This is not really helped by bugzilla only allowing a single assignee, but it can still work out, somehow.

Two examples here are two packages I co-maintain with Luca because of LScube – dev-util/ragel and net-misc/lksctp-tools – the former (which is actually quite used in Ruby-based packages, now that we rebuild them properly) is a state machine generator, and its testsuite is unusable because it needs software we don’t have in Portage (and that is unlikely we’ll ever have, given that it’s not proper FLOSS); the latter is the userland package for the SCTP protocol implementation in the Linux kernel, which we support as part of feng and libnemesi (and, hopefully, FFmpeg very soon).

We use our own code to make sure that these two work as intended, it might not be covering all of the possible problems, but it sure as the dependency hell that we can do a more thorough work than just randomly bumping the packages. It doesn’t make sense for either of us to be territorial about the packages, given that we work together on the same code, we can do almost the identical tests on it, so the solution for both is sharing the maintainership.

Another package I am listed as co-maintainer in is hplip: I have the hardware, so I can check whether it works, on the other hand, I don’t use all of it (I only use the scanner function of the all-in-one: the printer sucks for the most part, and I haven’t really needed to print something in full colour in a long time now — if I do need something printed in colour, I just call my print shop up). So I’m listed there among the maintainers, but I only do limited work, on the other hand, I’m “on hold” if something comes up and nobody else can take care of it.

Don’t be scared to share a package! Think of it like car-pooling or sleeping with a friend in the same bed if you’re on vacation. It’s fun, and the more the merrier.

2010-06-09 by Flameeyes English Technical 1

Same content, different file

I’d first like to apologise for my previous post ending up on Planet Gentoo rather than just Gentoo Universe: a click too much. On the other hand, I would like to note that if one post gets mis-categorized, with an average of 300 posts an year, I don’t think I deserves the words I received in the comments from one very lunatic user. Thanks to those of you who supported me, really!

You might remember that I have written a few posts about the Portage tree size after which I tried with the help of equilibrium from GeCHI a few possible solutions to reduce the overhead. One of these was a systematic approach to what a few users suggested in my posts: SquashFS.

While the final results aren’t really that much of importance for what I’m writing here, the test with SquashFS made it clear that there are indeed a lot of duplicated files in the tree, which is something I also said before. Unfortunately, it’s not really that easy to understand how they are supposed to work. Let me try to explain.

The duplicated files that we don’t want in the tree are those created by the insistence of developers to always use ${P} or ${PV} in their patch file names (or subdirectories of files/ even!). Instead of editing the ebuild to replace the version with the previous version, on bump the patches gets duplicated. While I’m hopeful that rsync can handle the duplicated files efficiently, they waste on-disk space and cache entries in the VFS cache (which is probably even more important in some cases, when going through the tree).

The duplicated files that we have lots of, and we’re forced to have in tree, mostly related to profiles: the parent file is usually either .. or a common relative path for quite a few different profiles. It’s bad, somehow, because they are very small files (<100 bytes) and yet will end up eating at a minimum 512 bytes of space (with obvious exceptions).

The kind of duplicated files we need more of is another instead: metadata.xml files. There are quite a few “homograph“ metadata.xml: they contain the same herd/maintainer data, but they have difference in indentation, whitespace, and other similar trivial things. Why do I think we need more duplicated metadata files? Well it’s simple: to improve possible data de-duplication. If the files have the same meaning, but different literal content, they will be always two distinct files.

On the other hand, if two files have the same literal content (md5 is the same for instance), this helps data de-duplication in many different ways:

SquashFS will properly detect identical files, and only store different (inode) metadata, but using only a copy of the file’s data; similar things can be done by other data de-duplication systems, like I think btrfs should have;
as I said I’m hopeful (although I don’t know the protocol well enough) that rsync can optimise sending multiple copies of the same file;
even the good old tarball has one possible way to optimise multiple identical metadata.xml files, thanks to the fact that the archives produced by tarball are so-called “solid archives” (the compression is applied over the whole content of the archive, rather than on the files individually – as it happens with zip files – even though there is a problem related to their distance in the archive, and the block size used by the compression algorithm, but I’m not really going to go that deep into the problem now).

So what is the problem at this point? There are a few: the first is that nowadays we added more information to the metadata.xml files, with the result that sometimes you can’t use just the same file as any other package (USE flags descriptions for instance); this reduces the usefulness of data de-duplication, as the content of the file is not the same, but it could also help to have a single indentation, order, and format standard, as compressing them together would still allow the compression algorithm to share at least part of the dictionary between them.

The biggest problem is that trying to get an agreement, in Gentoo, over the proper formatting (and indenting) of a file is like trying to herding cats. Sigh.

2010-01-11 by Flameeyes English Technical 10

PDFs and Metadata

You might remember I was thinking about archiving data a few months ago. Up to now, I only stopped at scanning out the docs in PDF (trying to keep quite current with the inbound flow of paper) so that I could have easier access to the documents, and also getting rid of the high amount of useless paper around home.

The experiment up to now seems to be working out decently well. In the sense that the amount of paper around the house started to fall down, and at the same time I’ve been able to archive most of my stuff in a decent way by just using proper paths. Unfortunately, now stuff starts to get complex as well.

What I’d be needing now is some method to arbitrarily tag PDF files (the archive is all in PDF; while Stuart noted that TIFF would also be a decent way to store the data, there is one problem there in the sense that sometimes TIFF files don’t appear correctly on OS X. And since I mix operating systems I needed something that worked on both). And obviously an easy way to get the data out searching for those tags is also needed.

I have been told that XMP from Adobe should do what I need, I remembered the technology name and I’m pretty sure that yes, the way it was designed allows for what I’m looking for; obviously the problem is whether there’s a software that allows me to write down the type of metadata that I need; I’m not really too keen on writing my own, right now.

There is also the other problem of finding the data; I remember from some years ago Beagle could be used to do some on-disk search for documents. I also remember, though, that it was tremendously heavy, eating up lots of CPU and RAM, and just partly because of Mono, the rest was Beagle itself quite easily. Does anybody know whether it has improved? Or can suggest an alternative software to do something similar? I tried merging Tracker, but it doesn’t seem like it’s interested in indexing anything on my system, I have no idea why…

In theory, I’d like something that, searching for “H3G July 2009” would find me the correct PDF with the cellphone bill for the month of July 2009, and searching for “Amazon Office 2007” would find me the invoice for Office 2007 from Amazon UK. I’m fine with writing my own description to the files to get the right one.

If somebody has suggestions, they are definitely welcome. Thanks!

2009-08-30 by Flameeyes English Personal Technical 7

Gentoo automation

Pavel wrote an interesting comment on a previous post of mine which I think is worth quoting and considering a bit:

I wonder if Gentoo has ways to automate mundane tasks. A simple script runned monthly somewhere on gentoo infra could use, i.e. curl to test if HOMEPAGE returns 4xx errors, if it redirects, where to and possibly even file a bug automatically that the HOMEPAGE is wrong for the given package.

Same goes for patch upstream pushing … many projects are hosted on sourceforge, some on distributed VCSs. It could save time for developers to be able to send the patch to upstream via a simple shell interface (sourceforge: curl POST with using a generic gentoo sf account, say bugbot @ gentoo.org) ..

BTW, does Gentoo have tools that do automatic bug filing (into gentoo’s bugzilla) – i.e. based on some build tests (thinking now of, say, gcc 4.3 testing). If yes, it might be really cool if the compilation testing workload could be distributed among gentoo users in a sandboxed+no.user.interaction manner. Might for example, BOINC be used to distribute compile test workunits? These would contain some commands to create a claan virtual host and compilation instructions to test a package/compiler/anything. Maybe it’s a naive approach, might put lots of extra load on gentoo infra but overall would help gentoo move much faster. This might be a cool GSoC project.

There isn’t really much automation going on in Gentoo, although I know there are people working on an automated tinderboxing framework. This has been a target of many people for years I think, although I think the most bugs filed with tinderboxing has been by me and Patrick, who recently joined back as a developer. In my case, the bugs are not automated at all, since I file them one by one using Firefox and bug templates. The boring part is looking up metadata, I guess.

Now, I sincerely don’t want to focus yet on automatic the big tasks of testing and checking; before reaching that point we have to automate smaller tasks in my opinion. This starts for instance with allowing automatic or semi-automatic assignment of bugs, which is something Robin has been working for a long time. But this has found quite a bit of problem, and I sincerely think there are a few details one has to work on for making this process easier; some of these points have been already described by Tiziano (dev-zero) some time ago, but I’m unable to find his blog post right now.

Please note that all that I’m going to write here is only my opinion and hasn’t been submitted to the gentoo-dev mailing list yet, I have a backlog of things to submit there, I hope to be able to deal with them one by one in the next weeks.

The first problem is that currently the metadata information is scattered among different repositories: the metadata.xml files that we have in packages to tell us who maintains them and other information are in the actual tree (gentoo-x86 repository), the DTD for those is with the website data (gentoo-xml) and the herds.xml file that tells us who is maintaining the herd, as well as the herd’s email address (which is not always the same as the herd name, and which might or might not be the same as the bugzilla contact!), is in the “gentoo” repository by itself.

You can see how this is awkward by the fact that repoman actually fetches the metadata.dtd file each week to make sure it has a fresh copy; but when I proposed having a packaged version of DTDs Josh (nightmorph) answered that there is no point having packages of the DTDs because they seem to be able to change those on a daily basis. Now this already brings me another problem but I’ll try to get some order first.

My first point here is that it makes sense to try consolidating the metadata lookup by at least moving herds.xml in the same repository as metadata.xml; this would make it possible to actually have a complete set of data in a single repository, since DTDs are not really needed for accessing it. This is something I have to propose and that have to be considered.

Now, to get down to the DTD issue that I’ve already brought up, leaving alone GuideXML-related DTDs, for metadata.xml and herds.xml you don’t expect the DTD to change on a daily basis. You actually want it to stay as stable as possible, which should make it much easier to have stable DTDs for those. If we were to have that, repoman wouldn’t be fetching the DTD every week, but it would just use the copy that is already on the system, which would be much easier; it’d be up to the developers to actually have the updated copy, one guesses.

But let’s take a step further and return to something I think Tiziano already addressed: DTD syntax does not allow us for expressing structure that would allow us to access more reliably the data. Using some more advanced schema syntax like XML Schema or Relax-NG would make it much simpler to express constrain; I admit I’m not sure of this but I think I remember that they also allows you to define a document where to find a list possible values for a field, which would make sure that the <herd> tag in metadata.xml would actually be present in herds.xml. It would then just be a matter to make sure that the schema is checked during CVS check in, so that the data is always correctly formed.

It’s even more problematic than this: currently there are herds that lack an email, like secure-tunneling, which means that you cannot really assign an email address to submit bugs to for net-misc/strongswan for instance (this is what happened to me last week). So we really have to crack down our own data and make sure that the files are always valid and consistent.

I wanted to talk about other details too but for now I’ll stop here, more posts on the topic will follow.

2008-12-01 by Flameeyes English Technical 2

Nitpicking and improving

Patrick has written about broken packages homepages, and called for other OCD developers to look after that. While I hope I’m not suffering from OCD myself, there is a very good point about making sure that the HOMEPAGE variable is correct, and I think this is one of the thing we have been overlooking for too long already.

The first problem is obviously that users get reported the wrong homepage, as well as developers that try to push changs upstream, but there is another consideration that I noticed the other day, looking to push some changes upstream. An homepage has disappeared from the net, the domain expired, maybe the upstream author lost interested in free software or simply changed domain; I didn’t go around to actually look for the right homepage sincerely. The important part is that the domain was no more, and instead a ~~domain squatter~~ parking site replaced it.

Can you see the problem yet? HOMEPAGE values are duplicated many times on ViewVC as well as on all our mirrors that put the ebuilds available on HTTP as well as RSync, and even more importantly, they are linked on Packages which means that domain squatter get inbound links from a supposedly reliable source. I guess if we don’t already we should set nofollow on the links, but that’s also not nice since for the valid homepages indexing should actually happen properly, for sake of the people who actually look for a software’s homepage and similar.

But this is not the only “nitpicking” that actually has effects on users. There are more things that most developers overlook with time and that instead should be taken care of so that users have the best possible experience. For instance, think of packages that download stuff from sites like kernel.org or Debian, Ubuntu or Fedora download servers; those should use the mirror:// special protocol that looks up a list of possible mirrors from the thirdpartymirrors file and also allows the user to override the choice with its own preferred mirror for that particular archive (I for instance set a series of preferred mirrors in Italy for the most used archives). Unfortunately, not all packages actually do this, and instead refer to the main first server, probably overloaded already, to download the files by default.

It is nitpicking since most users will most likely get the source files out of Gentoo mirrors rather than the upstream sites, but there are some times when it’s useful to use the mirrors, for instance when you want to bump a package before it actually enters portage, and thus the Gentoo mirrors, or when the bumped package is in an overlay. Or when you need to get an older version that is no longer available on our mirrors.

Want more nitpicking? Cleaning up FILESDIR is what you are looking for. You probably know that the tree does not only host all our ebuilds but also patch files, init scripts and default configuration files; this allows for the developers to make changes available in no time, but it has obvious downsides: the more patches are added to the tree, the more data uers have to download, even for packages they are not interested in. While we could work on a different strategy to distribute the tree, we can try to avoid overloading users. To do this, repoman already warns you if you’re committing to a package that has in FILESDIR files bigger than 20K; unfortunately there are a number of files in the tree that are bigger, sometimes much bigger; I estimated that yesterday there were between 1.5 and 2MB of big files in the CVS tree which were not supposed to be there.

While developers are not supposed to overlook these things when they add new files, there are subtle causes that has to be considered for this, one of this is the patching of Makefile.in and configure files together with Makefile.am and configure.ac. While Mart dislike my idea of not allowing that situation, it would certainly reduce the risks of having huge files in your downloaded tree that you don’t need.

But it’s not just huge files the problem, there are also packages that have, all in all, big FILESDIR since they have a number of small files (remember that for most users, each file is gong to take at least 4KiB each, since not everybody uses properly optimised filesystems for small files). And also there are duplicated files, which are something that I really have a beef for, since most of the times you don’t need duplicated files at all. It’s understandable (although it really screams HACK!) for KDE 3 ebuilds, since every patch is living in both the split and the monolithic ebuild, but it’s not understandable for other packages if there are duplicated identical files between versions (somebody said apcupsd?).

Then there is the need to push upstream more and more patches, trying to get various upstream to accept our patches, reworking them if needed, and so on (I sincerely think that stuff like git and other distributed VCS helps a lot to push the patches upstream since we can have official trees with the patches that just get merged, also allowing to attribute them to the users submitting them when needed), which is a very long ad boring task, but also very important to reduce the overhead that Gentoo as a distribution adds.

Now, I know all the things I wrote about this time are details, but when you explode these details on a massive scale you get quality improvements that actually matter. Now, writing down this gave me more than a couple of ideas that I should probably flesh out to see to push for Gentoo, I’ll write them down on FreeMind so I can get back to those tomorrow or, more likely, in the next week.

Also, if you want to reach me you might not be able to until next week; today I have a friend needing hardware help in the evening; tomorrow (Friday) is my birthday and I’ll go out in the evening/night, Saturday I’ll probably be having a sort of party with family, and Sunday I’ll be to watch a play directed by a friend of mine. I really need time to charge my batteries since lately I’m feeling blue a bit too easily.

2008-11-27 by Flameeyes English 8

Migrating documentation

Today’s plans of working were messed up, as I’m waiting for a friend of mine in about an hour, so I dedicated the first half of the afternoon at fixing up the repositories for ruby-hunspell and rust, as I wrote yesterday.

I also ended up adding a few more projects of mine to Ohloh as it helps me tracking down of stuff I’ve been doing and I left behind.

Then talking with a friend of mine (hi Alberto), I remembered that there were USE flags in xine-lib that are non obvious, and I shold have document them already. I asked drac for permission, and now xine-lib’s metadata.xml has documentation for real, win32codecs, mad, flac, gtk, imagemagick, gnome, mmap and truetype USE flags. It should cover all the non-obvious ones.

Checking xine-ui too, after fixing one of the bugs that were reported to xine’s bug tracker after I closed them on Gentoo’s bugzilla as UPSTREAM, I also found that there was a very old copy of xine-ui still hanging around in portage. Thanks to drac, that’s gone, as well as the ncurses USE flag on xine-ui that was unused anyway (now the code in configure is gone from CVS too). And if you use xine-ui from CVS you’ll now get a datafile for shared-mime-info for tox playlist files, thanks to Peter Fox.

I also took care of updating the xine’s maintainer guide removing some outdated and now pointless information, and providing the tracker as main submission entrypoint, rather than xine-devel. There was a lot of stuff to update, as the guide was not updated for exactly 11 months (last change being 30 January, before I left from Gentoo).

Now I’m mostly tempted to document a few more things on xine-ui’s flags, and add some IUSE defaults to xine-lib so that users gets some sane defaults (like enabling a52, dts and mad decoders, and disabling mmap and truetype).

So basically today is a day of migrating documentation so that it follows the current ebuilds, and adding proper flag documentation to metadata rather than relegating it around the maintainer’s guides, at least user side. Expect more commits from me just to add documentation 🙂

2007-12-30 by Flameeyes English Multimedia Technical 3

Proposing more USE flag documentation

No this time it’s not a proposal to add support for writing documentation like last time, it’s more a proposal for the users I want to make.

Doug wrote about adding documentation for HAL USE flags (and I’m glad for that because most of them were unknown to me too), and I’ve added documentation for the USE flags I found less understandable in three packages I maintain directly (pulseaudio, amarok, rbot).

What I want to ask our users is to propose which USE flag should be better explained: a maintainer might find them obvious, and then leave them undocumented as obvious. But the feature is focused on the users, which need to know what an USE flag do.

So, if you find an USE flag that is non-obvious, not documented in metadata.xml, and the package falls in PAM, Sound or Proaudio herds, feel free to open a bug to ask for the USE flag to be documented. I’ll try to do my best to keep up with the requests.

2007-11-24 by Flameeyes English 5

Let’s actually get some metadata!

This blog entry is intended to be a technical proposal, so you might need some knowledge on how Gentoo works if you want to follow it very well. I hope to be as clear as possible so that even non-technical users can read this, but I can’t guarantee it.

So you might or might not know (but if you are a Gentoo developer, and you don’t know, please leave your badge and CVS commit access on the table at the right of the door) that together with ebuilds, Manifests, ChangeLogs and files/ subdirectory, a package’s directory usually contain a metadata.xml file. This file right now contains very little information, although it’s very important information: which herd the package belongs to, who is/are the maintainer(s) of the package, and an optional long description that couldn’t fit in the DESCRIPTION variable of the ebuild. Optionally, you can encode them in different languages, although I see very little going on with that.

These are indeed metadata for the package, as they are data that describe the package. Good, then. Are there problems with this? I don’t.

So if we don’t have problems to solve we should be all set, right? No I don’t think so. As Doug pointed on #gentoo-dev a while ago, it’s very difficult to understand what generic USE flags do in a particular package. Sure you all can imagine that jpeg or png USE flags do in general, but what do they do in the particular case of CUPS for instance? Do they allow jpeg or png files to be printed? Do they allow the HTTP interface to use jpeg or png files?

This is not limited to CUPS and those USE flags. What does the minimal USE flag do for libcdio? And what does it do for vcdimager? And for other packages? While the general sense of “minimal” is clear – it cuts down the stuff that the package installs just to install the core of it, like a library for instance – the specific behaviour of the flag on a given package might require looking at the package.

Doug proposed to use use.local.desc more often, but the problem with that is that the file is already way too long, and you can’t really do much explanation with it, or you make it very very big. And it’s impossible to restrict the description of an USE flag per version.

Here is the problem then, how can represent enough information for this? Well, one way is to write extensive documentation for the package and add it to the Gentoo documentation. This works well for complex packages like, say, Apache, but it doesn’t scale well for simple packages that may not have documentation at all because they are not supposed to be directly used by users.

A possible solution is to consider the particular description of an USE flag for the package as metadata of that package, and, well, write down that information in metadata.xml. I suppose the main problem here is that a lot of Gentoo developers despise XML in any form, without even considering that, while I actually hate any configuration file in XML, it’s quite easy to use it to represent information that needs to be accessed through many different means, as an interexchange format.

Anyway, this is just a proposal, feel free to comment if you want to say something about it; please avoid XML bashing in general, although feel free to comment if you think that it’s inappropriate its use in this context.

I suppose that for this proposal to be accepted, there are three main obstacles to overcome: the first is modifying the DTD to allow some tags to document the USE flags; the second is deciding if the flag in metadata.xml is enough to skip IUSE.invalid warnings (by changing repoman in case), and the third is implementing the support for fetching the description of the USE flag in tools like ufed and similar.

I just hope this doesn’t have to become a GLEP to actually be used.

2007-11-19 by Flameeyes English 9