On test verbosity (A Ruby Rant, but not only)

Testing of Ruby packages is nowadays mostly a recurring joke to the point I didn’t feel I could add much more to my last post — but there is something I can write about, which is not limited to Ruby at all!

Let’s take a step back and see what’s going on. In Ruby you have multiple test harnesses available, although the most commonly used are Test::Unit (which was part of Ruby 1.8 and is available as a gem in 1.9) and RSpec (version 1 before, version 2 now). The two of them have two very different defaults for test output: the former uses, by default, a quiet “progress” style output (one dot per test passed), the latter uses a verbose “documentation” style. But both of them can use the other style as well.

Now, we used to just run tests through rake, using the Rakefile coming with the gem, or the tarball, or the git snapshot, to run tests and build documentation, since both harnesses have very tight integrations with it — but since the introduction of Bundler, this starts to get more troublesome than just writing the test commands out in the ebuild explicitly.

Today I went to bump a package (origin, required for the new Mongoid), and since I didn’t wire in the tests last time (I was waiting for a last minute fix), I decided to do so this time. It should be of note that while, when using rake to run the tests, you’re almost entirely left to the upstream’s preference on what style to use for the tests, it’s tremendously easy to override that in the ebuild itself.

The obvious question is then “which of the two styles should ebuild use for packaged gems?” — The answer is a bit complex, which is why this post came up. Please also note that this does not only apply to Ruby packages, it should be considered for all packages where the tests can be silent or verbose.

On a default usage, you only care about whether the tests fail or pass — if they fail, they’ll give you as much detail as you need, most of the times at least, so the quiet (progress) style is a very good choice: it’s less output, which means smaller logs, less context, and it’s a win-win. For the tinderbox, though, I found out that having more verbose output is useful: to make sure that other tests are not being skipped, to make sure that the build does not get stuck, and if it does, where it got stuck.

Reconciling these two requirements is actually easier than one might think at first because it’s an already solved problem: both perl.eclass and, lately, cmake-utils.eclass support one variable, TEST_VERBOSE which can be set to make the tests output more details while running. So right now origin is the first Ruby ebuild supporting that variable, but talking about this with Hans we came to the conclusion we need a wrapper to take care of that.

So what I’m working on right now is a few changes to the Ruby eclasses so that we support TEST_VERBOSE and NOCOLOR (while the new analysis actually strips out most of the escape codes it’s easier if it doesn’t have to do that!) with something such as ruby-ng-rspec call (which can also check whether rspec is properly in the dependencies!). Furthermore I plan on changing ruby-fakegem.eclass so that it also allows to switch between the current default test function (which uses rake), one that uses RSpec (adding the dependency!), and one that uses Test::Unit. Possibly, after this, we might want to do the same thing with the API documentation building, although that might be trickier — in general though having the same style of API docs installed, instead of using each project’s own, might be a good idea.

So if your package runs tests, please try supporting TEST_VERBOSE, and if it can optionally use colors, make sure it supports NOCOLOR as well, thanks!

Ruby-NG: Package in a Bottle (or, learn how to write a new Ruby ebuild)

I have to say that in the months we’ve been working on the new eclasses, I never went on describing properly how to use them. My hope was to write this documentation straight into the next-generation development manual for Gentoo, but since that project is far from coming, I’ll just rely on my blog for a little while more.

As described in my blog posts the idea behind the “new” (they are in tree for a few months already by now) eclasses is to be able to both handle “proper” Gentoo phases for packaging gems, and at the same time manage dependency and support tracking for multiple Ruby implementations (namely, Ruby 1.8, Ruby 1.9 and JRuby right now). How can we achieve this? Well, with two not-too-distinct operations; first of all we avoid using RubyGems as a package manager – we still use, in some cases, the gem format, and we always use the loader when it makes sense – and then we leverage the EAPI=2 USE-based dependencies.

Why should we not use RubyGems package management for our objective? With the old gems.eclass we used to encapsulate the install operation from RubyGems inside our ebuilds, but it was all done at once, directly into the install phase of the ebuild. We couldn’t have phases (and related triggers) such as prepare, compile, test and install. In particular we had no way to run tests for the packages at install time, which is one of the most useful features of Gentoo as a basis for solid systems. There are also other problems related to the way the packages are handled by RubyGems, including dependencies that we might want to ignore (like runtime dependencies injected by build-time tools), and others that are missing in the specification. All in all, Portage does the job better.

For what concerns the USE-based dependencies, when we merge a package for a set of implementations (one, two, three or any other number), we need its dependencies (at least, the non-optional ones) installed for the same set of implementations, otherwise it cannot work (this is a rehashing of the same-ABI, any-ABI dependencies problem I wrote about one and a half years ago). To solve this problem, our solution is to transforms the implementation into USE flags (actually, they are RUBY_TARGETS flags, but we handle them exactly like USE flags thanks to USE_EXPAND), at that point, when one is enabled for a package, the dependencies need to have the same flag enabled (we don’t care if a dependency has a flag enabled that is not enabled in the first package, though).

This actually creates a bit of a problem though, as you end up having two sets of dependencies: those that are used through Ruby itself (same-ABI dependencies) and those that are not (any-ABI dependencies), such as the C libraries that are being wrapped around, the tools used at runtime by system calls, and so on so forth. To handle this, we ended up adding extra functions that handle the dependencies: ruby_add_bdepend and ruby_add_rdepend, both of which “split the atoms” (yeah this phrase sounds nerdy enough), appending the USE-based dependencies to each. They also have a second interface, in which the first parameter is now a space-separated (quoted) list of USE flags the dependency is conditional to.

This is not the only deviation from the standard syntax that ruby-ng.eclass causes: the other is definitely more substantial: instead of using the standard src_(unpack|prepare|compile|test|install) functions, we have two sets of new functions to define: each_ruby_$phase and all_ruby_$phase. This ties into the idea of supporting multiple implementations, as there are actions that you want to take in almost the same way for all the supported implementations (such as calling up the tests), and others that you want to execute just once (for instance generating, and installing, the documentation). So you get one each and one for all function for each phase.

There are more subtle dependencies of course; in the call to the each type of functions you get ${RUBY} to be the command to call the current implementation, while in the all functions it’s set to the first-available implementation (this is important as we might not support the default implementation of the system). The end result is that you cannot call neither scripts, nor commands, directly; you should, instead, use the ${RUBY} -S ${command} format (for the commands in the search path, like rake, at least), so that the correct implementation gets called.

Oh and of course you cannot share the working directory between multiple implementations, most of the time, especially for the compiled extensions (those written in C). To solve this problem, at the end of the prepare phase, we create an implementation-private copy of the source directory, and we use that in the various each functions; to be on the safe side, we also keep a different source directory for the all functions, so that the results from one build won’t cause problems in the others. To avoid hitting performance too much here, we actually do exactly two tricks: the first is to use hardlinks when copying the source directories (this way, the actual content of the files is shared among the directories, and only the inodes and metadata is duplicated); the second is to invert the order of the all/@each@ calls on the prepare phase.

While in all other cases all is executed after the implementation-specific functions, the all phase is executed before the other prepare functions… which are preceded by the copying, of course. This means that the changes applied during the all_ruby_prepare function are done over the single generic directory and then is copied (hardlinked) to the others.

So this covers most of the functionality of the ruby-ng.eclass, but we had another tightly-related eclass added at the same time: ruby-fakegem.eclass. Like the name let you guess, this is the core of our ditching RubyGems as a package manager entirely. Not only it gives us support for unpacking the (newer) .gem files, but it also provides default actions to deal with testing, documentation and installation; and of course, it provides the basic tools to create fake RubyGems specifications, as well as wrapping of gem-provided binaries. An interesting note here: all the modern .gem files are non-compressed tarballs, that include a compressed metadata YAML file, and a compressed tarball with the actual source files; in the past, there has been a few gems that used instead a base64/mime encoding for sticking the two component files together. For ease of maintaining it, and for sanity, we’ve decided to only support the tarball format; the older gems can be either fixed, worked around or replaced.

The boilerplate code for ruby-fakegem assumes that most gems will have their documentation generation, and tests, handled through means of rake; this is indeed the most common situation, even though it’s definitely not the same situation among different projects. As I said before, Ruby’s motto is definitely “there are many ways to skin a cat”, and there are so many different testing frameworks, with different task names, that it’s not possible to have the same exact code to work for all the gems unless you actually parametrise it. The same goes for the documentation building, even when the framework is almost always the same (RDoc; although there are quite a few packages using YARD nowadays, and a few that are using Hanna — which we don’t have in tree, nor will support, as it requires a specific version of the RDoc gem. an older one). The result is that we have two variables to deal with that: RUBY_FAKEGEM_TASK_TEST and RUBY_FAKEGEM_TASK_DOC which you can set in the ebuild (before inheriting the eclass) to call the correct task.

Now, admittedly this goes a bit beyond the normal ebuild syntax, but we found it much easier to deal with common parameters through variables set before the inherit step, rather than having to write the same boilerplate code over and over… or have to deduce get it directly from the source code (which would have definitely wasted much more time). Together with the two variables above we have two more to handle documentation: RUBY_FAKEGEM_DOCDIR that is used to tell the eclass where the generated documentation is placed, so that it can be properly installed by the ebuild, and RUBY_FAKEGEM_EXTRADOC that provides a quick way to install “Read Me”, ”Change logs” and similar standalone documentation files.

Finally, there are two more variables that are used to handle more installation details. RUBY_FAKEGEM_EXTRAINSTALL is used to install particular files or directories from the sources to the system; this is useful when you have things like Rails or Rudy wanting to use some of the example or template files they are shipped with, at runtime; they are simply installed in the tree like they were part of the gem itself. RUBY_FAKEGEM_BINWRAP is the sole glob-expanded variable in the eclass, and tells it to call the “binary wrapper” (not really binary, but rather scripts wrapper; the name is due to the fact that it refers to the bin/ directory) for the given files, defaulting to all the files in the bin/ directory of the gem; it’s here to be tweaked because in some cases, like most of the Rudy dependencies, the files in the bin/ directory are not really scripts that are useful to be installed, but rather examples and other things that we don’t want to push in the system’s paths. It also comes useful when you might want to rename the default scripts for whatever reason (like, they are actually slotted).

What I have written here is obviously only part of the process that goes into making ebuilds for the new eclasses, but should give enough details for now for other interested parties to start working on them, or porting them even. Just one note before I leave you to re-read this long and boring post: for a lot of packages, the gem does not provide documentation, or a way to generate it, or tests, or part of the datafiles needed for tests to run. In those cases you really need to use a tarball, which might come out of GitHub directly, if the repository is tagged, or might require you toy with commit IDs to find the correct commit. Yup, it’s that fun!

Proper dependencies aren’t overcomplex

It seems like somebody read my previous post about using Gentoo for building embedded-system filesystems as a mission that would increase the complexity of the work for Gentoo, and would waste time for no god reason. And if you look at the post, I do call that bullshit, for a good reason: proper dependencies are not going to increase complexity of either ebuilds nor the work needed by the packager, they are only extensions to the standard procedure.

Let’s take, as example, the zlib package: it’s part of the system set and some people say that this is enough to ignore adding it to the dependencies. Why? Well that’s a very good question: most of the times the reason I’ve been given was to avoid cyclic dependencies, but given zlib has no dependencies itself… Instead, what do we gain, if we actually add it to all the packages that do use it, you have proper reverse-dependency information, which can be used for instance by a much more sophisticated tinderbox to identify which packages need to be rebuilt when one changes.

At the same time, the correct depgraph will be used by Portage to properly order zlib before any other package that do use it is merged; this is quite useful when you broke your system and you need to rebuild everything. And it’s not all, the idea is that you only need to specify dependencies on system packages only for other packages possibly in the system set; the problem is: how can you be certain you’re not in the system set? If you start to consider that pambase can bring gnome in the system set, it’s not really easy, and it’s a moving target as well.

So I beg to differ regarding complexity: if you simply follow the rule if it uses foo, it depends on foo the complexity will be reduced over time rather than increased: you don’t have to check whether foo and your package are in system set or not. The only two packages that QA approves of not depending upon are the C library and the compiler: all the rest has to be depended upon properly.

And in respect of the already-noted bug with distutils eclass: the same problem exists for other eclasses like apache, webapp and java, that would add their own dependencies by default… but have an explicit way to disable the dependencies and the code or tie them to an USE flag. You know, that is not complexity; that is a properly-designed eclass.

Boldly packaging what no man has tried before

It’s actually quite obvious with such an introduction that I’m going to write again about ruby-ng.eclass that I’ve been working on since last month. While my original idea was to write the eclass, submit it on gentoo-dev the week after, and have it in production in just a month, it wasn’t that easy at all.

The first problem is that Ruby on Rails demotivated me with all the tests failing badly for one reason or the other; the second was that, by using an extended set of ebuilds as a test, I was actually able to find enough flaws in my original logics which really improved the situation.

Tonight I worked a bit more on the eclasses (because in the mean time I split it in two: ruby-ng and ruby-fakegem – I was tempted to call it ruby-jasper to play with the names Ruby and Gems, but the joke was probably lost on most people and it’s very likely it would be misunderstood entirely), and finally went around to add some declarative functions to add dependencies, as well as deciding to change further the design of the ebuilds.

The first step was to get rid of the double-inheritance problem that I’ve created in the previous iteration. The fakegem eclass allowed to define a single variable that provided dependencies that are both run-time and test-time (so build-time under test USE flag conditional). Unfortunately since these dependencies needed to have the proper dependencies set by ruby_samelib (which creates the correct USE flag conditionals for the current Ruby implementations selected), which was provided by ruby-ng, but they also had to be defined before inheriting fakegem.

My new solution is to simply use a function call to add these dependencies, so for instance if a package were to need dev-ruby/tzinfo, it would be doing something like this:

ruby_add_rdepend dev-ruby/tzinfo

The eclass will translate that into adding that to RDEPEND, and, in addition, adding it with a test conditional to DEPEND. This has also the side effect of not limiting to fakegems, which is nice, although not strictly needed. It doesn’t stop here of course; because we have dependencies that are instead needed only at buildtime, and sometimes just when building docs or running tests. For that, there is a differently-named function that works just the same:

ruby_add_bdepend test virtual/ruby-test-unit

I’ve designed the function so that it only accepts either one or two parameters: with one parameter, it is interpreted as a space-separated list of atoms to depend on; with two, the first is interpreted as a space-separated list of conditions to have enabled to depend on the list, while the second is again the atoms list. Right now the actual code I have on the eclass is just written badly, before I commit it I have to make it less idiotic (because I admit it is, but it was also written at 3am on a night I was pretty depressed!).

You also might have noticed the virtual/ruby-test-unit atom in the previous snippet. Indeed I started creating virtuals for Ruby stuff. Initially, I wanted to use a sophisticated, overcomplex and error-prone interface for the ruby_add_[br]depend function that allowed to add a package to the list only when an implementation is selected, or for all implementations but one. But then the logical side of me took over and I finally decided to go with an easy, linear interface for the functions, and then add virtuals to the mix.

My reasons to choose virtuals are that the number of packages for which this treatment is needed is pretty low:

rubygems, only needed on Ruby 1.8; both JRuby and Ruby 1.9 bring their own code;
test-unit, that is provided by both Ruby 1.8 and JRuby, but has to be provided externally on Ruby 1.9, and the gem works on all three anyway;
minitest, that is provided by Ruby 1.9 but neither by Ruby 1.8 nor JRuby;
eventually rake (that JRuby provides in its own version!);
almost surely ruby-openssl to request either the Ruby implementation with SSL support enabled, or the jruby-openssl package.

Considering this very modest amount of packages, adding complexity on the eclasses is exactly what I’d like to avoid; also, if the next implementation of Ruby we’re adding already comes with rubygems, but needs test-unit, or the other way around, we don’t want to have to change the dependencies of all the ebuilds, we just want to change the virtual, and then eventually add the new ruby implementations when needed.

I’m still not sure whether we should create a ruby-virtuals category (like we have java-virtuals) or go with virtual/ruby-* (like Perl is doing); the latter approach looks to me like it’s messy, but the former requires adding a new category. I won’t decide alone anyway, luckily.

So, did I get the eclasses ready for submittal? Well… no. There are quite a few more problems that I need to solve, for this to happen. The first problem is, obviously, to make sure that the tests work as they should; there are quite a few that needs to be fixed as I said, but I cannot really do that alone. Then there is the problem of deciding what the virtuals should look like. I should also check out a new JRuby ebuild for the testing bed overlay because currently it aliases the gems directory from Ruby 1.8 to JRuby which it shouldn’t, especially with the new support.

Again, there is to make sure that all the ebuilds in the overlay are working well, with all the dependency properly set in place and so on. More to this, there is need to ensure that all the stuff that is almost identical between the ebuilds is present in the eclass, but no overcomplexity is added to that either. One thing I need to tackle is documentation (re)generation and installation, because that really is important.

And also there are some very important issues of compatibility with what we have in portage right now: Hans should have added the new slotted version of ruby-gettext; we need to properly support slotted fake gems; in theory this only means that just the fakegem bin wrapper should be made to install the wrappers as foo-$SLOT and find the correct version of the gem when looking up the path. Unfortunately I’ll have to study the gems API to find if and how that can be done. And we should make sure that all this works fine with eselect gem (or get it to!).

So the road is still long in front of us, but given that I’ve been working on this alone for the past weeks, I hope we’ll have something ready to enter the tree before end of summer.

After that somebody has to find a way to get Rails applications to install with ebuilds and be managed with webapp-config or similar!

The Ruby NG saga continues

In the past episodes, I’ve started working on a new ruby eclass and that work got me pretty frustrated because Ruby on Rails packages are messed up and even though I was going on with fighting the fight so that Gentoo can become the best vendor to develop and deploy Ruby (and Rails) software on, there have been quite a few setbacks.

First of all, the complexity of the eclass grew a lot as I decided to split it in two: ruby-ng allows handling generic Ruby extensions, while ruby-fakegem allows creation of “fake” gem by installing the data like it would for gems. Unfortunately for the way I’ve been handling dependencies right now, this requires a double-inherit line, first for ruby-ng (to make ruby_samelib() visible) and then another for ruby-fakegem (to pick up the fake gem dependencies – I cannot just use RDEPEND/DEPEND normally because each gem will have, in its DEPEND, the gem’s runtime-dependencies, when built with the test USE flag enabled; but if there are further dependencies like eselect tools, just using RDEPEND wouldn’t be correct). Probably this is one of the things that will change with this week.

Now, one thing I’ve been asked privately is why am I doing such a huge effort with converting the ebuilds if it’s just a proof of concept and it wasn’t submitted to gentoo-dev for approval yet. Well, the main problem is that it’s always difficult to foresee what the use for an eclass will be. for instance when I designed the original autotools.eclass, while I did take into consideration a few packages I knew of, most other devs pointed out that their packages needed something slightly different; with time, autotools.eclass grew quite different from what I designed in a bored afternoon. The result of this is that I finally understand that an eclass cannot be just designed on paper, it has to be applied to real world use cases to work out well.

So even if the workload is quite high, I’m working with real ebuilds, with real and complex packages, with all their deptrees and so on. While it was certainly boring to rewrite almost thirty ebuilds yesterday when I decided to split ruby-fakegem out of ruby-ng, and it’ll be even more boring when I – almost surely – will have to change forty ebuilds for a change in the interface, it’s what saves us from having an eclass that is complex for no good reason. For instance one of the problems with the current ruby eclass is that it tries to take into consideration an absurd amount of configuration scripts used to build extensions. While of course the situation wasn’t this complex at the time it was written, the result now is an overcomplex eclass that tries to do way too much of things that we could do without, and doesn’t do lots of things we now need.

And if you wonder, no I’m not doing this as a job for any company; it’s out of my own free time, and during the breaks I get from the projects I’m paid to work on, I have near to no personal free time lately. I’m mostly fine with it, but given I know there are a few companies that will have a very good time once this all is in place, I wouldn’t disregard a literal “thank you” from time to time… without lacking respect to them, up to now they always come from single users, which is pretty nice… it’s just upsetting that while most of this work ends up helping companies, they don’t seem to care…

But the problems don’t stop at writing and improving the eclasses themselves: there are multiple problems with the underlying packages, because of upstream failures. Now I cannot blame upstream if a package that was last released in 2007 fails its tests with Ruby 1.9 or even worse with JRuby, but there are quite a few cases where the tests fail, on Ruby 1.9, with code released *last month*… especially Rails-related packages, and one case when the 2007-released package had broken tests (that passed) on Ruby 1.8, while the fix for Ruby 1.9 shown that.

But Ruby itself wasn’t without problems: when built with parallel make it can result in an unusable dl extension (the one that dynamically load a shared object from the system to bind, at runtime, specific libraries), which causes it to abort at runtime (yeah, it’s not an exception, it’s an abort of the process because there were missing symbols). Nor the way we as a team acted on Ruby 1.9 build failures was free from fault: we have never forced --no-undefined on the extensions’ builds, which would have caught the dl problem at build-time, as well as would have shown that the compiled extension of ruby-fcgi is not fully Ruby 1.9 compatible (this, too, is deemed to abort, instead of throwing an exception, at runtime).

Also, the list of packages that lack releases, or that have gems but not tarballs nor tags available, is still growing: echoe and treetop are the latest additions. Luckily, gemtoo is supposed on the long term, to address these cases; unfortunately I don’t have the luck of using it right now.

Some people criticized the Ruby/Rails community just for the porn-based talk from Aimonetti.. frankly, the main issue with the development community is far from being that; if it was that, it would have been a trivial social problem. Instead it’s a much more rooted technical problem: people don’t know what a release is! Which makes the work for packagers tremendously more difficult!

Please, try to release code properly: tag it, push both gem and tarball. And please run the testsuite with both Ruby 1.8 and Ruby 1.9, nowadays!

The depressing world of RoR

*This is a rant so you have to take it with a grain of salt; on the other hand, what I’m going to write about is not just my emotional status working with Rails, and you should not take it like it was coming from a Ruby or Rails either; Ruby is still my language of choice, and the blog is still running on Typo, after all.*

After announcing ruby-ng in development, I’ve proceeded with expanding the testbed. While it’s a huge amount of work, and is not supposed to be the final version of the ebuilds that will enter portage, I think that it’s a very good idea to actually test it with real-world situation. One of the most common mistakes with eclasses (and I’m just as culprit of that) is to design them with what you think you should need; which almost never is the same of what you actually will need.

So anyway, the next obvious step was to convert out of gem something that was, before this, pretty impossible: Rails and its dependencies. The main advantage of getting Rails installed manually rather than through gem is that the tests can be executed, so you can make sure the code you’re going to install is fine. If the tests work.

The first problem I came to is that, stupid me for not knowing, activesupport bundles some gems (they coined the term “vendorizing” but it’s just bundling, and it’s just as well ); of these four (builder, i18n, memcache-client and tzinfo) one was already ported to the overlay as the second main testcase (tzinfo), one I knew about and had some trouble with it before (memcache-client: no tarball in releases, Subversion as SCM, no tags for versions!), one was in both tree and overlay (builder), and one I never heard about.

While removing tzinfo was trivial (given that the ebuild seems to work pretty fine and is also updated), the remaining three have been giving me headaches; as I said memcache-client does not have an easy way to install as non-gem; at the same time, builder’s tests don’t pass on Ruby 1.9, not even after changing the code from 2.1.2 release with the one shipped within activesupport (which, by the way, is just one sourcecode line different, all the remaining differences are whitespace and typo fix in comments!); it’s likely that the testsuite is broken rather than builder, but still, it’s not strictly amusing.

But what drove me off to bed last night was actually i18n : the website looks pretty uninteresting, just a single “blog-like” post from last summer, and links to the two gems (for ruby and rails) with their trackers, and the mailing list. The link to the i18n gem’s repository also say “will be moved”.. where to? No idea. There is no download link, since they assume you’re going to get it through gem; the GitHub page does not provide any download either, nor it provides any tag at all, which is even worse than last time .

So okay, let’s leave memcache-info and i18n alone for now and let’s also ignore the builder’s test failure.. activesupport has no testsuite! So I can’t even be sure that it does load the standalone tzinfo properly until I finish and put something up that runs on the new Rails; okay not tremendously important at this point, let’s go on with the rest of the Rails dependencies. Again no test for activerecord since it needs a database (it should be testable with sqlite3 but since I remember having problem with that particular ebuild’s testcase before, I haven’t ported that yet), nothing I can blame them for there.

The problem resumes with actionpack: it requires (or at least tries to) actionmailer when running the tests, but actionmailer depends on actionpack; so you cannot test them the first time around, you got to build them without test first and then you can enable the tests.. and I haven’t been able to test these yet because, as usual, actionmailer has two bundled libraries (text-format and tmail) that I want to get rid of (text-format is present and ported, tmail not present, not even in Portage).

This starts to get even longer than I expected.

Ruby Eclass: the Next Generation

Since I was pretty bored last night (my blog was down, and is up tentatively and temporarily until I can get my hands on a bigger vserver – and actually for me this time – so that I can the blog on its own system, and maybe use it as a mirror for xine’s bugzilla and site), I’ve found time to relax by writing a new Ruby eclass. This started as a way to actually support installation of Ruby packages for the newly-updated JRuby but quickly grew into some more interesting project.

The new eclass not only provides support for configurable, variable Ruby implementations (right now only Ruby 1.8, Ruby 1.9 and JRuby, more can be added) but also supports fake gems natively, which is important to avoid installing so many packages as gems (you remember why don’t you?) while not losing features. In particular, while I have been adding non-gems ebuilds with proper testing, there are a few issues with that method, starting from the fact you cannot slot packages, even when they should be installable side-by-side, like bluecloth or aws-s3.

The ruby-ng eclass, thus, installs tarballs in the same path and layout as real RubyGems, using a crafted fake gem specification, which provide the basic information needed for the them to be picked up. At the same time, though, it leaves the whole of the dependency tracking to Portage, like it should, and it allows for tests to be executed during merge so that you can be sure that the newly-installed code works.

It goes beyond this, though: the way the eclass is designed, you don’t install documentation more than once without reason (with the current eclass, data is copied for each implementation, even if just the last copy is the one that gets merged to the live filesystem), and you don’t extract the same tarball but once. Also, it doesn’t install the extra cruft that gems install (the copy of the gem archive itself, the tests, the unparsed, out of place documentation, the object files for compiled extensions, and so on so forth). The result is a pretty clean gem-compatible install, included binary commands wrappers — you get rake installed out of tarball, instead of gem!

As I said the implementations to install the Ruby extensions for are configurable, on a per-ebuild basis even, by using a new USE-expanded variable (for now called GENTOO_RUBYLIB because I was out of cool names to use, not definitive), and it makes heavy use of EAPI 2’s USE dependencies to bring in the libraries enabled for the right implementations, which is something that the previous code was pretty bad at. This means that both you can get a proper dependency tree for the extension, and repoman can make sure that you don’t commit something that is not supposed to work properly (like for instance a dependency is not supported for one of the implementations). And finally the actual used implementation is depended upon, which changes the paradigm of virtual/ruby tremendously: extensions should depend on the implementation they are built for and the virtual should only be used by scripts with no implementation-dependent libraries needed).

Right now it’s lacking documentation, polishing, and obviously approval for enter main tree. I hope we can coordinate on this in the next week or so, but it might take longer since I have absolutely no clue how this intersect and influences gemtoo. Also, it’s way far from complete, I haven’t worked on proper slotting yet, and I should try it with both Rails libraries and Rails plugins to make sure it works for those — it would be pretty darn nice if you could test the plugins on install finally!

Because I started working on this less than 24 hours ago, the interface is most likely deemed to change, so you should not rely on this for anything but a testbed; dependencies can be broken and stuff like that. On the other hand, once it’s actually implemented, I’ll see to spend my free time to port as much ebuilds as I can to the new format and eclass, so that we can get it running decently soon.

While the new interface is, to me, a lot simpler than the original ruby eclass, and thus the porting effort for non-gem ebuilds is quite negligible, I’d like to say that such a task is going to take a while, just like it took a while to our superb Gentoo Java team to work out the new Java eclasses, put them in production, and get rid of all the old eclasses and ebuilds and the old generation of Java support in Gentoo. On the other hand, what the Gentoo Java team achieved is that Gentoo is one of the best distributions to develop with Java on Linux (almost everything’s packaged); I wish before next fall people will be saying that Gentoo is the best distribution to develop with (and deploy) Ruby on Linux.

As an example of issues I’ve found and I know I’ll find many more times, there are packages that are not bumped in Gentoo and that probably should be together with the porting (so that the latest version, that hopefully works with Ruby 1.9 and all the other implementations we’re going to support, is used); there are packages that requires further packages to run their tests (I’ve said that before, but even today I had to spend most of my time packaging “RDoc 2” which is needed by BlueCloth to run tests… it won’t work with the current eclass so you won’t find it in Portage yet, but it’s in the testbed); there are packages whose tests are simply broken out of the imaginable (for instance Rake that fails its test on all three implementations; Ruby 1.8 and JRuby seem to only have minor issues, Ruby 1.9 seems strangely broken).

So I’ve talked this much about what I’ve been doing and what I hope to be able to do in the next week, but I haven’t shown you the code yet… well ~~here it is~~; but be warned that you should not be using it unless you’re a Gentoo Ruby developer! But you can read it and thank me with a gift if you want…

Work’s not done yet, though.

Problems and mitigation strategies for –as-needed

Doug and Luca tonight asked me to comment about the --as-needed by default bug. As you can read there, my assessment is that the time is ready to bring on a new stage of --as-needed testing through use of forced --as-needed on specs files. If you wish to help testing with this (and I can tell you I’m working on testing this massively), you can do it by creating your own asneeded specs.

Note, Warning, End of the World Caution!

This is not a procedure for the generic users, this is for power users, who don’t mind screwing up their system even beyond repair! But it would help me (and the rest of Gentoo) testing.

First of all you have to create your own specs file, so issue this command:

# export SPECSFILE=$(dirname "$(gcc -print-libgcc-file-name)")/asneeded.specs
# export CURRPROFILE=/etc/env.d/gcc/$(gcc-config -c)
# gcc -dumpspecs | sed -e '/link:/,+1 s:--eh-frame-hdr: --as-needed:' > "$SPECSFILE"
# sed "${CURRPROFILE}" -e '1iGCC_SPECS='$SPECSFILE > "${CURRPROFILE}-asneeded"
# gcc-config "$(basename "${CURRPROFILE}")-asneeded"
# source /etc/profile

This will create the specs file we’re going to use, it just adds —as-needed for any linking command that is not static, then it would create a new configuration file for gcc-config pointing to that file. After this, —as-needed will be forced on. Now go rebuild your system and file bugs about the problems.

A package known to break with this is xmlrpc-c, which in turn makes cmake fail; as some people lack common sense (like adding a way to not have xmlrpc-c as a dependency because you might not want your cmake to ever submit the results of tests), this can get nasty for KDE users. But maybe, just maybe, someone can look into fixing the package at this point.

But xmlrpc-c does require some reflection on how to handle --as-needed in these cases: the problem is that the error is in one package (xmlrpc-c) and the failure in another (cmake) which makes it difficult to asses whether --as-needed break something; you might have a broken package, but nothing depending on it, and never notice. And a maintainer might not notice that his package is broken because other maintainers will get the bugs first (until they get redirected properly). Indeed it’s sub-optimal.

Interestingly enough, Mandriva actually started working on trying to resolve this problem radically, they inject -Wl,--no-undefined in their build procedures so that if a library is lacking symbols, the build dies sooner rather than later. This is fine up to a certain point, because there are times when a library does have undefined symbols, for instance if it has a recursive dependency over another library (which is the case of PulseAudio’s libpulse and libpulsecore, which I discussed with Lennart some time ago). Of course you can work this around by adding a further -Wl,--undefined that then tells ld to discard the former, but it requires more work, and more coordination with upstream.

Indeed, coordination with upstream is a crucial point here, since having to maintain --as-needed fixes in Gentoo is going to be cumbersome in the future, and even more if we start to follow Mandriva’s steps (thankfully, Mandriva is submitting the issues upstream so that they get fixed). But I admit I also haven’t been entirely straight on that; I pushed just today a series of patches to ALSA packages, one of which, to alsa-tools, was for --as-needed support (the copy we have in Portage also just works around the bug rather than fixing it). Maybe we need people that starts checking the tree for patches that haven’t been pushed upstream and tries to push them (with proper credits of course).

Another thing that we have to consider is that many times we have upstream that provide broken Makefiles, and while sending the fix upstream is still possible, fixing it in the ebuild takes more time than it is worth; this is why I want to refine and submit for approval my simple build eclass idea, that at least works as a mitigation strategy.

Fixing CFLAGS/LDFLAGS handling with a single boilerplate Makefile (maybe an eclass, too?)

So, in the last few weeks I’ve been filing bugs for packages that don’t respect CFLAGS (or CXXFLAGS) using the beacon trick. Beside causing some possibly false positives, the testing is going well.

The problem is that I found more than a couple of packages that either do call gcc manually (I admit I’m the author of a couple of ebuilds doing that) or where the patch to fix the Makefile would be more complex than just using a boilerplate makefile.

So what is the boilerplate makefile I talk about? Something like this:

$(TARGET): $(OBJS)
        $(CC) $(LDFLAGS) -o $@ $^ $(LIBS)

Does it work? Yes it does, and it will respect CFLAGS, CXXFLAGS and LDFLAGS just fine, the invocation on an ebuild (taking one I modified earlier today) would be as easy as:

src_compile() {
    emake CC="$(tc-getCC)" 
        TARGET="xsimpsons" 
        OBJS="xsimpsons.o toon.o" 
        LIBS="-lX11 -lXext -lXpm" || die "emake failed"
}

Now of course this would suck if you had to do it for every and each ebuild, but what if we were to simplify it to an eclass? Something like having an ebuild just invoking it this way:

ESIMPLE_TARGET="xsimpsons"
ESIMPLE_OBJS="xsimpsons.o toon.o"
ESIMPLE_LIBS="-lX11 -lXext -lXpm"

inherit esimple

For slightly more complicated things you could make it use PKG_CONFIG too…

ESIMPLE_TARGET="xsimpsons"
ESIMPLE_OBJS="xsimpsons.o toon.o"
ESIMPLE_REQUIRED="x11 xext xpm"

inherit esimple

so that it would call pkg-config for those rather than using the libraries directly (this would allow to simplify also picoxine’s ebuild for instance that uses xine-lib).

Even better (or maybe I’m getting over the top here ;)), one could make the eclass accept a possibly static USE flag that would call pkg-config --static instead of standard pkg-config and append -static to the LDFLAGS, so that the resulting binary would be, well, static…

If anybody has comments about this, to flesh it out before it could actually be proposed for an eclass, it would be a nice time to say here so we can start with the right foot!

Translating ebuilds, a proof of concept

There was only one comment to my previous post, but I actually didn’t expect that one either, first because I probably lost most of my publicity as I’m no more on Planet Gentoo or Gentoo Universe, and second because I know it’s not an issue that concern most of the people who actually use Gentoo at the current stage.

Nonetheless, I wanted to give it a try to a proof of concept of ebuilds translation. If you’re interested in this, you probably want to fetch my overlay, and look at the media-sound/pulseaudio ebuild that is there. Right now there is only Italian translation for it, as that’s the only language I can translate it to, but it works as a proof of concept for me. To try it out, run LC_ALL=“it_IT” and then emerge –1 pulseaudio.

The trick is actually simple once you know it:

messages_locale() {
        locale | grep LC_MESSAGES | cut -d '=' -f 2 | tr -d '"' | cut -d '.' -f 1
}

This function is used to extract the locale currently set for LC_MESSAGE value. Why is this needed? Well, it’s simple: you might be using LC_ALL rather than LC_MESSAGE to set the locale, you also might be using just LANG rather than setting the LC_* variables, so at the end, using locale is the best shot to make sure we get the proper language for messages set up on the system. In the example I have you above, by rewriting LC_ALL we bypass all the other settings.

local msgfile="${FILESDIR}/${P}-postinst"
[[ -f "${msgfile}.$(messages_locale|cut -d '_' -f 1)" ]] && msgfile="${msgfile}.$(messages_locale|cut -d '_' -f 1)"
[[ -f "${msgfile}.$(messages_locale)" ]] && msgfile="${msgfile}.$(messages_locale)"

einfo ""
local save_IFS="${IFS}"
IFS=""
while read line; do
        elog "$line"
done < "${msgfile}"
einfo ""
IFS="${save_IFS}"

This is instead the code that actually handles the loading of the translated message that is then printed on screen for the user. It’s a very rough code as it is, I know already, so no need for pointing me at that: the tools’ chain shown above is ran at least two times up to four if the current language is in a country locale form (like pt_BR), rather than just a language name (it). The code is also prone to errors as it’s quite long by itself.

But as I said, this is a proof of concept rather than an actual implementation, this is just to demonstrate that it is possible to translate messages in ebuilds without filling the ebuilds with the messages in 20 different languages. Of course to avoid adding that big boilerplate code it should go either in portage itself in some way (but that makes adoption of translation a very long term idea, maybe EAPI=1 related) or in a more feasible i18n.eclass, that would handle all of it, included caching the value returned by $(messages_locale) so that it’s not called four times, but once only, and converting from UTF-8 (the usual encoding for in-tree files) to the local encoding, with iconv, if present.

This works well for the long log messages that are added at the postinst phase for instance, because they rarely change between one version and the next one and so have time to be translated. It doesn’t really fly for the short informative messages we have around, nor it works fine for eclasses messages.

For those, what I can think of on the fly is to try to standardise the strings as much as possible (for instance by letting the eclasses to the job), and then use gettext to translate those, with an “app-i18n/portage-i18n” package where the eclass can get their data from. I’ll try to see if I can get a proof of concept of that too.