Gentoo’s fake gems and RSpec

It was last month that I ranted about the lack of verbosity in Ruby ebuilds. While I haven’t had chance to work on the documentation building side of the fence just yet (simply because I really don’t know how documentation is usually built for Ruby packages) we have the RSpec 2-baed “recipe” implemented, which makes it dead easy to run tests with it from a fakegem ebuild.

Unfortunately, this does not apply to Test::Unit, either version 1 or 2, or MiniTest. For Test::Unit 2, I added a helper to ruby-ng.eclass that still makes it easier to deal with those tests, but it’s not possible to automate it as it requires quite a bit of work to be usable, as most of the software out there is not really tested with that framework but rather with standard Test::Unit or MiniTest. But you can’t have everything, can you?

On the other hand, today I’ve started porting packages using RSpec 1.x (dependency on dev-ruby/rspec:0) to RSpec 2 — for most, but not all, of them it’s straightforward. Features like shared example groups changed syntax, and require more hands-on porting, but for the most part I’ve seen them working just fine by calling rspec instead of spec and eventually removing the requirement of the old spec library. I’ll write a page in the Gentoo Wiki about porting to the new RSpec later on today.

What does this bring us? Well, first of all, during testing we no longer have to go through Rake, and eventual helper libraries such as Jeweler, Hoe, Echoe, Bones, etc. This saves us dependencies, which does not only mean that we have fewer packages to install, but also that we have fewer packages whose behaviour can modify the results of the tests. This is a huge difference.

Furthermore, both the RSpec and the Test::Unit 2 helpers respect TEST_VERBOSE and NOCOLOR which means that they don’t cause extra processing on my script on the Tinderbox (which has NOCOLOR set) and I have verbose output to see where Ruby is aborting when it is (it still seems to be random to me, but the memory is OK, memtested and everything).

So anyway, if you happen to rely on some gem and want to see it packaged in Gentoo… please consider making sure it uses RSpec 2 for testing!

On test verbosity (A Ruby Rant, but not only)

Testing of Ruby packages is nowadays mostly a recurring joke to the point I didn’t feel I could add much more to my last post — but there is something I can write about, which is not limited to Ruby at all!

Let’s take a step back and see what’s going on. In Ruby you have multiple test harnesses available, although the most commonly used are Test::Unit (which was part of Ruby 1.8 and is available as a gem in 1.9) and RSpec (version 1 before, version 2 now). The two of them have two very different defaults for test output: the former uses, by default, a quiet “progress” style output (one dot per test passed), the latter uses a verbose “documentation” style. But both of them can use the other style as well.

Now, we used to just run tests through rake, using the Rakefile coming with the gem, or the tarball, or the git snapshot, to run tests and build documentation, since both harnesses have very tight integrations with it — but since the introduction of Bundler, this starts to get more troublesome than just writing the test commands out in the ebuild explicitly.

Today I went to bump a package (origin, required for the new Mongoid), and since I didn’t wire in the tests last time (I was waiting for a last minute fix), I decided to do so this time. It should be of note that while, when using rake to run the tests, you’re almost entirely left to the upstream’s preference on what style to use for the tests, it’s tremendously easy to override that in the ebuild itself.

The obvious question is then “which of the two styles should ebuild use for packaged gems?” — The answer is a bit complex, which is why this post came up. Please also note that this does not only apply to Ruby packages, it should be considered for all packages where the tests can be silent or verbose.

On a default usage, you only care about whether the tests fail or pass — if they fail, they’ll give you as much detail as you need, most of the times at least, so the quiet (progress) style is a very good choice: it’s less output, which means smaller logs, less context, and it’s a win-win. For the tinderbox, though, I found out that having more verbose output is useful: to make sure that other tests are not being skipped, to make sure that the build does not get stuck, and if it does, where it got stuck.

Reconciling these two requirements is actually easier than one might think at first because it’s an already solved problem: both perl.eclass and, lately, cmake-utils.eclass support one variable, TEST_VERBOSE which can be set to make the tests output more details while running. So right now origin is the first Ruby ebuild supporting that variable, but talking about this with Hans we came to the conclusion we need a wrapper to take care of that.

So what I’m working on right now is a few changes to the Ruby eclasses so that we support TEST_VERBOSE and NOCOLOR (while the new analysis actually strips out most of the escape codes it’s easier if it doesn’t have to do that!) with something such as ruby-ng-rspec call (which can also check whether rspec is properly in the dependencies!). Furthermore I plan on changing ruby-fakegem.eclass so that it also allows to switch between the current default test function (which uses rake), one that uses RSpec (adding the dependency!), and one that uses Test::Unit. Possibly, after this, we might want to do the same thing with the API documentation building, although that might be trickier — in general though having the same style of API docs installed, instead of using each project’s own, might be a good idea.

So if your package runs tests, please try supporting TEST_VERBOSE, and if it can optionally use colors, make sure it supports NOCOLOR as well, thanks!

Ruby, tests and specs

While I started working on ruby-bombe, I also wanted to improve the tests of Ruby-Elf, possibly making them more explicit, comment them out so that they are more meaningful and not just to me, and so on. But when I started looking at the code, I found myself trying to start with some dynamic programming, trying to write methods that could generate the test methods, since otherwise I’ve been adding code over code that did just the same thing with very slight differences in names and similar.

So I decided to look into alternative testing frameworks, to see if there was something providing the features I felt I needed. While the basic Test::Unit support in Ruby suits pretty well ruby-bombe, which is mostly a functional library (I’m interested in behaviour rather than data), it does not seem to apply very well to Ruby-Elf where I’m instead testing returned values to ensure they conform to what I’m expecting from them.

I asked Jason from Ohloh if he had any suggestion, and pointed me at RSpec (which I knew by name but never tried myself) and ZenTest (which seems to be lacking any sort of decent documentation). I tried RSpec first and it seems to be almost what I need. Almost.

It might be that I haven’t found how to do it yet, so I’m here asking the ruby-knowing lazyweb some help. Basically, I’m fine with the description idea at the base of RSpec, and indeed it’s useful to see it this way:

describe "linux_amd64_dynamic_executable" do
  it_should_behave_like "dynamic ELF executables"
end

so that it actually runs the tests like I want them to be run:

linux_amd64_dynamic_executable
- should be an ELF file
- should be version 1
- should be an executable file
- should have a .dynamic section
- should have a .dynstr section

But I need to go much deeper. For instance, I’d like that I could for instance describe further the behaviour of the dynamic section, having code similar to this:

  it "should have a .dynamic section" do
    @elf.should have_section(".dynamic")
    @elf[".dynamic"].describe do
      it_should_behave_like "all dynamic sections"
    end
  end

and then having something like this as results:

linux_amd64_dynamic_executable
- should be an ELF file
- should be version 1
- should be an executable file
- should have a .dynamic section
  - should be a dynamic section
  - should be of dynamic type
  - should be called .dynamic
  - should have a final NULL entry
  - ...
- should have a .dynstr section
  - should be a string table
  - should be of string table type
  - ...

The problem is that, as far as I can see, I cannot use the describe method there to do what I need. Or if I can, the documentation does not tell me how and why… but as far as I can see, this only creates a new top-level example, which I don’t need in this case..

Does anybody know anything like this or do I have to hack my own rspec-alike?