Why Puppet?

Seems like the only thing everybody has to comment on my previous post was to ask me why I haven’t used $this, $that and ${the kitchen sink}. Or to be precise they asked about cfengine, chef and bcfg2. I have to say I don’t really like being forced into justifying myself but at this point I couldn’t just avoid answering, or I would keep getting the same requests over and over again.

So first of all, why a configuration management system? I have three production vservers at IOS (one is this, another is xine, and another is a customer’s of mine). I have a standby backup server at OVH. And then there’s excelsior, which has four “spacedocks” (containers that I use for building binpkgs for the IOS servers), three tinderbox (but only two usually running), and a couple of “testing” containers (for x32 and musl), beside the actual container I use in it to maintain stuff.

That’s a lot of systems, and while they are very similar between themselves, they are not identical. To begin with, they are in three different countries. And they us three different CPUs. And this is without adding the RaspberryPi I set up with the weather station for a friend of mine. The result is that trying to maintain all those systems manually is a folly, even though I already reduced the number of hosts, since the print shop customer – the one I wrote so often about – moved on and found someone else to pick up their sysadmin tasks (luckily for both of us, since it was a huge time sink).

But the reason why I focused almost exclusively onto Puppet is easy to understand: people I know have been using it for a while. Even though this might sound stupid, I do follow the crowd of friends of mine when I have to figure out what to use. This is because the moment when I have no idea how to do something, it’s easier to ask to a friend than going through the support chain at the upstream project. Gentoo infra people are using and working on Puppet, so that’s a heavy factor to me. I don’t know why they chose puppet but at this point I really don’t care.

But there is another thing, a lesson I learned with Munin: I need to judge the implementation language. The reason is simple, and that’s that I’ll find bugs, for sure. I have this bad knack at finding bugs in stuff I use… which is the main reason why I got interested in open source development: I could then fix the bugs I found! But to do so I have to understand what’s written. And even though learning Perl was easy, understanding Munin’s code… was, and is, tricky. I was able to get some degree of stuff done. Puppet being written in Ruby is a positive note.

I know, chef is also written in Ruby. But I do have a reason to not wanting to deal with chef: its maintainer in Gentoo. Half the bugs I find have to do with the way things are packaged, which is the reason why I became a developer in the first place. This means though that I have to be able to collaborate with the remaining developers, and sometimes that’s just not possible. Sometimes it’s due to upstream developers but in the case of chef the problem is the Gentoo developer who’s definitely not somebody I want to work with, since he’s been “fiddling” with Ruby ebuilds for chef messing up a lot of the work that the Ruby team, me included, kept pouring to improve the quality of the Ruby packages.

So basically these are the reason why I decided to start using Puppet and writing Puppet modules.

Future planning for Ruby-Elf

My work on Ruby-Elf tends to happen in “sprees” whenever I actually need something from it that wasn’t supported before — I guess this is true for many projects out there, but it seems to happen pretty regularly for me with my projects. The other day I prepared a new release after fixing the bug I found while doing the postmortem of a libav patch — and then I proceeded giving another run to my usual collisions check after noting that I could improve the performance of the regular expressions …

But where is it directed, as it is? Well, I hope I’ll be able to have version 2.0 out before end of 2013 — in this version, I want to make sure I get full support for archives, so that I can actually analyze static archives without having to extract them beforehand. I’ve got a branch with the code to get access to the archives themselves, but it can only extract the file before actually being able to read it. The key in supporting archives would probably be supporting in-memory IO objects, as well as offset-in-file objects.

I’ve also found an interesting gem called bindata which seems to provide a decent way to decode binary data in Ruby without having to actually fully pre-decode it. This would probably be a killer for Ruby-Elf, as a lot of the time I’m forcibly decoding everything because it was extremely difficult to access it on the spot — so the first big change for Ruby-Elf 2 is going to be to drop down the task of decoding to bindata (or, otherwise, another similar gem).

Another change that I plan is to drop the current version of the man pages. While DocBook is a decent way to deal with man pages, and standard enough to be around in most distributions, it’s one “strange” dependency for a Ruby package — and honestly the XML is a bit too verbose sometimes. For the most horsey beefy man pages, the generated roff page is half as big as the source, which is the other way around from what anybody would expect them.

So I’m quite decided that the next version of Ruby-Elf will use Markdown for the man pages — while it does not have the same amount of semantic tagging, and thus I might have to handle some styling in the synopsis manually, using something like md2man should be easy (I’m not going to use ronn because of the old issue with JRuby and rdiscount) and at the same time, it gives me a public HTML version for free, thanks to GitHub conversion.

Finally, I really hope that by Ruby-Elf 2 I’ll be able to get least the symbol demangler for the Itanium C++ ABI — that is the one used by modern GCC, yes, it was originally specified for the Itanic. Working toward supporting the full DWARF specification is something that is on the back of my mind but I’m not very convinced right now, because it’s huge. Also, if I were to implement it I would then have to rename the library to Dungeon.

Passive web logs analysis. Replacing AWStats?

You probably don’t know that, but for my blog I do analyse the Apache logs with AWStats over and over again. This is especially useful at the start of the month to identify referrer spam and other similar issues, which in turn allows me to update my ModSecurity ruleset so that more spammers are caught and dealt with.

To do so, I’ve been using for, at this point, years, AWStats which is a Perl analyzer, generator and CGI application. It used to work nicely, but nowadays it’s definitely lacking. It doesn’t filter referrers search engines as much as it used to be (it’s still important to filter out requests coming from Google, but newer search engines are not recognized), and most of the new “social bookmark” websites are not there at all — yes it’s possible to keep adding to them, but with upstream not moving, this is getting harder and harder.

Even more important, for my ruleset work, is the lack of identification of modern browsers. Things like Android versions and other fringe OSes would be extremely useful for me, but adding support for all of them is a pain and I have enough things on my plates that this is not something I’m looking forward to tackle myself. It’s even more bothersome when you consider that there is no way to reconsider the already analyzed data, if a new URL is identified as a search engine, or an user agent a bot.

One of the most obvious choices for this kind of work is to use Google Analytics — unfortunately, this means that it will only work if it’s not blacklisted from the user side — that includes NoScript users and of course most of the spammers. So this is not a job for them. It’s something that has to be done on the backend, on the logs side.

The obvious point at that point is to find something capable to import the data out of the current awstats datafiles I got, and keep importing data from the Apache log files. Hopefully this should be done by saving the data in a PostgreSQL database (which is what I usually use); native support for vhost data, but the ability to collapse it in a single view would also be nice.

If somebody knows of a similar piece of software, I’d love to give it a try — hopefully, something that is written in Ruby or Perl might be the best for me (because I can hack on those) but I wouldn’t say no to Python or even Java (the latter if somebody helped me making sure the dependencies are all packed up properly). This will bring you better modsec rules, I’m sure!

When the tests are more important than the functions

I’ve got to admit that the whole “Test-Driven Development” hype was not something that appealed to me as much — not because I think tests are wrong, I just think that while tests are important, focusing almost exclusively on them is just as dumb as ignoring them altogether.

In the Ruby world, there is so much talking about tests, that it’s still very common to find gems that don’t work at all with newer Ruby versions, but their specs pass just fine. Or even the tests pass fine for the original author, but they will fail on everyone else’s system because they depend heavily on custom configuration — sometimes, they depend on case-insensitive filesystems because the gem was developed on Windows or Mac, and never tested on Linux. Indeed, for the longest time, Rails own tests failed to work at all chances, and the “vendored” code they brought in, never had a working testsuite. Things have improved nowadays but not significantly.

Indeed, RubyGems do not make it easy to perform testing upon install, which means that many gems distributed lack part of the testsuite altogether — sometimes this is an explicit choice; in the case of my own RubyElf gem the tests are not distributed because they grow and grow, and they are quite a bit of megabytes at this point; if you want to run them you fetch the equivalent snapshot from GitHub — the ebuild in Gentoo uses that as a basis for that reason.

Sometimes even gems coming from people who are considered nearly Ruby Gods, like “rails_autolink”https://rubygems.org/gems/rails_autolink by tenderlove end up with a gem that fails tests, badly, in its release — the version we have in Portage is patched up, and the patch is sent upstream. Only the best for our users.

Now unfortunately, as I noted in the post’s title, some projects care more about the tests than the functionality — the project in question is the very same Typo that I use for this blog, and which I already mused forking to implement fixes that are not important for upstream. Maybe I should have done that already, maybe I will do that.

So I sent a batch of changes and fixes to upstream, some of them fixing issues compelled by their own changes, other implementing changes to allow proper usage of Typo over SSL vhosts (yes my blog is now available over SSL — I have to fix a few links and object load paths in some of the older posts, but it will soon work fine), other again simply making it a bit more “SEO”-friendly, since that seems to be a big deal for the developers.

What kind of response do I get about the changes? “They fail spec” — no matter that the one commit I’m first told it breaks specs actually fix editing of blog post after a change that went straight to master, so it might break specs, but it solve a real life issue that makes the software quite obnoxious. So why did I not check specs?

group :development, :test do
  gem 'thin'
  gem 'factory_girl', '~> 3.5'
  gem 'webrat'
  gem 'rspec-rails', '~> 2.12.0'
  gem 'simplecov', :require => false
  gem 'pry-rails'
end

I have no intention to start looking into this whole set of gems just to be able to run the specs for a blog which I find are vastly messed up. Why do I think so? Well, among other reasons, I’ve been told before quite a few times that they wouldn’t ever pass on PostgreSQL — which happens to be the database that has been powering this very instance for the past eight years. I’m pretty sure it’s working good enough!

Well, after asking (a few times) for the specs output — turns out that most of the specs broken are actually those that hardcode http:// in the URLs. Of course they break! My changes use protocol-relative URIs which means that the output changes to use // — no spec is present that tries to validate the output for SSL-delivered blogs which would otherwise break before.

And what is the upstream’s response to my changes? “It breaks here and there, would you mind looking into it?” Nope! “The commit breaks specs.” — No offer (until I complained loudly enough on IRC) for them to look into it, and fix either the patches or the specs are needed. No suggestion that there might be something to change in the specs.

Not even a cherry-pick of the patches that do not break specs.

Indeed as of this writing, even the first patch in the series, the only one that I really would care about get merged, because I don’t want to get out-of-sync with master’s database, at least until I decide to just get to the fork, is still there lingering, even if there is no way in this world that it breaks specs as it introduces new code altogether.

Am I going to submit a new set of commits with at least the visible specs’ failures fixed? Not sure — I really could care more about it, since right now my blog is working, it has the feature, the only one missing being the user agent forwarding to Akismet. I don’t see friendliness coming from upstream, and I keep thinking that a fork might be the best option at this point, especially when, suggesting the use of Themes for Rails to replace the currently theme handling, so that it works properly with the assets pipeline (one of the best features of Rails 3), the answer was “it’s not in our targets” — well it would be in mine, if I had the time! Mostly because being able to use SCSS would make it easier to share the stylesheets with my website (even though I’m considering getting rid of my website altogether).

So my plead to the rest of the development community, which I hope can be considered part of, is to not be so myopic that you care more about tests passing than features working. For sure Windows didn’t reach its popularity level being completely crash-proof — and at the same time I’m pretty sure that they did at least a level of basic testing on it. The trick is always in the compromise, not on the absolute care or negligence for tests.

I think I’ll keep away from Python still

Last night I ended up in Bizarro World, hacking at Jürgen’s gmaillabelpurge (which he actually wrote on my request, thanks once more Jürgen!). Why? Well, the first reason was that I found out that it hasn’t been running for the past two and a half months, because, for whatever reason, the default Python interpreter on the system where it was running was changed from 2.7 to 3.2.

So I tried first to get it to work with Python 3 keeping it working with Python 2 at the same time; some of the syntax changes ever so slightly and was easy to fix, but the 2to3 script that it comes with is completely bogus. Among other things, it adds parenthesis on all the print calls… which would be correct if it checked that said parenthesis wouldn’t be there already. In a script link the one aforementioned, the noise on the output is so high that there is really no signal worth reading.

You might be asking how comes I didn’t notice this before. The answer is because I’m an idiot! I found out only yesterday that my firewall configuration was such that postfix was not reachable from the containers within Excelsior, which meant I never got the fcron notifications that the job was failing.

While I wasn’t able to fix the Python 3 compatibility, I was able to at least understand the code a little by reading it, and after remembering something about the IMAP4 specs I read a long time ago, I was able to optimize its execution quite a bit, more than halving the runtime on big folders, like most of the ones I have here, by using batch operations, and peeking, instead of “seeing” the headers. At the end, I spent some three hours on the script, give or take.

But at the same time, I ended up having to workaround limitations in Python’s imaplib (which is still nice to have by default), such as reporting fetched data as an array, where each odd entry is a pair of strings (tag and unparsed headers) and each even entry is a string with a closed parenthesis (coming from the tag). Since I wasn’t able to sleep, at 3.30am I started re-writing the script in Perl (which at this point I know much better than I’ll ever know Python, even if I’m a newbie in it); by 5am I had all the features of the original one, and I was supporting non-English locales for GMail — remember my old complain about natural language interfaces? Well, it turns out that the solution is to use the Special-Use Extension for IMAP folders; I don’t remember this explanation page when we first worked on that script.

But this entry is about Python and not the script per-se (you can find on my fork the Perl version if you want). I have said before I dislike Python, and my feeling is still unchanged at this point. It is true that the script in Python required no extra dependency, as the standard library already covered all the bases … but at the same time that’s about it: it is basics that it has; for something more complex you still need some new modules. Perl modules are generally easier to find, easier to install, and less error-prone — don’t try to argue this; I’ve got a tinderbox that reports Python tests errors more often than even Ruby’s (which are lots), and most of the time for the same reasons, such as the damn unicode errors “because LC_ALL=C is not supported”.

I also still hate the fact that Python forces me to indent code to have blocks. Yes I agree that indented code is much better than non-indented one, but why on earth should the indentation mandate the blocks rather than the other way around? What I usually do in Emacs when I’m getting stuff in and out of loops (which is what I had to do a lot on the script, as I was replacing per-message operations with bulk operations), is basically adding the curly brackets in different place, then select the region, and C-M- it — which means that it’s re-indented following my brackets’ placement. If I see an indent I don’t expect, it means I made a mistake with the blocks and I’m quick to fix it.

With Python, I end up having to manage the space to have it behave as I want, and it’s quite more bothersome, even with the C-c < and C-c > shortcuts in Emacs. I find the whole thing obnoxious. The other problem is that, while Python does provide basics access to a lot more functionality than Perl, its documentation is .. spotty at best. In the case of imaplib, for instance, the only real way to know what’s going to give you, is to print the returned value and check with the RFC — and it does not seem to have a half-decent way to return the UIDs without having to parse them. This is simply.. wrong.

The obvious question for people who know would be “why did you not write it in Ruby?” — well… recently I’ve started second-guessing my choice of Ruby at least for simple one-off scripts. For instance, the deptree2dot tool that I wrote for OpenRC – available here – was originally written as a Ruby script … then I converted it a Perl script half the size and twice the speed. Part of it I’m sure it’s just a matter of age (Perl has been optimized over a long time, much more than Ruby), part of it is due to be different tools for different targets: Ruby is nowadays mostly a long-running software language (due to webapps and so on), and it’s much more object oriented, while Perl is streamlined, top-down execution style…

I do expect to find the time to convert even my scan2pdf script to Perl (funnily enough, gscan2pdf which inspired it is written in Perl), although I have no idea yet when… in the mean time though, I doubt I’ll write many more Ruby scripts for this kind of processing..

The usual Typo update report

You probably got used to read about me updating Typo at this point — the last update I wrote about was almost an year ago when I updated to Typo 6, using Rails 3 instead of 2. Then you probably remember my rant about what I would like of my blog …

Well, yesterday I was finally able to get rid of the last Rails 2.3 application that was running on my server, as a nuisance of a customer’s contract finally expired, and since I was finally able to get to update Typo without having to worry about the Ruby 1.8 compatibility that was dropped upstream. Indeed since the other two Ruby applications running on this server are Harvester for Planet Multimedia and a custom application I wrote for a customer, the first not using Rails at all, and the second written to work on both 1.8 and 1.9 alike, I was able to move from having three separate Rails slot installed (2.3, 3.0 and 3.1), to having only the latest 3.2, which means that security issues are no longer a problem for the short term either.

The new Typo version solves some of the smaller issues I’ve got with it before — starting from the way it uses Rails (now no longer requiring a single micro-version, but accepting any version after 3.2.11), and the correct dependency on the new addressable. At the same time it does not solve some of the most long-standing issues, as it insists on using the obsolete coderay 0.9 instead of the new 1.0 series.

So let’s go in order: the new version of Typo brings in another bunch of gems — which means I have to package a few more. One of them is fog which includes a long list of dependencies, most of which from the same author, and reminds me of how bad the dependencies issue is with Ruby packages. Luckily for me, even though the dependency is declared mandatory, a quick hacking around got rid of it just fine — okay hacking might be too much, it really is just a matter of removing it from the Gemfile and then removing the require statement for it, done.

For the moment I used the gem command to install the required packages — some of them are actually available on Hans’s overlay and I’ll be reviewing them soon (I was supposed to do that tonight, but my job got in the way) to add them to main tree. A few more requires me to write them from scratch so I’ll spend a few days on that soon. I have other things in my TODO pipeline but I’ll try to cover as many bases as I can.

While I’m not sure if this update finally solves the issue of posts being randomly marked as password-protected, at least this version solves the header in the content admin view, which means that I can finally see what drafts I have pending — and the default view also changed to show me the available drafts to finish, which is great for my workflow. I haven’t looked yet if the planning for future-published posts work, but I’ll wait for that.

My idea of forking Typo is still on, even though it might be more like a set of changes over it instead of being a full-on fork.. we’ll see.

Why I don’t trust most fully automatic CI systems

Some people complain that I should let the tinderbox work by itself and either open bugs or mail the maintainers when a failure is encountered. They say that it should make it faster for the bugs to be reported and so on. I resist the idea.

While it does take time, most of the error logs I see in the tinderbox are not reported as bug. They can be either the same bug happening over and over and over again ­– like in the case of openldap lately, which fails to build with gnutls 3 – or they can be known/false positives, or they simply might not be due to the current package but something else that broke, like if a KDE library is broken because one of its dependencies changed ABI.

I had a bad encounter with this kind of CI systems when, for a short while, I worked on the ChromiumOS project. When you commit anything, a long list of buildbots pick up the commits and validate them against their pre-defined configurations. Some of these bots are public, in the sense that you can check their configuration and see the build logs, but a number of them are private and only available on Google’s network (which means you either have to be at a Google office or connect through a VPN).

I wasn’t at a Google office, and I had no VPN. From time to time one of my commits would cause the buildbots to fail and then I had to look up the failure; I think more or less half the time, the problem wasn’t a problem at all but rather one of the bots going crazy, or breaking on another commit that wasn’t flagged. The big bother was that many times the problem appeared in the private buildbots, which meant that I had no way to check the log for myself. Worse still, it would close the tree making it impossible to commit anything else than a fix for that breakage… which, even if there was one, I couldn’t do simply because I couldn’t see the log.

Now when this happened, my routine was relatively easy, but a time waster: I’d have to pop in the IRC channel (I was usually around already), and ask if somebody from the office was around; this was not always easy because I don’t remember anybody in the CEST timezone to have access to the private build logs at the time, at least not on IRC; most where from California, New York or Australia. Then if the person who was around didn’t know me at least by name, they’d explain to me how to reach the link to the build log… to which I had to reply that I have the link, but the hostname is not public, then I’d have to explain that no, I didn’t have access to the VPN….

I think in the nine months I spent working on the project, my time ended up being mostly wasted on trying to track people down, either asking them to fetch the logs for me, review my changes, or simply “why did you do this at the time? Is it still needed?”. Add to this the time spent waiting for the tree to come “green” again so I could push my changes (which often times meant waiting for somebody in California to wake up, making half my European day useless), and the fact that I had no way to test most of the hardware-dependent code on real hardware because they wouldn’t ship me anything in Europe, and you can probably see why both I didn’t want to blog about it while I was at it and why I haven’t continued longer than I did.

Now how does this relate to me ranting about CI today? Well, yesterday while I was working on porting as many Ruby packages as possible to the new testing recipes for RSpec and Cucumber, I found a failure in Bundler — at first I thought about just disabling the tests if not using userpriv, but then I reconsidered and wrote a patch so that the tests don’t fail, and I submitted it upstream — it’s the right thing to do, no?

Well, it turns out that Bundler uses Travis CI for testing all the pull requests and – ‘lo and behold! – it reported my pull request as failing! “What? I did check it twice, with four Ruby implementations, it took me an hour to do that!” So I look into the failure log and I see that the reported error is an exception that is telling Travis that VirtualBox is being turned off. Of course the CI system doesn’t come back at you to say “Sorry, I had a {male chicken}-up”. So I had to comment myself showing that the pull request is actually not at fault, hoping that now upstream will accept it.

Hopefully, after relating my experiences, you can tell why the tinderbox still applies a manual filing approach, and why I prefer spending time to review the logs instead of spending time to attach them.

Gentoo’s fake gems and RSpec

It was last month that I ranted about the lack of verbosity in Ruby ebuilds. While I haven’t had chance to work on the documentation building side of the fence just yet (simply because I really don’t know how documentation is usually built for Ruby packages) we have the RSpec 2-baed “recipe” implemented, which makes it dead easy to run tests with it from a fakegem ebuild.

Unfortunately, this does not apply to Test::Unit, either version 1 or 2, or MiniTest. For Test::Unit 2, I added a helper to ruby-ng.eclass that still makes it easier to deal with those tests, but it’s not possible to automate it as it requires quite a bit of work to be usable, as most of the software out there is not really tested with that framework but rather with standard Test::Unit or MiniTest. But you can’t have everything, can you?

On the other hand, today I’ve started porting packages using RSpec 1.x (dependency on dev-ruby/rspec:0) to RSpec 2 — for most, but not all, of them it’s straightforward. Features like shared example groups changed syntax, and require more hands-on porting, but for the most part I’ve seen them working just fine by calling rspec instead of spec and eventually removing the requirement of the old spec library. I’ll write a page in the Gentoo Wiki about porting to the new RSpec later on today.

What does this bring us? Well, first of all, during testing we no longer have to go through Rake, and eventual helper libraries such as Jeweler, Hoe, Echoe, Bones, etc. This saves us dependencies, which does not only mean that we have fewer packages to install, but also that we have fewer packages whose behaviour can modify the results of the tests. This is a huge difference.

Furthermore, both the RSpec and the Test::Unit 2 helpers respect TEST_VERBOSE and NOCOLOR which means that they don’t cause extra processing on my script on the Tinderbox (which has NOCOLOR set) and I have verbose output to see where Ruby is aborting when it is (it still seems to be random to me, but the memory is OK, memtested and everything).

So anyway, if you happen to rely on some gem and want to see it packaged in Gentoo… please consider making sure it uses RSpec 2 for testing!

Bye bye, Mongo

It was not even six months ago that I written about my monster — the web app that used Mongo and sent PDF to be printed to my home server over IPv6 Internet. But still, the monster is now dead; I’m now considering whether I want to unleash the sources to the world or whether I should just bury it altogether.

What’s going on? Well, from one side I stopped working as a MSP — it doesn’t pay enough for me to actually pay taxes as well as live in Italy. Not working on that anymore I no longer have the need to actually have a database of my customers’ computers. On the other side, I’m having quite a few concerns regarding my choice of MongoDB as a data backend.

So MongoDB seems to be great for rapid prototyping: it allows you to work without having to define a schema you might not know beforehand, and it still allows you decent performance by declaring indexes — which seems to be the one feature that most NoSQL-era databases seem to ignore, if not shun, altogether. But even that seems to lose its traction in front of things like Amazon’s SimpleDB, that I’m currently using for tinderbox log analysis with quite a bit of success.

This is not much because, once you actually finish with the design, the database is solid as a rock, but rather because using MongoDB actually involves quite a bit of work in administration that, honestly, I find hard to justify even for a pet project of mine. First of all there are the dependencies, which include v8 (but that seems to be the case for execjs as well); then there is the fact that MongoDB was not really designed to work in an Unix environment.

This seems to be another constant of NoSQL approaches: not only they laugh at the whole SQL idea (which served us for scores of years, and seems to still serve us fine for the most part), but also at other Unix conventions, such as the FHS or syslog, or simply the format of command-line parameters.

More importantly to me, though, is the bad attitude that the upstream developers have, toward proper versioning and compatibility. Why do I say this? Well, you probably remember my rant about their Ruby bindings and the way they mixed two extensions’ in the same git repository. That’s just the tip of the iceberg.

My monster above actually used Mongoid for the Rails support – Durran did a terrific job with the project! – but I’ve been keeping on version 2, rather than 3, because the new version is Ruby 1.9 only (and JRuby in 1.9-mode, but that’s beside the point). I was looking forward for 3 anyway because it’s based not on the original gems, but on new code written by Durran, who has much more clue about Ruby. While waiting, though, I lost interest in MongoDB due to the way the original gems became even worse than they were before — and all of this is not caused by Durran, but it’s despite his huge effort to make MongoDB work nice in Ruby.

What happened? Gentoo currently provides the bson gem, and the version-synced mongo gem, up to version 1.6.2 — after that, they broke their API — and not by calling the new version 2.0, or 1.7… it’s 1.6.3, followed by 1.6.4! This wouldn’t be that bad if upstream actually fixed this quickly by either releasing a new version with the API that were removed, so we could just ignore the two broken release, or at least release a 1.7 so that the new software could use that, and we could slot it, while keeping the old software on version 1.6.2 (“revoking” the other two).

But upstream seem to have no intention to continue that way; so right now Gentoo looks like lagging behind on three gems (bson, mongo and mongoid) due to this problem. Why mongoid? Well, Durran couldn’t make a new release of 2 working with the new bson gem because they removed something he was using. And since he’s pouring all his work into version 3, he’s not interested in trying to fix a bigger screwup from upstream — so the version of mongoid that is not in Gentoo, and likely will never be, is just changing the gem’s dependencies to require bson 1.6.2 and nothing later.

I’m sorry but if this is the way MongoDB is developed, I have no intention to put my own data to risk with it. Tomorrow I’ll probably export all the data from my current MongoDB database into some XML file, so I can access it and then I’ll just stop the app and remove the MongoDB instance altogether.

On test verbosity (A Ruby Rant, but not only)

Testing of Ruby packages is nowadays mostly a recurring joke to the point I didn’t feel I could add much more to my last post — but there is something I can write about, which is not limited to Ruby at all!

Let’s take a step back and see what’s going on. In Ruby you have multiple test harnesses available, although the most commonly used are Test::Unit (which was part of Ruby 1.8 and is available as a gem in 1.9) and RSpec (version 1 before, version 2 now). The two of them have two very different defaults for test output: the former uses, by default, a quiet “progress” style output (one dot per test passed), the latter uses a verbose “documentation” style. But both of them can use the other style as well.

Now, we used to just run tests through rake, using the Rakefile coming with the gem, or the tarball, or the git snapshot, to run tests and build documentation, since both harnesses have very tight integrations with it — but since the introduction of Bundler, this starts to get more troublesome than just writing the test commands out in the ebuild explicitly.

Today I went to bump a package (origin, required for the new Mongoid), and since I didn’t wire in the tests last time (I was waiting for a last minute fix), I decided to do so this time. It should be of note that while, when using rake to run the tests, you’re almost entirely left to the upstream’s preference on what style to use for the tests, it’s tremendously easy to override that in the ebuild itself.

The obvious question is then “which of the two styles should ebuild use for packaged gems?” — The answer is a bit complex, which is why this post came up. Please also note that this does not only apply to Ruby packages, it should be considered for all packages where the tests can be silent or verbose.

On a default usage, you only care about whether the tests fail or pass — if they fail, they’ll give you as much detail as you need, most of the times at least, so the quiet (progress) style is a very good choice: it’s less output, which means smaller logs, less context, and it’s a win-win. For the tinderbox, though, I found out that having more verbose output is useful: to make sure that other tests are not being skipped, to make sure that the build does not get stuck, and if it does, where it got stuck.

Reconciling these two requirements is actually easier than one might think at first because it’s an already solved problem: both perl.eclass and, lately, cmake-utils.eclass support one variable, TEST_VERBOSE which can be set to make the tests output more details while running. So right now origin is the first Ruby ebuild supporting that variable, but talking about this with Hans we came to the conclusion we need a wrapper to take care of that.

So what I’m working on right now is a few changes to the Ruby eclasses so that we support TEST_VERBOSE and NOCOLOR (while the new analysis actually strips out most of the escape codes it’s easier if it doesn’t have to do that!) with something such as ruby-ng-rspec call (which can also check whether rspec is properly in the dependencies!). Furthermore I plan on changing ruby-fakegem.eclass so that it also allows to switch between the current default test function (which uses rake), one that uses RSpec (adding the dependency!), and one that uses Test::Unit. Possibly, after this, we might want to do the same thing with the API documentation building, although that might be trickier — in general though having the same style of API docs installed, instead of using each project’s own, might be a good idea.

So if your package runs tests, please try supporting TEST_VERBOSE, and if it can optionally use colors, make sure it supports NOCOLOR as well, thanks!