Why I don’t trust most fully automatic CI systems

Some people complain that I should let the tinderbox work by itself and either open bugs or mail the maintainers when a failure is encountered. They say that it should make it faster for the bugs to be reported and so on. I resist the idea.

While it does take time, most of the error logs I see in the tinderbox are not reported as bug. They can be either the same bug happening over and over and over again ­– like in the case of openldap lately, which fails to build with gnutls 3 – or they can be known/false positives, or they simply might not be due to the current package but something else that broke, like if a KDE library is broken because one of its dependencies changed ABI.

I had a bad encounter with this kind of CI systems when, for a short while, I worked on the ChromiumOS project. When you commit anything, a long list of buildbots pick up the commits and validate them against their pre-defined configurations. Some of these bots are public, in the sense that you can check their configuration and see the build logs, but a number of them are private and only available on Google’s network (which means you either have to be at a Google office or connect through a VPN).

I wasn’t at a Google office, and I had no VPN. From time to time one of my commits would cause the buildbots to fail and then I had to look up the failure; I think more or less half the time, the problem wasn’t a problem at all but rather one of the bots going crazy, or breaking on another commit that wasn’t flagged. The big bother was that many times the problem appeared in the private buildbots, which meant that I had no way to check the log for myself. Worse still, it would close the tree making it impossible to commit anything else than a fix for that breakage… which, even if there was one, I couldn’t do simply because I couldn’t see the log.

Now when this happened, my routine was relatively easy, but a time waster: I’d have to pop in the IRC channel (I was usually around already), and ask if somebody from the office was around; this was not always easy because I don’t remember anybody in the CEST timezone to have access to the private build logs at the time, at least not on IRC; most where from California, New York or Australia. Then if the person who was around didn’t know me at least by name, they’d explain to me how to reach the link to the build log… to which I had to reply that I have the link, but the hostname is not public, then I’d have to explain that no, I didn’t have access to the VPN….

I think in the nine months I spent working on the project, my time ended up being mostly wasted on trying to track people down, either asking them to fetch the logs for me, review my changes, or simply “why did you do this at the time? Is it still needed?”. Add to this the time spent waiting for the tree to come “green” again so I could push my changes (which often times meant waiting for somebody in California to wake up, making half my European day useless), and the fact that I had no way to test most of the hardware-dependent code on real hardware because they wouldn’t ship me anything in Europe, and you can probably see why both I didn’t want to blog about it while I was at it and why I haven’t continued longer than I did.

Now how does this relate to me ranting about CI today? Well, yesterday while I was working on porting as many Ruby packages as possible to the new testing recipes for RSpec and Cucumber, I found a failure in Bundler — at first I thought about just disabling the tests if not using userpriv, but then I reconsidered and wrote a patch so that the tests don’t fail, and I submitted it upstream — it’s the right thing to do, no?

Well, it turns out that Bundler uses Travis CI for testing all the pull requests and – ‘lo and behold! – it reported my pull request as failing! “What? I did check it twice, with four Ruby implementations, it took me an hour to do that!” So I look into the failure log and I see that the reported error is an exception that is telling Travis that VirtualBox is being turned off. Of course the CI system doesn’t come back at you to say “Sorry, I had a {male chicken}-up”. So I had to comment myself showing that the pull request is actually not at fault, hoping that now upstream will accept it.

Hopefully, after relating my experiences, you can tell why the tinderbox still applies a manual filing approach, and why I prefer spending time to review the logs instead of spending time to attach them.

Bundler and JRuby — A Ruby Rant

This post is supposed to be auto-posted on Saturday, March 3rd, since I’m testing the new Typo 6 — I used to write posts in advance and let them be posted, but then the support broke, I hope the new version supports it again. If not, it might be a good excuse to fork Typo altogether.

So after a quick poll on Twitter it seems like people are not really disliking my series of Ruby rants, so I think I’ll continue writing a bit more about them. Let me start with making it clear that I’m writing this with the intent of showing the shortcoming of a development environment – and a developers circle – that has closed itself up so much lately that everything is good as long as some other Ruby developer did it before. This is clearly shown on some of the posts defending bundler in the reddit comments about one of my previous posts.

For what concerns Bundler, I don’t think I’m blaming the wrong technology. Sure, Bundler has simplified quite a bit the handling of dependencies and that’s true for Gentoo developers as well (it’s still better than the stupid bundling done by the Rails components in 2.3 series), but the problem here is that it’s way too prone to abuse, especially due to the use of “one and only one” dependencies. I could show you exactly what you mean by simply pointing out that the bourne I already complained about depends on a version of mocha (0.10.4) that is broken on Ruby 1.9 (neither 0.10.3 nor 0.10.5 suffer from this problem, and bourne works fine with either, but they still depend on the only broken one because that was out when they released bourne-1.1.0), but I’ll show you exactly how screwed the system is with a different issue altogether.

Can you vouch that Bundler works on JRuby just as well as it works on MRI 1.8 and 1.9?

If you just answered “Yes”, you have a problem: you’re overconfident that widely used Ruby gems work just because others use them. And this kind of self-validation is not going to let you understand why my work (and Hans’s, and Alex’s) is important.

Let’s take a step back again; one of the proclaimed Ruby strengths is always considered to be the fact that most developers tie long and complex tests in their code so that the code is behaving as expected when changing dependencies, implementation and operating systems. While this can build confidence on one’s code, the sheer presence of these tests is not going to be enough, as many times the developer is the only one running them, and he or she might not have many different operating systems to try it on (multiple implementations are somewhat solved by using RVM and similar projects). Of course it is true that Travis is making it more accessible to try your code on as many different configurations as possible, but it’s still no magic wand.

So there are tests in the libraries, and that’s good — tests are also supposed to be installed by RubyGems, and it should be possible to run them through the gem package manager itself (although I can’t find an option to do that right now), but sometimes you end up with one file missing here or there; in some cases it’s datafiles that the tests use (rather than the test files themselves), or it’s a configuration file that is required for them to run — lately, the nasty and ironic part is that it’s the Gemfile file that is missing from the .gem packaging, making it impossible to run the tests from where, especially when dependencies are only listed there.

An aside, for those of you who might want to find me at fault; I only own a single RubyGems package: Ruby-Elf and it does not come with tests. The reason is that the test data itself is quite big, and most of the Gem users wouldn’t care about the tests anyway (especially since I couldn’t find how to run them). For this reason I decided that the .gem package is only used for execution, and not for packaging. The tarball is always released and tagged so you should use that for packaging; which is exactly what dev-ruby/ruby-elf does.

Okay now you’re probably confused: what has it to do with JRuby that some gems lack a Gemfile (which is used by bundler)? Nothing really, I was just trying to give you an idea of how tricky it can be for us to make sure that the gems we package actually work. I’m now going back to Bundler.

So Bundler comes with its own set of tests — and most importantly it’s designed to be tested against different versions of RubyGems library which is very important for them since they hook deep into the RubyGems API — to the point one wonders if it wouldn’t make more sense to merge Bundler’s dependency rigging and RubyGems loading code in one library, and the bundle and gem commands on a package manager tool project. This is good although we might not want to test it with a number of RubyGems libraries but just the one we have installed. Different target, same code to leverage though.

Anyway, Bundler’s tests work fine in Gentoo for both Ruby 1.8, 1.9 and Ruby Enterprise… but they can’t work with JRuby, and it’s not just a matter of Gentoo packaging. The problem is that Bundler’s Rakefile insists on running tests related to the manual and documentation as well. It’s not too bad, indeed the bundle documentation is installed as man pages, even though these man pages are not installed in the standard man directory, so while they are displayed on bundle --help they are not available to man bundle. I still find it nice: it’s not too different from what I’ve done with Ruby-Elf, where the tools’ help is only available as man pages (but the Gentoo packaging make them available to man as well).

These man pages are generated through a tool that could probably interest me as well — the current man pages for Ruby-Elf are generated through DocBook, which does make them more well organized, but is still one hefty dependency. Anyway the tool is ronn which is a Ruby gem that builds man pages out of Markdown-written text files. The problem is that rather than leaving it possible for the man pages to be generated beforehand, the Rakefile wants to make sure ronn can be called by the current Ruby implementation.

Which should be easy: since ronn is written in Ruby, just make it possible to install it for JRuby. It would be easy indeed if not for the library it uses for Markdown parsing and output generation; it’s not the usual BlueCloth that we’re mostly used to, instead it uses rdiscount which is another (faster?) implementation of Markdown, which is based once again on a C native extension. Being a C extension, that is not available to JRuby, which means that you can’t use ronn with JRuby as it is now.

Since both BlueCloth and rdiscount are based on C extensions, one would wonder how do you make it possible for JRuby to handle Markdown? The answer comes with another gem, which I have learnt about while bumping Radius (a support gem for Radiant), and is called kramdown and is a pure-Ruby, yet fast, implementation of Markdown. For basic usage it’s identical to use to BlueCloth to the point that my monster switched this week from BlueCloth to kramdown with just two lines changes (one to the Gemfile the other to the code).

So I reported to Charles that Bundler can’t have its tests running with Bundler, and he obviously suggested me to poke the Bundler developers to see if they can help out to fix this up. Unfortunately the suggestion is something like you can patch it up but we won’t do the work for you — which is understandable to a point.

The problem with this is that this means nobody has ever tested Bundler on JRuby; if you’re using it there, then you’re assuming it works because everyone else is using it, but there might well be a completely unknown, nasty bug that is just waiting to eat one user’s work on a corner case.

If you do care about JRuby, one very nice thing you could do here is getting in touch with ronn’s upstream, and convert it to use kramdown; then it should be possible to have a fully functional Bundler, relying solely on JRuby. Until that’s done, Gentoo’s JRuby will lack Bundler, and that also means it’ll lack a long list of other software.

Rails is not for fire-and-forget hosting!

in my position of Gentoo Ruby team member (and thus Ruby packager), I’d like to give a couple more insights regarding what Jürgen wrote about on the Diaspora topic. I guess I should be considered weighting slightly more than him not only because I dislike Python myself (which seems to be the base criticism moved to Jürgen) but also because I did work with Rails, and this very blog runs on Rails!

I have some personal grudges with Rails, but in general I don’t dislike the MVC approach they take. Unfortunately, while the theory is often pretty solid, the implementation leaves to be desired a lot. A framework that suggests doing agile “Test Driven Development” and fails its own tests is not something nice to look at. But that’s not the problem at this point.

The first problem sits at the way the Diaspora developers worked on the project; someone else already dissected some of the security issues (some, not all of them, do note); and it shows that whoever wrote the code wasn’t a huge expert of Rails to begin with. Beside the number of mistakes in security ideas, working on the older Rails 2 framework is, by itself, a bad idea; considering they are targeting a “later” release, working on the newer Rails 3 branch would have reduced the possible upgrade headaches in the future.

Here comes the biggest problem of all though: Rails is far from a fire-and-forget hosting framework; while PHP has had a long history of incompatibilities between versions I don’t think it ever reached the amount of incompatible versions between one release and the next. Basically, when you write an application with Rails, unless you rely only on the very basic architecture, there is an almost 100% chance that the application will only work with the current version of Rails; full version of it, so if you write it on 2.3.5 it might not work on 2.3.8. And Typo, that I use for this blog, still only work on 2.3.8, not 2.3.9!

To complicate the matter, the single-version-compatibility problem extends not only to Rails itself (which include its own helper gems, such as activerecord) but also to their dependencies, such as Rack, and to the other gems that the application might require. This is one thing that makes maintaining Ruby and Rails packages in a distribution such a hell. To “solve” this problem, Rails 3 mandates the use of Bundler, which, well, creates bundled libraries in every Rails application. Indeed this “solves” (or to be precise, hides) the problem of different versions of packages for different applications, but at the price of possibly leaving around older libraries for older applications.

One thing has to be told about Bundler at least: it should make it much easier to update an application running on Rails, as it takes care of keeping the dependencies at the right version, without causing all the dependency hell that was happening before. Unfortunately, the dependency hell is not the only thing that makes the upgrade of a Rails application (or, for what matters, any other web application) complicated; the other is migrating the database. Rails supposedly provides tools for migrating the databases in a clean way, but often enough they only work with a subset of the database drivers Rails comes with. This is supposed not to be a problem with the “NoSQL” databases, but I’m sincerely quite sceptic about those. Finally, there is the problem of customisations, even if it’s just themes, since they tend to embed enough Ruby logic to require full porting.

This brings us to two points: updating the dependencies of a Rails app only happens when updating the app and updating a Rails app is far from trivial. This easily translates to “updating a Rails app often enough happens only when the administrator is forced to”; newer version of the interpreter breaking the older application, new hosting, new protocol versions and so on so forth.

And if you am to tell me that Rails is pretty secure a framework, please note that there are at least six bugs in the GLSA Archive since 2006, which means an average of slightly over once an year. I can’t think of many security notices even coming for third party gems, and that makes it very difficult to assess their security status.

Up to now, I don’t think I have seen much of an interest for security-evaluating third party gems, as most of them don’t have so much known use (as far as I can tell, most of the Rails applications out there are developed as closed-source, proprietary web applications, not Free applications). Those who are actually used by more widely used applications such as Typo and Redmine probably undergo more scrutiny, but even those can be considered small fishes (WordPress, Trac, Drupal definitely look much meatier targets). With the coming of Diaspora, especially with its idea of “distributed social network” (see the related post for more details), these gems are likely to become an interesting target as well. Especially the eventually older versions that the Diaspora releases will be looking into.

Do I think a different language/framework would have worked better? Not sure sincerely; I don’t like PHP for many reasons, but I have been told that newer versions have much better frameworks than what we had around version 4.2 which is the last version I actively used. Since PHP is (even more than Ruby) designed to be used for web applications, one should suppose it has gained more and more features to make it difficult to make mistakes, even when that makes it more difficult to use it for general purposes. For what I’m concerned, I doubt Python would get much better results either.

Rails is, in my opinion, a pretty good framework if you actually maintain a web application; it’s not good if you write an application for somebody to use and leave it there. By its own definition, it’s “agile” development, but “agile development” require that you actually follow it. Do you think that the average “Diaspora enthusiast” is going to follow the Agile development cycle, or will just set up an instance and stop caring about updating it when it becomes too difficult?

It’s not a matter of language or framework, it’s a matter of architecture; in a project whose basic idea was to allow each user to maintain its own instance, the choice should have gone toward software that made it difficult to misuse, mismanage or keep out-of-date instances running. In this, they seem definitely to have failed.

Humoristic note of the day: if it proves that my concerns come true, and Diaspora become a vessel for more spam, I’ll be renaming that DiaSPAMora…