When the tests are more important than the functions

I’ve got to admit that the whole “Test-Driven Development” hype was not something that appealed to me as much — not because I think tests are wrong, I just think that while tests are important, focusing almost exclusively on them is just as dumb as ignoring them altogether.

In the Ruby world, there is so much talking about tests, that it’s still very common to find gems that don’t work at all with newer Ruby versions, but their specs pass just fine. Or even the tests pass fine for the original author, but they will fail on everyone else’s system because they depend heavily on custom configuration — sometimes, they depend on case-insensitive filesystems because the gem was developed on Windows or Mac, and never tested on Linux. Indeed, for the longest time, Rails own tests failed to work at all chances, and the “vendored” code they brought in, never had a working testsuite. Things have improved nowadays but not significantly.

Indeed, RubyGems do not make it easy to perform testing upon install, which means that many gems distributed lack part of the testsuite altogether — sometimes this is an explicit choice; in the case of my own RubyElf gem the tests are not distributed because they grow and grow, and they are quite a bit of megabytes at this point; if you want to run them you fetch the equivalent snapshot from GitHub — the ebuild in Gentoo uses that as a basis for that reason.

Sometimes even gems coming from people who are considered nearly Ruby Gods, like “rails_autolink”https://rubygems.org/gems/rails_autolink by tenderlove end up with a gem that fails tests, badly, in its release — the version we have in Portage is patched up, and the patch is sent upstream. Only the best for our users.

Now unfortunately, as I noted in the post’s title, some projects care more about the tests than the functionality — the project in question is the very same Typo that I use for this blog, and which I already mused forking to implement fixes that are not important for upstream. Maybe I should have done that already, maybe I will do that.

So I sent a batch of changes and fixes to upstream, some of them fixing issues compelled by their own changes, other implementing changes to allow proper usage of Typo over SSL vhosts (yes my blog is now available over SSL — I have to fix a few links and object load paths in some of the older posts, but it will soon work fine), other again simply making it a bit more “SEO”-friendly, since that seems to be a big deal for the developers.

What kind of response do I get about the changes? “They fail spec” — no matter that the one commit I’m first told it breaks specs actually fix editing of blog post after a change that went straight to master, so it might break specs, but it solve a real life issue that makes the software quite obnoxious. So why did I not check specs?

group :development, :test do
  gem 'thin'
  gem 'factory_girl', '~> 3.5'
  gem 'webrat'
  gem 'rspec-rails', '~> 2.12.0'
  gem 'simplecov', :require => false
  gem 'pry-rails'
end

I have no intention to start looking into this whole set of gems just to be able to run the specs for a blog which I find are vastly messed up. Why do I think so? Well, among other reasons, I’ve been told before quite a few times that they wouldn’t ever pass on PostgreSQL — which happens to be the database that has been powering this very instance for the past eight years. I’m pretty sure it’s working good enough!

Well, after asking (a few times) for the specs output — turns out that most of the specs broken are actually those that hardcode http:// in the URLs. Of course they break! My changes use protocol-relative URIs which means that the output changes to use // — no spec is present that tries to validate the output for SSL-delivered blogs which would otherwise break before.

And what is the upstream’s response to my changes? “It breaks here and there, would you mind looking into it?” Nope! “The commit breaks specs.” — No offer (until I complained loudly enough on IRC) for them to look into it, and fix either the patches or the specs are needed. No suggestion that there might be something to change in the specs.

Not even a cherry-pick of the patches that do not break specs.

Indeed as of this writing, even the first patch in the series, the only one that I really would care about get merged, because I don’t want to get out-of-sync with master’s database, at least until I decide to just get to the fork, is still there lingering, even if there is no way in this world that it breaks specs as it introduces new code altogether.

Am I going to submit a new set of commits with at least the visible specs’ failures fixed? Not sure — I really could care more about it, since right now my blog is working, it has the feature, the only one missing being the user agent forwarding to Akismet. I don’t see friendliness coming from upstream, and I keep thinking that a fork might be the best option at this point, especially when, suggesting the use of Themes for Rails to replace the currently theme handling, so that it works properly with the assets pipeline (one of the best features of Rails 3), the answer was “it’s not in our targets” — well it would be in mine, if I had the time! Mostly because being able to use SCSS would make it easier to share the stylesheets with my website (even though I’m considering getting rid of my website altogether).

So my plead to the rest of the development community, which I hope can be considered part of, is to not be so myopic that you care more about tests passing than features working. For sure Windows didn’t reach its popularity level being completely crash-proof — and at the same time I’m pretty sure that they did at least a level of basic testing on it. The trick is always in the compromise, not on the absolute care or negligence for tests.

Bye bye, Mongo

It was not even six months ago that I written about my monster — the web app that used Mongo and sent PDF to be printed to my home server over IPv6 Internet. But still, the monster is now dead; I’m now considering whether I want to unleash the sources to the world or whether I should just bury it altogether.

What’s going on? Well, from one side I stopped working as a MSP — it doesn’t pay enough for me to actually pay taxes as well as live in Italy. Not working on that anymore I no longer have the need to actually have a database of my customers’ computers. On the other side, I’m having quite a few concerns regarding my choice of MongoDB as a data backend.

So MongoDB seems to be great for rapid prototyping: it allows you to work without having to define a schema you might not know beforehand, and it still allows you decent performance by declaring indexes — which seems to be the one feature that most NoSQL-era databases seem to ignore, if not shun, altogether. But even that seems to lose its traction in front of things like Amazon’s SimpleDB, that I’m currently using for tinderbox log analysis with quite a bit of success.

This is not much because, once you actually finish with the design, the database is solid as a rock, but rather because using MongoDB actually involves quite a bit of work in administration that, honestly, I find hard to justify even for a pet project of mine. First of all there are the dependencies, which include v8 (but that seems to be the case for execjs as well); then there is the fact that MongoDB was not really designed to work in an Unix environment.

This seems to be another constant of NoSQL approaches: not only they laugh at the whole SQL idea (which served us for scores of years, and seems to still serve us fine for the most part), but also at other Unix conventions, such as the FHS or syslog, or simply the format of command-line parameters.

More importantly to me, though, is the bad attitude that the upstream developers have, toward proper versioning and compatibility. Why do I say this? Well, you probably remember my rant about their Ruby bindings and the way they mixed two extensions’ in the same git repository. That’s just the tip of the iceberg.

My monster above actually used Mongoid for the Rails support – Durran did a terrific job with the project! – but I’ve been keeping on version 2, rather than 3, because the new version is Ruby 1.9 only (and JRuby in 1.9-mode, but that’s beside the point). I was looking forward for 3 anyway because it’s based not on the original gems, but on new code written by Durran, who has much more clue about Ruby. While waiting, though, I lost interest in MongoDB due to the way the original gems became even worse than they were before — and all of this is not caused by Durran, but it’s despite his huge effort to make MongoDB work nice in Ruby.

What happened? Gentoo currently provides the bson gem, and the version-synced mongo gem, up to version 1.6.2 — after that, they broke their API — and not by calling the new version 2.0, or 1.7… it’s 1.6.3, followed by 1.6.4! This wouldn’t be that bad if upstream actually fixed this quickly by either releasing a new version with the API that were removed, so we could just ignore the two broken release, or at least release a 1.7 so that the new software could use that, and we could slot it, while keeping the old software on version 1.6.2 (“revoking” the other two).

But upstream seem to have no intention to continue that way; so right now Gentoo looks like lagging behind on three gems (bson, mongo and mongoid) due to this problem. Why mongoid? Well, Durran couldn’t make a new release of 2 working with the new bson gem because they removed something he was using. And since he’s pouring all his work into version 3, he’s not interested in trying to fix a bigger screwup from upstream — so the version of mongoid that is not in Gentoo, and likely will never be, is just changing the gem’s dependencies to require bson 1.6.2 and nothing later.

I’m sorry but if this is the way MongoDB is developed, I have no intention to put my own data to risk with it. Tomorrow I’ll probably export all the data from my current MongoDB database into some XML file, so I can access it and then I’ll just stop the app and remove the MongoDB instance altogether.

Do-s and Don’t-s of Ruby testing

This article is brought to you by some small time I had free to write something down. I’m clearing my own stuff out of the way in Italy, before going back to Los Angeles at the end of the month.

I’ve been working on updating some Gentoo Ruby ebuilds lately, and I start to feel the need to scream at some of the Gems developers out there… to be more constructive, I’ll try to phrase them this way.

Do provide tests in your gem file, or in a tarball

You’d be surprised how many times we have to rely on GIT snapshots in Gentoo simply because the gem file itself lacks the test files; even worse is when you have the test units, but these rely on data that is not there at all. This usually is due to the test data being too big to fit in a quick gem.

Interestingly enough there used to be a gem command parameter that let it run the tests for the gem at install time. I can’t seem to find it any more — this is probably why people don’t want to add big tests to their gem files, what’s the point? Especially given most people won’t be using them at all?

It is true that even my Ruby-Elf also doesn’t ship neither test units nor test data in the gem, but that’s why I still release a tarball together with the gem file itself. And yes I do release it not just tag it on GitHub.

Do tag your releases in GitHub!

Especially if your tests are too big or boring to fit into the gem file, please do tag your release versions in GitHub; this way we already have a workaround for the lack of testing: we’ll just fetch the tarball from there and use it for building our fake gem installation.

Do not mandate presence of analysis tools.

Tools such as simplecov, or ruby-debug, or lately guard, seems to be all the rage with Ruby developers. They are indeed very useful, but please do not mandate their presence, even if only in development mode.

While using these tools is dramatically important if you plan on doing real development on the gem, we don’t care about how much coverage is the gem’s tests having: if the tests pass we know our environment should be sane enough; if they fails we have a problem. That’s enough for us, and we end up having to fight with stuff like simplecov on a per-package basis most of the time.

Do not require test dependencies to build API docs (and vice versa)

This is unfortunately tied with the very simplistic dependency handling of RubyGems. When you look up a Gem’s dependencies you only see “runtime” and “development” — it says nothing about what’s needed to build the docs versus what’s required to run the tests.

Since testing and documentation are usually both handled through a Rakefile, and often enough, especially for complex testing instrumentations, the Rakefile uses the task files provided by the testing framework to declare the targets, it’s hard to even run rake -D on a gem that doesn’t have the testing framework installed.

Now in Gentoo you might want to install the API documentation of your gem but not run the tests, or vice-versa you might want to run the tests but don’t care about API documentation. This is a pain in the neck if the Rakefile requires all the development dependencies to be present.

This together with the previous point ties down to the Bundler simplistic tracking — there is no way to avoid having installed the gems in Gentoo even if they are properly grouped and you “exclude” them. Bundler will try resolving the dependencies anyway, and that will defeat the purpose of using Gentoo’s own dependency tracking.

Do not rely on system services.

This is actually something that happens with non-Ruby packages as well. Many interface libraries that implement access to services such as PostgreSQL, or MongoDB, will try connecting to the system instance during their test routine. This might be okay when you’re the developer, but if you’re doing this on a continuous basis, you really don’t want to do that: the service might not be there, or the service might just not be setup how you expect it to be.

Please provide (and document!) environment variables that can be set to change the default connection strings to the services. This way we can set up a testing instance when running the tests so that you have your full own environment. Even better if you’re able to write setup/teardown scripts that run unprivileged and respect $TMPDIR.

Do have mocks available.

Possibly due to the high usage for Rails and other webapp/webservice technologies based on Rails, there are gems out there for almost every possible third-party webservice. Unfortunately testing these against the full-blown webservice requires private access keys; most gems I’ve seen allow you to specify new keys that are used during the testing phase — it’s a step in the right direction, but it’s not enough.

While by allowing the environment to declare the access keys allows you to set up testing within Gentoo as well, if you care about one particular package, it’s not going to let everybody run their tests automatically. What can you do then? Simple: use mocks. There are tools and libraries out there designed to let you simulate a stateless or partially-stateful webservice so that you can try out your own method to see if they behave.

One important note here: the mocks don’t strictly have to be written in Ruby! While gems don’t allow you to specify non-Ruby dependencies, distributions can do that as well, so if you rely on a mock that is written in Python, C or Java, you can always document it. It’s still going to be easier than getting an access key and using that.

Even better, if your webservice is friendly to open source (which one isn’t nowadays? okay don’t answer), ask them to either provide an “official mock”, or create some standard access keys to be used for sandbox testing, in some kind of sandbox environment.

Do verify the gems you rely upon.

Especially for tests, it’s easy to forget to make sure that the software you rely upon works. Unfortunately, I’ve seen more than a couple of times that the gems used for testing do not work as expected themselves, causing a domino effect.

This is part of the usual “make sure that the software you rely upon really works”, but seems like while most people do verify their own direct runtime dependencies, they more often fail to verify those used during testing.

Of course this list is not complete, and there probably are more things that I didn’t think about, but it’s enough of a rant for now.

Ruby pains, May 2012 edition

While I’m still trying to figure out how to get the logs analysed for the tinderbox, I’ve been spending some time to work on Gentoo’s Ruby packaging again, which is something that happens from time to time as you know.

In this case the spark is the fact that I want to make sure that my systems work with Ruby 1.9. Mostly, this is because the blog engine I’m using (Typo) is no longer supported on Ruby 1.8, and while I did spend some time to get it to work, I’m not interested in keeping it that way forever.

I started by converting my box database so that it would run on Ruby 1.9. This was also particularly important because Mongoid 3 is also not going to support Ruby 1.8. This was helped by the fact that finally bson-1.6 and mongo-1.6 are working correctly with Ruby 1.9 (the previous minor, 1.5, was failing tests). Next step of course will be to get them working on JRuby.

Unfortunately, while now my application is working fine with Ruby 1.9, Typo is still a no-go… reason? It still relies on Rails 3.0, which is not supported on 1.9 in Gentoo, mostly due to its dependencies. For instance it still wants i18n-0.5, which doesn’t work on 1.9, and it tries to get ruby-debug in (which is handled in different gems altogether for Ruby 1.9, don’t ask). The end result is that I’ve still not migrated my blog to the server running 1.9, and I’m not sure when and if that will happen, at this point.. but things seem to fall into place, at least a bit.

Hopefully, before end of the summer, Ruby 1.9 will be the default Ruby interpreter for Gentoo, and next year we’ll probably move off Ruby 1.8 altogether. At some later point, I’d also like to try using JRuby for Rails, since that seems to have its own advantages — my only main problem is that I have to use JDBC to reach PostgreSQL, as the pg gem does not work (and that’s upsetting as that is what my symbol collision analysis script is using).

So, these are my Ruby 1.9 pains for now, I hope to have better news in a while.

Bundler and JRuby — A Ruby Rant

This post is supposed to be auto-posted on Saturday, March 3rd, since I’m testing the new Typo 6 — I used to write posts in advance and let them be posted, but then the support broke, I hope the new version supports it again. If not, it might be a good excuse to fork Typo altogether.

So after a quick poll on Twitter it seems like people are not really disliking my series of Ruby rants, so I think I’ll continue writing a bit more about them. Let me start with making it clear that I’m writing this with the intent of showing the shortcoming of a development environment – and a developers circle – that has closed itself up so much lately that everything is good as long as some other Ruby developer did it before. This is clearly shown on some of the posts defending bundler in the reddit comments about one of my previous posts.

For what concerns Bundler, I don’t think I’m blaming the wrong technology. Sure, Bundler has simplified quite a bit the handling of dependencies and that’s true for Gentoo developers as well (it’s still better than the stupid bundling done by the Rails components in 2.3 series), but the problem here is that it’s way too prone to abuse, especially due to the use of “one and only one” dependencies. I could show you exactly what you mean by simply pointing out that the bourne I already complained about depends on a version of mocha (0.10.4) that is broken on Ruby 1.9 (neither 0.10.3 nor 0.10.5 suffer from this problem, and bourne works fine with either, but they still depend on the only broken one because that was out when they released bourne-1.1.0), but I’ll show you exactly how screwed the system is with a different issue altogether.

Can you vouch that Bundler works on JRuby just as well as it works on MRI 1.8 and 1.9?

If you just answered “Yes”, you have a problem: you’re overconfident that widely used Ruby gems work just because others use them. And this kind of self-validation is not going to let you understand why my work (and Hans’s, and Alex’s) is important.

Let’s take a step back again; one of the proclaimed Ruby strengths is always considered to be the fact that most developers tie long and complex tests in their code so that the code is behaving as expected when changing dependencies, implementation and operating systems. While this can build confidence on one’s code, the sheer presence of these tests is not going to be enough, as many times the developer is the only one running them, and he or she might not have many different operating systems to try it on (multiple implementations are somewhat solved by using RVM and similar projects). Of course it is true that Travis is making it more accessible to try your code on as many different configurations as possible, but it’s still no magic wand.

So there are tests in the libraries, and that’s good — tests are also supposed to be installed by RubyGems, and it should be possible to run them through the gem package manager itself (although I can’t find an option to do that right now), but sometimes you end up with one file missing here or there; in some cases it’s datafiles that the tests use (rather than the test files themselves), or it’s a configuration file that is required for them to run — lately, the nasty and ironic part is that it’s the Gemfile file that is missing from the .gem packaging, making it impossible to run the tests from where, especially when dependencies are only listed there.

An aside, for those of you who might want to find me at fault; I only own a single RubyGems package: Ruby-Elf and it does not come with tests. The reason is that the test data itself is quite big, and most of the Gem users wouldn’t care about the tests anyway (especially since I couldn’t find how to run them). For this reason I decided that the .gem package is only used for execution, and not for packaging. The tarball is always released and tagged so you should use that for packaging; which is exactly what dev-ruby/ruby-elf does.

Okay now you’re probably confused: what has it to do with JRuby that some gems lack a Gemfile (which is used by bundler)? Nothing really, I was just trying to give you an idea of how tricky it can be for us to make sure that the gems we package actually work. I’m now going back to Bundler.

So Bundler comes with its own set of tests — and most importantly it’s designed to be tested against different versions of RubyGems library which is very important for them since they hook deep into the RubyGems API — to the point one wonders if it wouldn’t make more sense to merge Bundler’s dependency rigging and RubyGems loading code in one library, and the bundle and gem commands on a package manager tool project. This is good although we might not want to test it with a number of RubyGems libraries but just the one we have installed. Different target, same code to leverage though.

Anyway, Bundler’s tests work fine in Gentoo for both Ruby 1.8, 1.9 and Ruby Enterprise… but they can’t work with JRuby, and it’s not just a matter of Gentoo packaging. The problem is that Bundler’s Rakefile insists on running tests related to the manual and documentation as well. It’s not too bad, indeed the bundle documentation is installed as man pages, even though these man pages are not installed in the standard man directory, so while they are displayed on bundle --help they are not available to man bundle. I still find it nice: it’s not too different from what I’ve done with Ruby-Elf, where the tools’ help is only available as man pages (but the Gentoo packaging make them available to man as well).

These man pages are generated through a tool that could probably interest me as well — the current man pages for Ruby-Elf are generated through DocBook, which does make them more well organized, but is still one hefty dependency. Anyway the tool is ronn which is a Ruby gem that builds man pages out of Markdown-written text files. The problem is that rather than leaving it possible for the man pages to be generated beforehand, the Rakefile wants to make sure ronn can be called by the current Ruby implementation.

Which should be easy: since ronn is written in Ruby, just make it possible to install it for JRuby. It would be easy indeed if not for the library it uses for Markdown parsing and output generation; it’s not the usual BlueCloth that we’re mostly used to, instead it uses rdiscount which is another (faster?) implementation of Markdown, which is based once again on a C native extension. Being a C extension, that is not available to JRuby, which means that you can’t use ronn with JRuby as it is now.

Since both BlueCloth and rdiscount are based on C extensions, one would wonder how do you make it possible for JRuby to handle Markdown? The answer comes with another gem, which I have learnt about while bumping Radius (a support gem for Radiant), and is called kramdown and is a pure-Ruby, yet fast, implementation of Markdown. For basic usage it’s identical to use to BlueCloth to the point that my monster switched this week from BlueCloth to kramdown with just two lines changes (one to the Gemfile the other to the code).

So I reported to Charles that Bundler can’t have its tests running with Bundler, and he obviously suggested me to poke the Bundler developers to see if they can help out to fix this up. Unfortunately the suggestion is something like you can patch it up but we won’t do the work for you — which is understandable to a point.

The problem with this is that this means nobody has ever tested Bundler on JRuby; if you’re using it there, then you’re assuming it works because everyone else is using it, but there might well be a completely unknown, nasty bug that is just waiting to eat one user’s work on a corner case.

If you do care about JRuby, one very nice thing you could do here is getting in touch with ronn’s upstream, and convert it to use kramdown; then it should be possible to have a fully functional Bundler, relying solely on JRuby. Until that’s done, Gentoo’s JRuby will lack Bundler, and that also means it’ll lack a long list of other software.

One’s DoS fix is one’s test breach (A Ruby Rant)

What do you know, it did indeed become a series!

Even though I’m leaving for Los Angeles again on Monday, March 5th, I’m still trying to package up the few dependencies that I’m missing to be able to get Radiant 1.0 in tree. This became more important to me now that Radiant 1.0.0 has finally been released, even though it might become easier for me to just target 1.1.0 at some later point, given that it would then use Rails 3. Together with that I’m trying to find what I need to get Typo 6 running to update this blog’s engine under the hood.

Anyway one of the dependencies I needed to package for Radiant 1.0. was delocalize which is an interesting library that allows dealing with localise date and time formats (which is something I had to hack together for a project myself, so I’m quite interested in it). Unfortunately, the version I committed to the tree is not the right one I should have packaged, which would have been 0.2.6 or so — since Radiant 1.0 is still on Rails 2.3. Unfortunately there is not a git tag for 0.2.6 so I haven’t packaged that one yet (and the .gem file only has some of the needed test files, not all of them — it’s the usual issue: why do you package the test files in the .gem file if you do not package the test data?), but I’m also trying to find why the code is failing on Ruby 1.9.

Upstream is collaborative and responsive, but as many others is now more focused on Ruby 1.9, rather than 1.8. What does this mean? Well, one of the recent changes to most of the programming languages out there involves changing the way keys are mangled in hash-based storage objects (such as Ruby’s Hash); this has changed the way Ruby 1.8 reported the content of those objects (Ruby 1.9 has ordered hashes which means that they don’t appear to have changed), which caused headaches for both me and Hans for quite a while.

This shows just one of the many reasons why we might have to step in on the throttle and try to get Ruby 1.9 stable and usable soon, possibly even before Ruby 1.8 falls out of security updates, which is scheduled for June — which was what I promised in the FOSDEM talk and I’m striving to make sure we deliver.

Lies, damn lies and dependencies (Yet Another Ruby Rant)

In the previous installment I have complained at how compass required a long list of OSX-specific dependencies, as well as Chrome and a (now dead) gem for reloading pages for its tests.

Well, let’s be honest: it doesn’t require them. It simply tells you it does.

The tests seem to run fine without Chrome, without livereload and most importantly without any of the Darwin-specific dependencies. So why are they still required? The answer lies in Bundler, the “nice” idea Ruby developers had to deal with dependencies.

While from one side Bundler is nice (disallows gems that you don’t enable from being loaded) it has caused all distributions quite a few headaches: not only older versions insisted in copying the content of the gem (even when coming from the system) into the project’s directory, but to work correctly, it expects all gems to provide correct gemspecs with their dependencies. You’d expect that to make them better, but that’s not always the case.

More than a few times the dependencies are set too strict (since Bundler is designed to allow relying on old and obsolete gems without clashing with newer gems requiring newer dependencies — which sounds good at first, until you understand that it might as well be a security issue), but much more commonly you have what I can only call “depend creep”.

Let’s take a step back and go back at the design of Bundler: given it seems to live in symbiosis with Rails 3, it makes sense for it to be able to specify dependencies only needed during development (webservers, logging and debug), or only during production (performance improvements) or testing (testing frameworks, mockers, and so on)… which is why it allows you to define “groups”, which you can disable when calling bundle

If it wasn’t the case that they still require you to install the dependencies! While excluding a particular group allows you to avoid loading the gems, Bundler still insists to make sure that there are no clashing dependency restriction between the various groups… which in turn means that you need to have the development and testing gems available in production as well. The documentation makes a point that the whole :platforms setting is also just an automated group handling, which means that Bundler will try to create the full dependency tree on Linux counting the Darwin and Windows gems as well, which definitely is not what we’re looking forward to.

Oh and it goes one step further: since Gems do not really have a way to explicit dependencies that are valid only for one Ruby implementation, and even if Gemfile has a way to do that, you still end up requiring gems in Ruby 1.9 (for instance) where they make no sense at all. And this brought us a number of “placeholder” gems, that when executed in Ruby 1.9 have no purpose at all.

This without counting the way the ruby-debug gem, that was already tricky due to the incompatibility between the original code and JRuby’s implementation, has now a sibling in ruby-debug19, a different gem that only works in 1.9 (why not maintain them together and simply build one or the other extension is still something I’m wondering).

In the case of compass the Gemfile was listing a number of dependencies that are used by the author during development and make no sense for us during package-testing; which means I don’t have to waste time with all of them, but just with part. Actually, almost all the dependencies are already in portage; I guess the one that is missing would be timecop — I have an ebuild for it, since I wrote it for some other gem before, but its tests fail consistently, and the last release being made over an year ago, with a lot of noise on the GitHub issues’ page, does not let me hope for the best. Honestly, though, even timecop seems to be unneeded, as I see it required but I don’t see its functions ever used.

I sure hope that most of the same is true for paperclip — I was finally able to package cocaine, since bourne was updated to mocha 0.10 (and yet it requires forcefully version 0.10.4 … which is not working under Ruby 1.9, for a regression which will be fixed in 0.10.5 — the ebuild hacks the version in the specification to make sure it does not complain as long as any 0.10 version is installed, for now), but now I’m looking forward the very long list of development dependencies, and wondering if I’ll ever be able to get it to work for me.

Now there is a good question: why am I not giving up? Because once it works, Radiant is a cool CMS; just like Typo is a cool blog engine in my opinion. I’m looking forward for Jim’s reduced dependencies (which he noted on the comments to my other post), but even as it is, it’s still the best web application for the task I could find. Would I be able to find something else? Maybe, but I’m not sure it would be the same. And even if Jim makes the clipped extension optional, I have no doubt I’ll be using it (and thus fighting with paperclip anyway): Radiant 1.0.0 is even better than the previous ones because it solves the problem with extensions’ compatibility…

Test comprehensiveness versus replicability — A Ruby Rant

I guess that “A Ruby Rant” could become a regular column on this blog, given how many of my posts over time has been “Ruby Rants”, but let’s not dig further.

I’m trying my best to package the new dependencies introduced with Radiant 1.0.0rc4 so that we can update the package in Portage (given that Radiant 0.9 is in tree already). This is proving quite difficult; even though Radiant upstream helped me out by replacing the old dependency on highline with a modern one after the issue was fixed upstream, there are a few new gems that require hours and hours of work to package.

The first big issue comes with cocaine — this gem is developed by ThoughtBot, which are the Rubygems.org designers, one would expect highly professional development from them, but that’s by far not the case. The gem requires another ThoughtBot-developed library for testing, bourne which in turn requires mocha, but not any mocha, up to yesterday it required strictly version 0.9.5; now it requires strictly version 0.10.4, which is an improvement but still not kosher. Why this happens?

@flameeyes It hooks deep into mocha and breaks on internal changes. We’d love an internal api for mocha, but there isn’t one.

So here you can see one huge issue with Ruby development: since it’s very hard to actually make internal interfaces internal, due to monkey-patching and scope games, people will end up relying on things that they shouldn’t be relying upon. And they pretend that’s a good way to solve the issue because there is not a better way. Damn.

Okay strike one, I have put cocaine aside for now (and that was a good idea, seeing how prodding them about it have gotten them to at least update bourne’s dependency on a version that is not quite as ancient), and worked on another dependency: compass which is yet another CSS framework.

Testing this particular package has proven quite difficult, because it has a long list of extra dependencies that you won’t find listed in the gemspec at all, but just on the Gemfile. This list among others include compass-validator which is a gem only ever required by compass, which requires compass on its own (without listing it in the gemspec)… even if this sounds fishy enough it doesn’t cover how fishy that gem is.

The compass-validator gem bundles a series of prebuilt Java libraries up to W3C’s CSS Validator and executes Java at runtime. No, this has nothing to do with JRuby, and even when using JRuby it wouldn’t load the Java classes but still execute Java. Thanks to Krzysztof at least I’ve been able to get the gem to not bundle so much Java code; instead it relies on dev-java/css-validator and java-config to call into the right commands. Still nasty but at least usable.

But this is not yet a nasty problem: the Gemfile lists autotest which is a dummy gem for ZenTest instead (so why not list ZenTest directly? bah!), and then proceeds with two different ways to wrap around gems specific for OS X: autotest-fsevent is under a RUBY_PLATFORM check, while rb-fsevent is under a group. The problem is that Bundler always install all the groups, despite not loading them if you exclude them.

Okay so the gem has some specific codepaths for OS X, does that matter much? Probably not as long as you can still ignore them and run them on Linux. But there is more trouble ahead when I reach the terrific livereload gem, which is described this way:

LiveReload is a Safari/Chrome extension + a command-line tool that: 1. Applies CSS and JavaScript file changes without reloading a page. 2. Automatically reloads a page when any other file changes (html, image, server-side script, etc).

Okay so it requires Chrome at test-time.. not so surprising as another of the gems I use (best_in_place) indirectly uses selenium-webdriver, which uses Firefox. The problem is that when you go to the source repository for that gem, you’re told that it’s deprecated (compass is still actively developed!) in favour of … a graphical Mac/Windows application.

So okay I’m all to cover as much as possible with tests, but how does that help if you make it impossible for anyone else but you to actually run the tests because you tie them to one specific platform, and one that is very unlikely to be used as production server?!

Sigh.

Ego-stroke development

I’ll write another post, expanding from a single tweet of mine:

If you release code and are not interested in helping distributions package it… don’t release code. Honestly.

It might sound harsh and mean, but I mean it, and I think it’s for the best.

You might have a number of different reasons to work on opensource projects, but you can basically find that the people who write open source projects and then don’t care about distributions and other kind of integrators, tend to be those developers that only work for their own ego. I don’t mean they write bad code, sometimes they might actually be the best programmer, but their dislike for distributors is poisonous for the whole opensource environment.

Why do I say this? Well, the problem is that if you release good opensource code, whether it’s packaged by distributions or not, it might very well happen that some other developer will find and use it, that’s what you want, when you release it opensource, isn’t it? Now when that project is no longer a library or backend tool, but rather an user facing tool that the developer wants to see used, there will be requests to package it. But once that happens, you might be stuck with some backend dependency that cannot be clearly packaged. In that situation, the whole frontend can’t be packaged, and that means that the distribution’s users will end up suffering.

This is getting even worse with gems, especially when the most insightful comment in my latest post comes from an user that seems to proclaim himself a troll (or at least the nick suggests strongly so). Indeed, trying to excuse Ruby gems developers for not caring about our needs (which usually boil down to having tests around, a clear repository, the ability to rebuild documentation and tests without bringing in a whole lot of gem-building tools – jewelwer, hoe, and so on – and most importantly code that works and tests green, which is by far not obvious when dealing with gems), is just saying that you care about open source for your own feel good, rather than for the whole environment.

This gets worrisome when it comes to Ruby stuff, especially because of the ties with Rails. I guess the majority of the gems are developed for usage with Rails, which means web-facing code; if they are developed without caring about the opensource environment and their possible users, it would be silly not to expect that at least a part of them are so insecure they are likely to cause vulnerabilities in code that is actually published for the general usage.

So I think I just decided to make two personal rules, that I wish were widespread:

  • if you do not plan on helping distributions to get your software packaged, don’t release it opensource;
  • if you plan on making your software opensource, do not use software that distributions can’t package.

Gems make it a battle between the developer and the packager

It is definitely not a coincidence that whenever I have to dive into Gentoo Ruby packaging I end up writing a long series of articles for my blog that should have the tag “Rant” attached, until I end up deciding that it’s not worth it and I should rather do something else.

The problem is that, as I said many times before (and I guess the Debian Ruby team agrees as well), the whole design of RubyGems makes it very difficult to package them properly, and at the same time provides the developers with enough concepts to make the packaging even more tricky than it would by merely due tot he format.

As the title says, for one reason or another, RubyGems’s main accomplishment is simply to put extensions’ developers and distributions’ packages one against the other, with the former group insisting on doing things “fun”, and the latter doing things “right’. I guess most of the members of the former group also never tried managing a long term deployment of their application outside of things like Heroku (that are paid to take care of that).

And before somebody tells me I’m being mean by painting the developers puny with their concept of fun, it’s not my fault if in the space of an hour after tweeting a shorter version of the first paragraph of this post, two people told me that “development is fun”… I’m afraid for most people that’s what matters, it being fun, not reliable or solid…

At any rate… even though as we speak nobody expressed interest (via flattr) on packaging of the Ruby MongoDB driver that I posted about yesterday, I started looking into it (mostly because I’m doing another computer recovery for a customer and thus I had some free time in my hands while I waited for antivirus to complete, dd_rescue to copy data over, and so on so forth).

I was able to get some basic gems for bson and mongo working, which were part of the hydra repository I noted, but the problems started when I looked into plucky which is the “thin layer” used by the actual ORM. It is not surprising that this gem also is “neutered” to the point of being useless for Gentoo packaging requirements, but there are more issues. First of all it required one more totally new gem to be packaged – log_buddy which also required some fixes – that is not listed in the RubyGems website (which is proper, if you consider that the tests are not executable from the gem file), but most importantly, it relied on the matchy gem.

This is something I already had to deal with, as it was in another long list of dependencies last year or the one before (I honestly forgot). This gem is interesting: while the package is dev-ruby/matchy, it was only available as a person-specific gem in Gemcutter: jnunemaker-matchy and mcmire-matchy; the former is the original (0.4.0), while the latter is a fork that fixed a few issues, among which there was the main problem: jnunemaker-matchy is available neither as a tarball nor as a git tag.

For the package that originally required matchy for us (dev-ruby/crack), mcmire’s fork worked quite well, and indeed it was just a matter of telling it to use the other gem for it to work. That’s not the case for plucky, even thought jnunemaker didn’t release any version of matchy in two years, it only works with his version of matchy. Which meant packaging that one as well, for now.

Did I tell you that mcmire’s version works with Ruby 1.9, while jnunemaker’s doesn’t? No? Well, I’m telling you now. Just so you know, almost in 2012, this is a big deal.

And no, there is not a 0.4.0 yet. Two years after release. The code stagnated since then.

Oh and plucky’s tests will fail depending on how Ruby decides to sort an Hash’s keys array. Array comparison in Ruby is (obviously) ordered.

Then you look at the actual mongo_mapper gem that was the leaf of the whole tree.. and you find out that running the tests without bundler fixing the dependencies is actually impossible (due to the three versions of i18n that we have to allow side-installation of). And the Gemfile, while never declaring dependencies on the official Mongo driver (it gets it through plucky), looks for bson_ext (the compiled C extension, that in Gentoo was not going to exist, since it’s actually installed by the same bson package — I’ll have to create a fake gemspec for it just so it can be satisfied).

And this actually brings us to a different problem as well: even though plucky has been updated (to version 0.4.3) in November, it still requires series 1.3 of the Mongo driver. Version 1.4.0 was released in September, and we’re at version 1.5.2.

And I didn’t name SystemTimer gem, which is declared a requirement during development (but not by the gem of course, since you’re not supposed to run tests there) only for Ruby 1.8 (actually only for mri18, what about Ruby EE?) which lacks an indication of a homepage in the RubyGems website….

I love Ruby. I hate its development.