Test comprehensiveness versus replicability — A Ruby Rant

I guess that “A Ruby Rant” could become a regular column on this blog, given how many of my posts over time has been “Ruby Rants”, but let’s not dig further.

I’m trying my best to package the new dependencies introduced with Radiant 1.0.0rc4 so that we can update the package in Portage (given that Radiant 0.9 is in tree already). This is proving quite difficult; even though Radiant upstream helped me out by replacing the old dependency on highline with a modern one after the issue was fixed upstream, there are a few new gems that require hours and hours of work to package.

The first big issue comes with cocaine — this gem is developed by ThoughtBot, which are the Rubygems.org designers, one would expect highly professional development from them, but that’s by far not the case. The gem requires another ThoughtBot-developed library for testing, bourne which in turn requires mocha, but not any mocha, up to yesterday it required strictly version 0.9.5; now it requires strictly version 0.10.4, which is an improvement but still not kosher. Why this happens?

@flameeyes It hooks deep into mocha and breaks on internal changes. We’d love an internal api for mocha, but there isn’t one.

So here you can see one huge issue with Ruby development: since it’s very hard to actually make internal interfaces internal, due to monkey-patching and scope games, people will end up relying on things that they shouldn’t be relying upon. And they pretend that’s a good way to solve the issue because there is not a better way. Damn.

Okay strike one, I have put cocaine aside for now (and that was a good idea, seeing how prodding them about it have gotten them to at least update bourne’s dependency on a version that is not quite as ancient), and worked on another dependency: compass which is yet another CSS framework.

Testing this particular package has proven quite difficult, because it has a long list of extra dependencies that you won’t find listed in the gemspec at all, but just on the Gemfile. This list among others include compass-validator which is a gem only ever required by compass, which requires compass on its own (without listing it in the gemspec)… even if this sounds fishy enough it doesn’t cover how fishy that gem is.

The compass-validator gem bundles a series of prebuilt Java libraries up to W3C’s CSS Validator and executes Java at runtime. No, this has nothing to do with JRuby, and even when using JRuby it wouldn’t load the Java classes but still execute Java. Thanks to Krzysztof at least I’ve been able to get the gem to not bundle so much Java code; instead it relies on dev-java/css-validator and java-config to call into the right commands. Still nasty but at least usable.

But this is not yet a nasty problem: the Gemfile lists autotest which is a dummy gem for ZenTest instead (so why not list ZenTest directly? bah!), and then proceeds with two different ways to wrap around gems specific for OS X: autotest-fsevent is under a RUBY_PLATFORM check, while rb-fsevent is under a group. The problem is that Bundler always install all the groups, despite not loading them if you exclude them.

Okay so the gem has some specific codepaths for OS X, does that matter much? Probably not as long as you can still ignore them and run them on Linux. But there is more trouble ahead when I reach the terrific livereload gem, which is described this way:

LiveReload is a Safari/Chrome extension + a command-line tool that: 1. Applies CSS and JavaScript file changes without reloading a page. 2. Automatically reloads a page when any other file changes (html, image, server-side script, etc).

Okay so it requires Chrome at test-time.. not so surprising as another of the gems I use (best_in_place) indirectly uses selenium-webdriver, which uses Firefox. The problem is that when you go to the source repository for that gem, you’re told that it’s deprecated (compass is still actively developed!) in favour of … a graphical Mac/Windows application.

So okay I’m all to cover as much as possible with tests, but how does that help if you make it impossible for anyone else but you to actually run the tests because you tie them to one specific platform, and one that is very unlikely to be used as production server?!

Sigh.

4 thoughts on “Test comprehensiveness versus replicability — A Ruby Rant

  1. I spent some time trying to figure out what exactly is the problem with Ruby. Until someone made an irl subculture of the Ruby way: and lo, there were hipsters. And lo, all was clear to me and I decided not to waste my braincells on that empty promise no more.

    Like

  2. I hope you mean www-misc/css-validator as I have already created an ebuild for it in java-overlay. Admittedly I didn’t know I needed to create a bug for it back then. Could do with a bump and clean up anyway. Let me know if you’d like me to.

    Like

  3. I’m very interested in ways to fix things like this. Sadly, I think you’re a victim of others’ “works for me” syndrome.I’m the lead developer for Radiant but regardless of that, there are aspects of it that I haven’t touched so I don’t know *everything* about it. Almost, but not everything.Previously we shipped without a way to upload files; it was up to you to add that via several optional extensions.Now, we include an option by default but it’s up to you to turn it off or switch it out if you like. I’m realizing now that our dependency is specified in exactly the wrong spot. We put this into our gem spec instead of the generated Gemfile. I had plans to get the new gem out today, but I may take time to alter this before I do that.I think it’s really important that these complaints bubble up to the top and others discuss and debate them.Does it sound like my solution would fix this for you?

    Like

  4. Thanks Jim, that’s definitely good to hear (dropping a few dependencies), it means we gan give our users some basic Radiant support for 1.0.0 series as well, even though that might mean we have to create more packages for the various extensions.At any rate, while I can’t yet give up on the paperclip extension (I’ve used it as well).. I wish I could avoid compass but, well, for now I’ll wait to see if thenew release of compass makes it better. Upstream says that the tests are broken on stable (and they sure are, a couple of tests try to test trigonometric functions… expecting a wrong value!), so I’d have to check against the new pre-release.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s