Virtualising C in Ruby

As I’ve now left Gentoo, one of my major projects is Rust, that if I can actually complete will be a nice piece of software for ruby developers out there, as it will allow to write Ruby extensions based on C libraries without the need to write all the boilerplate code that can easily be copy pasted wrong, or that, if flawed, would need to be fixed for N instance of the same piece of code.

As I said before, Rust is not yet to the same level of C++ bindings generation as my previous generator, but I’m still quite satisfied with what it’s coming to be because the code is way cleaner that the generator, and way more flexible.

One of the features that my previous generator had and Rust didn’t up to tonight were the «virtual variables». What is a virtual variable? Well, when you write a C++ class, you don’t usually make variables public, or the user (of the library, not the final user of course) can change these to values that are not supposed to be set, and does not allow the code of the class to take changes into account; for this reason the usual method to allow the class’s users to actually change the values of the internal variables you use the so-called getter and setter methods, usually named foo and setFoo or getFoo and setFoo (where foo is the actual name of the variable with a recognisable meaning, and m_foo – or variation thereof – is the name of the private variable). You can do the same in Ruby of course, but there is also the option of naming the methods foo and foo=, and then use the variable foo as a generic variable for the instance, both getter and setter methods will be called automatically.

What a virtual variable do in Rust, is wrapping getters and setters so that they are visible as a variable, with foo and foo= as Ruby function names (and at the same time leave the true name of the methods available for library compatibility). It’s a nice way to wrap around a lot of getter/setter methods pair, and RubyTag++ had quite a lot of them.

Even on the previous generator, the setter and getter pairs were simply expanded in functions, but with the new architecture, this is made even simpler, as I just have to create the two method objects, and then yield them so that the user (of Rust) can provide the extra support needed, like providing the presence of an instance parameter for the C class wrappers (that now I think I should have called virtual classes).

Now, the other feature that is needed for RubyTag++ (although I’m not yet sure if that is the last feature) is an easy way to convert from Ruby integers (be them literals or constants) to C/C++ enum values. While the generator didn’t really carry the values of the enum together with the enum itself, I’ve been now thinking that it would be nice to check if the value coming from Ruby is actually in the list of supported values for the enum, and throw an exception otherwise, instead of passing a maybe incompatible parameter to the C/C++ code.

From this I also considered another useful thing to do with enums: virtual enums. Basically, many C libraries provide some list of #define constants that are progressive, and are used to indicate one out of a series of possible values for a function parameter (for instance). Xine does that, for visual types. I can leave the parameter for the xine_open_video_port() function to accept an integer, and then hope that xine-lib handles it properly, or I can make Rust smart and create a virtual enum that would check for the value sent by the Ruby layer, and again throw an exception if an incompatible parameter is passed.

And again, I’ve been thinking a way to handle the usual situation where a function returns 0 for a success and then a positive or negative value with the error number, or when they return 1 for success, 0 for failure and require calling another function to get the errno (this method is often used in PulseAudio, Avahi and other libraries too). What I’m thinking here is doing sorta like Ruby itself does with syscall errors (Errno:: module) and provide virtual exceptions, converting the integer errno values in proper Ruby exceptions.

What I’m trying to do now is to make Rust be not just a way to produce the C code needed to bind 1:1 the C++ classes or C function sets into Ruby, but also a way to produce an extension that uses Ruby coding standards, as much as possible (this is also why camelCase names for methods are replaced by underlined_names, like QtRuby/Korundum do already).

I’ll try to write more in the next days about the limits of Rust (there are some obvious and some less obvious), and why I’m writing it using C++ features; hopefully by dumping here my thoughts, I’ll be able to extract some documentation to ship with Rust itself.

Moving from FLAC to WavPack

Some of you might have noticed that I blogged about adding WavPack support in xine ; now I’ve also committed to the tree the ebuild for a new CVS snapshot of xine-lib with WavPack supported. The reason why I did this cvs snapshot rather than asking for a new xine-lib release next week (thing that I’m still thinking about) is because one of my additions to this release, a sanity check for the symbols used by the plugins being defined at link time, so that they will fail during build if symbols are missing rather than die on the user at runtime, requires a bit of testing. Compiles both on my systems and on Debian’s buildhost already found some missing link lines that are now fixed, but I cannot really test all the possible combinations. For this reason, the ebuild is now in Portage under package.mask.

Now, to use WavPacks easily with Amarok, you also need TagLib to support them.. luckily aumuell found on the Net some TagLib plugins, now imported into Amarok, that support both WavPack and TrueAudio (the other lossless format I added support for in xine-lib in the past days). The patch is applied in the repository and will be released with version 1.4.5, but for my own pleasure, I’ve patched 1.4.4-r3 in my overlay to add those plugins, as I wanted to be able to test WavPack as much as I could. The result is really good, the plugin works well both in reading and writing APEv2 tags on WavPack files.

Now, I wanted to give WavPack a better go… when I first implemented OggFlac support into xine-lib, I got in a really bad fight with the FLAC format for the use of 3-bits 7-bits and 22-bits fields that are a mess to read without implementing a bitstream parser altogether (which is not practical to be done when the most of the reading is done at byte blocks, rather than bits, like a demuxer does usually), and having the choice of backing off it, I wanted to try it.

Well, I’m satisfied with WavPack way more than with FLAC; encoding a CD Rip (I rip my own CDs, as I want to listen to them with Amarok rather than having to change the CD every time) takes a fraction of the time needed with FLAC, and the files are smaller; now that xine and Amarok supports it, it’s worth the hassle.

Of course, WavPack is not at the same level of FLAC yet, at least when it comes to popularity: neither KAudioCreator nor ruby-ripper have a preset to rip to WavPack, the wavpack commandline utility does not allow you to set APEv2 tags, ls does not recognise them, nor does GNU file, nor KDE’s mimetypes, but it works quite fine to me.

As no commandline tagging application besides Amarok (edit: Lukáš Lalinský pointed out that there are taggers for WavPack files.. unfortunately I was too sleepy when I wrote the entry and I didn’t specify, I meant commandline tools.. and I was sidetracked thinking that Amarok had a DCOP interface for tagging files… instead it was JulTagger I was designing the DCOP tagging interface… so here it is what I meant :)) seemed to be available for WavPack files, I’ve resumed my work with RubyTag++, even if it’s still half broken due to TagLib being incomplete (and Wheels hasn’t released the long promised 1.5 yet), and started adding the support for the extra plugins found in Amarok…

The result is quite nice to me, as I now have a simple script that decompress a flac file, re-compress it in WavPack and copies the basic tags from one to the other, so I could transcode my whole library from FLAC to WavPack, sparing about a gigabyte out of the total (23/22GB) of my music collection.

I’m planning to release in the next days the first alpha of RubyTag++, and then start shipping at least some commandline tool to allow tags manipulation, and hopefully resume work on JulTagger soon too.

And now a little question for my readers: what should I do with MTP support being now lingering? Seems like no interested dev has a MTP device to test with, and after Thomas retired, the libmtp package is also unmaintained. I’m not that keen on buying myself an MTP player just to test Amarok (especially since my cellphone – well, the broken one I have to get repaired – is also an MP3 player of tis own). Is anyone available to be a tester/maintainer for MTP stuff?

I want to cry, even more TagLib problems

So, after yesterday, I continued to implement even more tests for RubyTag++ and to try to debug the issues. Unfortunately I didn’t make much progresses on the debug part, although I have carewolf the command I used to create the test file and he said he’ll try to take a look o that when he’ll have time.

I started writing a simple test for the audio properties of mp3 files, that are quite useful to have also on JulTagger. Unfortunately, although the first tests gone well, when I added the complete test of the functions, I found that one of the functions documented and present in the header file is not implemented, so, test failure, as it hits a missing symbol.

Sigh. I send an email to taglib-devel, but then I see that it’s already fixed on SVN, so I think of taking a snapshot of it.. shouldn’t be difficult to get it from KDE SVN, should it?

Well, I would take and test daily snapshots of xine-lib, xine-ui and vlc rather than trying again to take TagLib’s snapshot, as it’s entirely messed up. It requires kdesupport from trunk, that uses trunk admin dir, that requires unsermake, even if TagLib does not need it at all.
The ChangeLog, AUTHORS, README files are not in their places, the Makefile.am files are not ready to get the correct behaviour when using make dist…

What I had up to now is a incomplete tarball that ships with a missing NEWS file, a taglib.pc file that should be autogenerated (another bug in Makefile.am), and misses examples and C bindings.

Writing RubyTag++ and JulTagger is getting harder and harder, but it is mainly TagLib’s fault now :( I do hope for a 1.4.1 release soon, at least addressing part of the issues I’m hitting.

Again, if anyone wants to help, with either RubyTag++ or JulTagger or to write TagLib’s testsuite, please please please just contact me.

Hitting TagLib’s problems

Continues my work to provide reliable Ruby bindings for TagLib library. Unfortunately this mission is becoming harder with the time, as I find runtime problems that needs to be nailed down and fixed. Sigh.

Let’s start with news updates: since yesterday I moved RubyTag++ from Bazaar-NG to GIT. This move was due to dev.gentoo.org being under heavy load and bzr taking ages to checkout the repository via HTTP; I moved it locally, too, so I do have two complete copies of the repository on the two boxes, and I cross-backup them daily. GIT also seems to be quite faster to work with, too, and has a CIA commit script that does not break every other release :P (although it sends an SHA1 instead of a revision number :( ). I had to hack it a bit to make it send my name as commit author, although I might ask Micah to add a rule for my email address.

Now let’s see the problems I’m facing. Some of the problems were actually related to the way bindings are generated. I fixed a couple today, especially related to vararg functions (rather than vararg constructors), after adding a few more test and seeing them fail.

Unfortunately there are still failures that seems to be caused by TagLib. In particular writing to specific tags like ID3v1 or APE tags on mp3 files, or any kind of tag on mpc files seems to merely fail.

First I thought I did call save() functions in the wrong way, but after adding consants declaration support to the generator, that wasn’t the case.

After a whole day writing testcases and running the increasing number of tests over and over, I got to a series of bad conclusions, unfortunately :( Musepack (MPC) support in TagLib is mostly broken: although I didn’t try to actually read the tags (as mppenc does not allow me to set them), trying to write them fails. FLAC support is unstable, a lot unstable: it can’t read tags nor audio properties from a simple FLAC generated on the fly by flac command, although the data is there when I look at it with metaflac; I can write ID3v2 tags but I can’t write v1 or Xiph comments; OggFlac support is incomplete too, as it marks as invalid OggFlac files created without tags in the first place. Ogg Vorbis support seems quite solid tho. And it’s actually a lot simpler to deal with them through TagLib’s API than dealing with ID3v2 tags, for once.

The RubyTag++ testsuite is now over 30 different test, trying to check as much things as possible on the filetypes available. For every file type I wrote at least a “basic” test, where the tag fields are set by the encoder (if it can, so not the case of MPC), a “tags” test (for formats that supports more than one tag type) and a “fields” test (for accessing fields directly through the map) that replaces the “tags” one for Ogg formats, a “fileref” test for FileRef class (that fails for OggFlac files as .ogg extension is always considered Ogg Vorbis), and it’s quite unrealible anyway; and a series of “write” test with the generic tag interface or with the specific interface of the various types of tags. I’m going to add more tests about audioProperties tomorrow, too.

Although Wheeler’s (and the rest of TagLib’s team’s) work is pretty amazing, right now it’s reliable only for reading, and that might be the reason for which there are not much KDE taggers I’m afraid. The API is pretty consistent, the documentation almost errorless, and the code well written. Unfortunately there are many corner cases that were never tested. I think that missing a testsuite have a good amount of cause behind this. If someone wants to write a suite of crosstests against mine, using the native C++ API it would really be useful to have it ran on different TagLib’s versions to make sure that all goes well.

I hope to be able to report the problems I find with a solution on taglib-devel in the next days.

Anyway, I’m absolutely satisfied with my generator script. I’ve refined it more and more after working with RubyTag++ in practice, it now handles correctly vararg functions and constants, it also alias the functions that uses CamelCase capitalization to the ruby_underlined_style (initially by defining more functions, but tonight I made it use the actual alias support in Ruby).

After RubyTag++ testsuite is complete enough, I’ll work on adding a proper testsuite to the generator code, some C++ classes with a description file and some ruby test scripts that checks that functions, constants and similar are correclty defined :)

If anybody wants to help me on this project, I’m just waiting your mails ;)

And now, let’s test

I think this is a common problem when working on a program and its backend implementation in parallel. Whenever I come to need a new function in JulTagger, I need to start hacking at RubyTag++. Unfortunately, it’s not difficult to hose the whole package when doing a change, so I’ve decided to start adding a series of regression tests.

But this is not news, I already blogged about my fights with cmake to have it running my tests.
At the end I decided to do it in a strange way: every ruby script is responsible for decompressing and converting the wave file to the compressed file I need it to.

Now, the problem is to write a sufficiently complete test suite, I do have 10 tests, but I miss entirely tests on flac files, as I don’t seem to be able to set the tags correctly with either the flac or metaflac commands. I also miss lots of extended tests, and there’s no tests about mp3 headers right now.

Of course writing these tests has to be done in parallel with adding of new classes and functions to RubyTag++, and indeed yesterday I added MPC support just to add the tests, although I didn’t have much to do with them (that’s also the reason why I merged musepack-tools here and on farragut – I ran the tests on both Linux and FreeBSD – and why I did keyword a few packages with it).

Now the tests are really just partial, tests if tags are being loaded correctly, if the tags types are reported correctly, and so on. I’d like to get some help, so I’ll try to write some documentation about RubyTag and the generator script so that if anybody wants to help, he’ll know how to do it. After that, it should be pretty straightforward to avoid adding bugs when the tests all passed.

By the way: the only test right now that fails is related to mp3 tags, for some reason TagLib::MPEG::File::ID3v2() function returns a TagLib::APE::Tag instance instead, and that might as well be an error in my generator script, or an error in TagLib itself, for what I can tell. Sigh.

Anyway, today I don’t feel that well, and I’m unable to go out as I should and would have done, this mean that my mood will be very very unstable in the next hours at least.

Really, I should drop the coke entirely..

CMake, tests, headaches

So, JulTagger is proceeding rapidly, unfortunately yesterday I was shopping (or trying to find a book, to be more precise) and had not much time to work on it, although I worked a bit on RubyTag++ so that more bindings are now present.

With regard to RubyTag++, I thought last night that it would have been good if, instead of my silly simple test script (that uses my own MP3s and OGGs), I could write a series of battery tests so that i can just use “make test” to make sure I didn’t break anything while adding functionalities.

Not a bad idea… the first problem was to get an empty wave file so that it would be converted to mp3/ogg/flac to run the tests. I asked a friend for a simple 5 seconds wave file, but unfortunately he wasn’t able to get me a mute one (lovely when there’s no way to record the “null of universe”), so I did it the hacker’s way: took the header out of that file, created a new file, truncated it at the size of the original one, put the old header there. A simple bzip2 after, a 800KB wave file became a 122 bytes file.

Then it comes the time to get the tests written and run. Unfortunately the documentation about writing tests with CMake is quite vague, the man page does not really contain much useful things a part telling me to use ENABLE_TESTING() and ADD_TEST(), but how to tell it that for a test I need some files created seems to be a bit difficult.

Anyway, I’ll get a bigger headache later trying to grasp this, I just hope not to get one too big as tomorrow night I’ll be partying for the birthday of a friend of mine. Sometimes I get out of this room, and not to enter another machine room :P

Ruby, the new bindings and C++

So, I’m still working on getting RubyTag++ finished; today I also started again working in JulTagger, converted it to use RubyTag++ and started supporting more than one file selected at a time.

Unfortunately, when I was trying it, I ended up with a crash before and a runtime error (coming from RubyTag++), which meant that my current way to bind instances is not yet complete. Sigh.

Luckily, I had an answer from Richard Dale about the problem I blogged a while back with the evil evil hack needed to be able to get a correspondence between the ruby objects and the C++ instances. Less luckily, that way makes more difficult to create C++ objects, as the allocate function that replaces the initialize one does not pass the parameters that I need to pass to the new operator.

I’ve looked for documentation about it with Google, but I found nothing; Programming Ruby is completely useless about this as it still describe the old method. I found a few question asked in ruby’s mailing lists, but nothing actually useful for what I needed.

I finally found the solution in looking at wxruby code, that is C++ and uses the new interface. The trick seems to be to provide a fake pointer in the alloc function, and then just create the object in the initialize function.

I’ll be updating RubyTag++ later today, and then return working on JulTagger.

RubyTag++ is alive!

So, after last evening work, I passed the night (yes, the whole night) working on RubyTag++. It wasn’t easy, I had to smash my head against the wall a couple of times at least as there are things on the documentation that doesn’t really seem to behave as told.

At the end I worked around by using an empyrical evil evil hack, that I won’t comment over for now, as it was probably sheer luck if it worked, but anyway that allowed me to proceed to concentrate on more things.

At about 7am CEST I was able to read basic tags out of FLAC files, a couple of minutes later I was also able to set them. Okay not much useful this randomness, but worked :)

I had to fight also with optional parameters, but I resolved by adding more data to the yml and then having a switch/case statement to get the right call when the parameters are omitted.
Maybe the generated code is not elegant, but screw that, it’s generated code, after all :P

I’ve then added a few more supports in the yml that’s still in flux, as I’ve changed a couple of declarations in the run, it will probably remain in flux until it’s entirely finished.

Right now I can: get and set the basic tags as if they were variables in Ogg and FLAC tags; strips tags from MP3s; get basic audio information on files; get extra information out of Ogg Vorbis files.
Not bad after all, considering that I’ve wrote a good deal of it during the night instead of sleeping.

Adding more support like ID3v1/ID3v2 access should be just matter of tweaking the description yml at this point.
As soon as I can do that reliably, I’ll restart the work on JulTagger, my Korundum audio file tagger, so that I can get rid of EasyTag entirely :)

On a little note, I’d like to say that functions like File::isReadable and File::isWritable and similar are not being binded, why should I, when Ruby already provides the standard library for those? :P

RubyTag++ finally loads

Okay, tonight I wanted to relax a bit as it’s being a really bad real-life period, although I’m having a few “professional” satisfactions that keeps me up. I’ve decided to finish the install of Gentoo on my iBook, but it was halted at about January, so I’m still waiting for GCC 4.1 to build there, so that I can use distcc after.

I decided also to cleanup the metadata of video-herded applications, by setting the right alias as email of maintainer in the files, but that was a completely scripted series of commits so it didn’t actually relax me at all :P

I decided to continue my work on RubyTag, that I have moved to CMake some days ago.
Albeit the first impressions, CMake is working fine after some fighting, so today I decided to go a step further, split the generator script in more than one file, so that I can structure it a bit more, and then go down to compile. The main problem I had was of having CMake drop the “lib” prefix, but I solved that by setting the PREFIX property to nothing.

I got first a static library, totally useless, but then looking at cmake’s man page and googling a bit, I was able to get a rubytag.so module that was the base to get it loaded.
I had to fix a couple of things that I didn’t take into account before, the first being that the built code was exported as C++ as TagLib is C++ and I did forget to add extern “C” declaration, the second was that I was naming the initialisation function in the wrong way. I fixed both and now the extension load.

Yes it doesn’t do anything actually, as the constructor is not bound to the “new” method, so it just stays there and doing nothing, but it loads and that’s more than I have achieved with M4. Too bad I don’t have the money to add to my Amazon order also Programming Ruby, or it would have been probably easier.

Anyway, look forward for having C++-based Ruby bindings to TagLib, as soon as those are done, I’ll return working on JulTagger, too :)

Now, I should probably go to sleep.

My CMake impressions

So, as I said, I wanted to try some alternative to automake for RubyTag++ as I was able to get rid of M4 for the bindings’ generation. CMake was my choice as KDE is using that for the work on the new KDE major version (KDE 4).
This was also an experiment as I wanted to know how it worked, as I’ll have probably to fight with that build system when KDE 4 will be released.

So I’ve emerged it, and for first thing, I looked if it had some kind of support for ruby, I found the /usr/share/cmake/Modules/FindRuby.cmake file and looked at it. I think I rarely seen something that broken. First of all, it doesn’t really look for all the needed information to make use of Ruby, like the path where to install the ruby extensions.

A part that, that might just be a simple omissis that’s not “the best”, it also hardcodes the include path to “i386-linux” that, mind you, it’s entirely broken on anything that’s not i386 and Linux, and if your CHOST is i686, that breaks, too.
It also looks for the library libruby1.8.so and the ruby1.8 executable, that are not common, on Gentoo for instance they are libruby18.so and ruby or ruby18.

So now I have at least to rewrite the Ruby check module, to be able to use CMake, possibly in a better way so that the important variables I need to build and install the extension. Then I have to submit it upstream, and then maybe it will be available next version. In the mean time, I don’t think I can release rubytag even if my generator is completed.

This said, I don’t think the first impressions with CMake are that good, for me.