The mad muxer

I have expressed my preference for the MP4 format some time ago, which was to be meant mostly over the Ogg format from Xiph, most of the custom audio container formats like FLAC and WavPack (to a point; WV’s format could be much worse), and of course the AVI format. Although I do find quite a few advantages of MP4 over Matroska, I don’t despise that format at all.

The main problem I see with Matroska, actually, is the not so widely available implementation; I cannot play a Matroska video on either my cellphone (which, if somewhere were to think about it, is a Nokia E71 right now, not an iPhone, and will probably never be, I have my limits!), the PlayStation3 or the PSP. Up to last month, I couldn’t play it with my AppleTV either, which was quite a bit of a problem. Especially considering a lot of Anime fansubbers use that.

Now, since I was actually quite bored by the fact that, even though I was transcoding them, a lot of subtitles didn’t appear properly, I decided to try out XBMC; it was quite a pleasing experience actually, the Linux-based patch stick works like a charm, without going too much in the way of infringing Apple’s license as far as I can see, and the installation is quite quick. Unfortunately the latest software update on AppleTV (2.3.1) seems to have broken xbmc, which now starts no longer in fullscreen but rather in a Quartz window in the upper half of the screen.

So I ditched also XBMC (for now) for a derivative, boxee which, while being partially proprietaryware, seems to be a little more fine-tuned for AppleTV; it’s still more free software than the original operating system. Both XBMC and Boxee solve my subtitles problem since they both have a video calibration setup that allows to tell the software how much overscan the LCD panel is doing, and which pixel size ratio it has. Quite cool, Apple should really have done that too.

Also, I started using MediaTomb to serve my content without having to copy it on the AppleTV itself; this is working fine since I’m using ethernet-over-powerline adapters to connect the office with my bedroom, and thus there is enough bandwidth to stream it over. Unfortunately, here starts the first problem: while I was able somehow to get XBMC to play AVI files with external .srt subtitles, it fails on Boxee. Since the whole thing is bothersome anyway, I wanted to try an alternative: remux the content without re-encoding it, in Matroska Video files, with the subtitles embedded in them as a third track.

This seems to work fine from an AVI file, but fails badly when the source is an MP4 file, the resulting file seems corrupted with MPlayer and crashes Totem/GStreamer (that’s no news, my dmesg output fills easily with Totem’s thumbnailer’s crashes when I open my video library). Also, I have been yet unable to properly set the language of the tracks, which would help me to have the jap-sub-eng setup automatic on XBMC. If somebody knows how to do that properly, I’d be glad.

Anyway, there it goes another remuxing of the video library…

For A Parallel World. Home Exercise n.1: a drop-in dynamic replacement for memcpy()

Since I’ve written about OpenMP I’ve been trying to find time to test it on real usage scenarios; unfortunately between health being far from optimal the past few days with general aches, and work piling up, I haven’t been able to get to work on it at all. It would be much nicer if I could get a job that would allow me to spend time on these things, but I don’t want to rant about that since I’m happy to have jobs from time to time, as it is.

Yesterday I toyed a bit around with OpenMP and xine-lib, I wanted to try implementing a memcpy() replacement that could use OpenMP to work in parallel, it’s especially useful for multimedia applications. Beside some issues with autotools and OpenMP which I’m going to address in a future post, I ended up with a few more things in my mind (the usual problem with trying out new things: you want to achieve one result, and you get material for other three tests; now I know why Mythbusters starts with one idea and then end up doing four or five similar tests).

My parallel memcpy() replacement was just as lame as my byteswapping attempt from before, a single for parallelised with the proper pragma code. Just to make it not too lame, I used 64-bit copies (unaligned, but I would expect that not to matter on x86-64 at least, it was just a test). The reason why I didn’t go for a less lame method is that from a second test on byteswapping, which I didn’t have time to write about yet, I found that using more complex tricks does not really help. While splitting the memory area to swap in X equally-sized areas, with X being the maximum number of threads OpenMP is managing, identified dynamically, caused a slight improvement on the huge areas (half a gigabyte and a full gigabyte), it didn’t make any acceptable difference (considering the cost of the more complex code) on smaller blocks, and I really doubt that such huge memory areas would ever go swapped all at once. Splitting the area in page-sized (4KiB) blocks actually made the code slower, I guess, since I didn’t go deeper to check, that the problem there is that the threads are usually all executed on either the same core or on anyway on the same CPU, which means that the pages are all mapped on the memory directly connected to that CPU; splitting it up in pages might make it swap between the two different CPUs and thus make it slower. I’ll look more deeply into that when I have time.

Unfortunately, using this particular memcpy() implementation didn’t let me start xine properly, I probably made a mistake, maybe unaligned 64-bit copies on x86-64 don’t work, just like on any other architecture, but I didn’t go around trying to fix that for the very same reason why I’m writing this post.

It turns out that xine, just like MPlayer, and other multimedia application, have their own implementation of “fast memcpy()”, using SIMD instructions (MMX, MMXEXT, SSE, AltiVec, …). They benchmark at runtime which one has the best result (on my system it’s either the Linux kernel implementation, not sure which version, or the MMX version), and then they use that. This has some problems that are obvious, and some that are much less obvious. The first problem is that the various implementations do have to take care of some similar issues which cause code duplication (handling of small memory area copies, handling of unaligned copies and so on). The second is much more subtle and it’s what I think is a main issue to be solved.

When a programmer in a C program uses functions like memcpy(), strlen() and others, the compilation process (with optimisations) will hit some particular code called “builtins”. Basically the compiler knows how to deal with it, and emits different machine code depending on the way the function is called. This usually happens when the parameters to the call are known at build time, because they are either constant or can be derived (for static functions) from the way the function is called. How this affects mathematical functions and functions like strlen() can be better understood reading an old article of mine; for what concerns memcpy(), I’ll try to be brief but explain it here.

Let’s take a very easy function that copies an opaque type that is, in all truth, a 64-bit data field:

#include 

void copy_opaque(void *to, void *from) {
  memcpy(to, from, 8);
}

Building this code on x86-64 with GCC 4.3 and no optimisation enabled will produce this code:

copy_opaque:
        pushq   %rbp
        movq    %rsp, %rbp
        subq    $16, %rsp
        movq    %rdi, -8(%rbp)
        movq    %rsi, -16(%rbp)
        movq    -16(%rbp), %rsi
        movq    -8(%rbp), %rdi
        movl    $8, %edx
        call    memcpy
        leave
        ret

As you can see there is a call to memcpy() after setting up the parameters, just like one would expect. But turn on the optimisation with -O2 and the resulting code is quite different:

copy_opaque:
        movq    (%rsi), %rax
        movq    %rax, (%rdi)
        ret

The function has been reduced to two instructions, plus the return, with no stack usage. This because the compiler knows that for 64-bit copies, it can just emit straight memory access and simplify the code quite a bit. The memcpy() function is not a static inline, but the compiler knows its interface and can produce optimised code just fine when using builtin. Similarly, when using -O2 -fno-builtin to ask the compiler not to use builtins knowledge, for instance because you’re using special access functions, you can see that the resulting code is still composed of two instructions, but of a different type:

copy_opaque:
        movl    $8, %edx
        jmp     memcpy

Let’s go back to the builtin though, since that’s what it’s important to know before I can explain why the dynamically-chosen implementation in xine and company is quite suboptimal.

When you change the size of the memory area to copy in copy_opaque() from 8 to a different constant, you can see that the code changes accordingly. If you use a number that is not a multiple of 8 (that is the biggest size that x86-64 can deal with without SIMD), you can see that the “tail” of the area is copied using smaller move operations, but it’s still expanded. If you compare the output with multiple power-of-two values, you can see that up to 128 it inlines multiple movq instructions, while starting with 256, it uses rep movsq. With very big values, like (1 << 20), the compiler emits a straight memcpy() call. This is because the compiler can assess the overhead of the call and decide when it’s big enough to use a function rather than inlining code.

It can also decide this based on what type of optimisation is requested, for instance I said above that rep movsq starts to get used after the value 256 (1 << 8), but that was intended with the -O2 level; with -Os, it’s already when you have more than two 64-bit words.

Since the library functions like memcpy() and similar are very widely used, the fact that the compiler can emit much simpler code is very useful. But this works just as long as the compiler knows about them. As I said, turning off the builtin replacement will cause the code to be compiled “literally” with a call to the function, which might have a much higher overhead than a straight copy. Now it might be quite easier to grasp what the problem is with the dynamic memcpy() replacement used by xine and other software.

Let’s change the code above to something like this:

#include 

extern void *fast_memcpy(void *to, void *from, size_t foo);

void copy_opaque(void *to, void *from, size_t foo) {
  fast_memcpy(to, from, 8);
}

Now, even turning on the optimisations, won’t make any difference, the compiler will always emit a call to memcpy():

copy_opaque:
        movl    $8, %edx
        jmp     fast_memcpy

As you might guess, this is certainly slower than the straight copy that we had before, even if the memcpy() replacement is blazing fast. The jump will also require a symbol resolution since fast_memcpy() is not statically defined, so it’ll have to pass through the PLT (Procedure Linking Table) which is an expensive operation. Even if the symbol were defined internally to the same library, this would still most likely cause a call to the GOT (Global Offset Table) for shared objects.

By redefining the memcpy() function, xine and others are actually slowing the code down, at least when the size of the copy is known, a constant at build time. GCC extensions actually allow to define a macro, or even better a static inline function, that can discern whether a compile-time constant is used, and then fall back to the original memcpy() call, which the compiler will mangle as it prefers, but this is quite complex, and in my opinion not worth the bother.

Why do I say this? Well the first issue is that sometimes even if a value is not properly a constant at build-time, the compiler can find some particular code path where the function can be replaced, and thus emit adaptive code. The second is that you might just as well always use properly optimised memcpy() functions when needed, and if the C library does not provide anything as fast, you just need to use the Force ELF.

When the C library does not provide functions optimised for your architecture, for compatibility or any other reason, you can try to replace them through the ELF feature called symbol interposing, which basically work in the same as symbol collisions (I have some “slides” I’ve been working on for a while on the subject, but I’ll talk more extensively about this in a future post), and allows to intercept or replace calls to C library functions. It’s the same method used to implement the sandbox used by Portage, or the OSS wrappers for ALSA, PulseAudio, ESounD and so on.

What I’d like to see is a configurable library that would allow to choose between different memcpy() implementations, maybe on a per-size basis too, parallel and non-parallel, at runtime, through a file in /etc. This is quite possible, and similar features, with replacement for many common library functions, are available with freevec (which unfortunately only implements AltiVec on 32-bit PPC). But a more arch-neutral way to handle this would most likely help out.

Anyway, if somebody is up to take the challenge, I’d be glad to test, and to add to the tinderbox to test on the packages’ testsuites too. Let’s see what happens.

Frontends to command-line or libraries?

I know I’m still convalescing so I should be resting, not thinking about development problems, but this is something that ended up in my mind because I have one thing to absolutely do (scan the release documents from the hospital to send to my GP), and for which I miss an easy interface I could instruct my mother, or my sister, to use.

Don’t get me wrong, I know there are a few SANE frontends, starting from xsane itself, but the problem is that I don’t usually stop at scanimage when I do it from the console. What I usually do is to launch a batch scan to TIFF format, then use tiffcp to join the different TIFF files in a single file with multiple pages, and then use tiff2pdf to convert it to a PDF file, which is opened by any computer I might need to send the data to (and it also is quite smaller than the original TIFF file). Lately, I’ve started trying to add to the chain also unpaper, a tool that removes some of the defects usually found in scanning pages from books (like black borders and similar), which works on PNM (thus requiring a change in the scanning command, and a further conversion later on).

But I don’t want to start fleshing down how to actually write such a tool, or I might actually start writing it right now, which is not what I’m supposed to do while I’m convalescing.

What I want to think about is that here comes one huge debate between writing frontends for command-line tools, which just interfaces with them as process calling, or writing a full-fledged program that interfaces with the low-level libraries to provide similar functionalities.

I already happen to discuss that quite often since xine and mplayer embody the two spirits: xine has its library, frontends interface with that, and a single process is used; mplayer instead has a command interface, and frontends execute a new process for playing videos.

There are of course advantages and disadvantages, one easy to spot disadvantage to xine’s approach is that a crash or freeze in xine results in a crash or freeze of the frontend, which is something Amarok users have been unfortunately familiar with.

In the case of the scanning toolchain, though, I guess it’d be probably easier to use a frontend for the tools, as re-implementing all the different functionalities would be a non-trivial work.

The disadvantages of doing it this way, though, is that you’ll have to make sure that the tools don’t change their parameters between versions, otherwise it’d be a problem to ensure the correct functionality of the tool. Also, the tools need to provide enough options to control with granularity the execution of their task, which is something unpaper actually does, but in turn makes them almost unusable for final users.

I know I’m not going much anywhere with this post, I’m afraid, but I just wanted to reflect on the fact that to have a command line tool designed to be used by frontends, you almost certainly make its syntax so complex that users would fail to grasp the basic concepts, and in turn, you’d need a command line interface to the tool too… which is why there are so many scripts interfacing to ffmpeg for converting videos, I guess.

On the other hand, one can easily write such a frontend using scripting languages, even if it’s graphical, such as with ruby-gtk2 (think RubyRipper).

And what about imported libraries?

Following the previous blog here also a list of projects that seem to like importing libraries, causing code duplication even for code that was designed to be shared.

  • cdrkit, again, contains a stripped down version of libdvdread, added, of course, by our beloved Jörg Schilling; bug #206939; additionally it contains a copy of cdparanoia code; bug #207029

  • ImageMagick comes with a copy of libltdl; bug #206937

  • not even KDE4 seems to have helped libkcal which even in its newest incarnation ships with an internal copy of libical, causing me to have three copies of it installed in my system;

  • libvncserver comes with a copy of liblzo2; actually there are two, one in libvncserver and one in libvncclient; even the source files are duplicated!; bug #206941

  • SDL_sound, Wine and LAME seem to share some mp3 decoding code, which seems to come originally from mpg123;

  • cmake couldn’t stay out of this, it comes with a copy of libform (which is part of ncurses); follow bug #206920

  • I’m not sure what it is, but DigiKam, Numeric (for Python) and numpy have a few functions in common; the latter seems to have even more than that in common; bug #206931 per Numeric and numpy, and bug #206934 for DigiKam.

  • ghostscript comes with internal copies of zlib, libpng, jpeg and jasper; unfortunately jasper is also modified, for the other three there’s bug #206893; by the way, the copies are present in both the gs command and in the libgs library;

  • OpenOffice comes with loads of duplicated libraries; in particular, it comes with its own copy of icu libraries; see on bug #206889

  • TiMidity++ comes with a copy of libmikmod; bug #206943

  • Korundum for KDE3 has a copy of qtruby embedded, somehow; I wonder if it isn’t a fluke of our buildsystem; bug #206936

  • gdb contains an internal copy of readline; –bug #206947

  • tork contains a copy of some functions coming from readline; bug #206953

  • KTorrent contains a copy of GeoIP (and to think I removed the one in TorK as soon as I’ve spotted it); bug 206957

  • both ruby and php use an internal copy of – I think – oniguruma; I haven’t looked if it’s possible to add that as a system library and then use it; bug #206963

  • MPlayer seems to carry a copy of libogg together with tremor support; bug #206965

  • pkg-config ships with an internal copy of glib; bug #206966

  • tor has an internal copy of libevent’s async dns support; funny, as it links to libevent; bug #206969

  • gettext fails to find the system copy of libxml2, falling back to use the internal copy; at least it has the decency of using a proper commodity library; bug #207018

  • both Perl and Ruby have a default extension based on SDBM, a NDBM workalike; there seems not to be a shared version of it, so they just build the single source file in their own extensions directly, without hiding the symbols; beside the code re-use not being available, if a process loads both libperl and libruby, and in turn they load their sdbm extension, stuff’s gonna hurt;

  • enchant has an internal copy of Hunspell; probably due to the fact that old Hunspell built only static non-PIC libraries, and enchant uses plugins; bug #207025; upstream fixed this in their Subversion repository already;

  • gnome-vfs contains an internal copy of neon; funny as it depends on neon already, in the ebuild; bug #207031

  • KOffice’s Karbon contains an internal copy of gdk-pixbuf; bug #209561;

  • kdegraphics’s KViewShell contains an internal copy of djvulibre; bug #209565;

  • doxygen contains internal copies of zlib and libpng; bug #210237 ; this time I used a different method to identify it as doxygen does not export the symbols;

  • rsync contains an internal copy of zlib; bug #210244 ;

Unfortunately making sure that what I’m reading is true data and not false positive, looking at the output of my script, becomes more difficult now for the presence of multiple Sun JDK versions; I have to add support for alternatives, so that different libraries implementing the same interface don’t show up as colliding (they are that way by design).

Yai! More FFmpeg in xine!

I have to thank Aurelien a lot, now xine-lib-1.2-libavutil branch works fine out of the box with an unpatched FFmpeg copy!

He modified the CRC code from FFmpeg to actually be installable, and then made sure that he header was installed, with this, xine-lib can use FFmpeg’s libavutil to calculate checksums (CRC and SHA1) and to encode to base64 (which was used in three places already: CDDA input plugin, to fetch CDDB data, librtsp for RTSP protocol handling and the HTTP input plugin for digest authentication).

Reducing the redundant code in xine also means saving memory space, and as we already use FFmpeg, this does not increase the amount of code that is loaded and dynamically linked to.

The libavutil branch also spots a huge change: internal FFmpeg is no more, now you need to build a copy of FFmpeg from SVN (or use the one your distribution packaged), to be able to use xine-lib-1.2; it actually requires a fresh copy out of current FFmpeg SVN at the moment, so that Aurelien’s patches are present. I think Debian should package a new snapshot soon enough, and probably should Luca although I use a live FFmpeg ebuild for my development needs ;) ).

There were still a few rough spots regarding the removal of internal FFmpeg in the Mercurial code, the first was the need for the MANGLE definition, that FFmpeg project does not want to export, so I added it to an header inside xine-lib for now, until we can look at the code to see if it can be rewritten not to use MANGLE at all.

Another was the dvdata.h header that is used by the “FFmpeg-based” DV Audio decoder. I put the quotes because no FFmpeg code was used, just the dvdata.h header; the plugin is now moved away from the ffmpeg plugin directory, and a copy of dvdata.h was imported as ff_dvdata.h.

As I was looking at that, I also removed the code used by the DXR3 plugin from the ffmpeg plugin. Up to now, to use the FFmpeg mpeg encoder, the DXR3 plugin added code to the FFmpeg decoder plugin, to initialise the encoder, then the video output driver dlopened the FFmpeg plugin, loaded the init function and called it. This was because the FFmpeg functions were built in the FFmpeg plugin when internal FFmpeg is used. Now that FFmpeg is no more used internally, I simply moved the FFmpeg-based encoder into dxr3 itself, make it link to FFmpeg, and be done with it. This actually removes the need of exporting an extra function from the FFmpeg plugin.

Now that I think of it, I forgot to remove DYNAMIC_LD_LIBS from the linked libraries of the DXR3 plugin, it shouldn’t be needed anymore as no dlopen() is used, which should make it faster at runtime too.

Talking about FFmpeg, with one of my recent warning-fixing spree in xine-lib (too bad I can’t remove all the warnings without breaking API – double pointers are evil), I found a function in libpostproc not accepting constant strings by prototype (see this entry), and then seen the pp_help entry which got defined in .data; RSS for libpostproc should be down now, but I start to wonder if the library is a bit rough, and in need for some improvement. I’ll probably look at it too, as xine uses it.

FFmpeg itself should instead be quite more friendly when it comes to table being constant and properly in .rodata; I found a few things that are in .data but could go in .data.relro though, I’ll probably take a look at it tomorrow – remember: .data.relro stuff could be mitigated with prelink, if prelink worked properly .

Instead, unrelated to FFmpeg but still related to xine-lib, tomorrow I hope to refresh the new libdvdnav branch, and then proceed to merge it in 1.2 quite soon; the new libdvdnav branch uses the libdvdnav code that is now developed by the MPlayer developers (Nico, mostly), while xine-lib currently uses an internal copy of libdvdnav coming from linux-dvd’s CVS. If we migrate to use MPlayer’s libdvdnav, we can finally use an external version, and we slim down the DVD input plugin quite a lot, which is not bad at all.

I sent two patches for libdvdnav tonight before going to bed, one fixes a lot of string tables so that they can go to .rodata and so take up less memory, and one to mark a single variable static (the variable could also be removed by using UINT16_MAX, but that’s for tomorrow).

Too bad that libdvdread implements its own MD5 routines, if it used libavutil we’d be saving more memory again. I suppose I could look at patching libdvdread too in the future. A perfect setup would have libavutil providing basic implementations for various multimedia software, so we don’t have to invent the wheel every time…

What if GPL software couldn’t make use of OpenGL?

As we all know, what keeps Free Software alive is respecting the licenses: if people didn’t respect the GPL, Free Software wouldn’t have any appeal for most amateurs who wouldn’t want their code stolen.

This also means that Free Software needs to respect licenses, even when the user can’t make use of features he’d very much like.

Now, let’s make an hypothesis that GPL software can’t make use of OpenGL, that’s a very bad view you’ll have: no more games, no more accelerated Xorg, no more compiz or things like those. At least they wouldn’t be redistributable in binary form, and the legality of the code would be probably put in doubt by many.

But is it just an hypothesis? I’m afraid it might not be that unlikely now.

Most of the code using OpenGL requires headers, usually shipped with mesa (previously with XFree itself), like glext.h, which are distributed under SGI Free Software License B (see the text), which for FSF is not a Free Software license at all. The same view is shared by Debian, as you can see on their bug #211765.

While Debian is mostly ignoring the issue for now, the problem is there, and the new Xorg setup, with the packages split, didn’t fix the problem. It’s probably a big issue, it’s strange that it’s not so well known.

Of course, if I recall correctly, even nVidia and ATi/AMD drivers install some OpenGL headers, so the problem is even more complex than that. I wonder why there isn’t a fully free set of headers available for use yet.

Unfortunately, I found this out while reviewing xine-lib’s licenses, as there is a copy of glext.h that is shipped for the OpenGL output plugin to build correctly; this will soon be removed from both 1.1 and 1.2 series, probably, so that at least the sources in the repository are a bit cleaner. But the problem here is that I need to follow this issue deeply, as xine-lib is vulnerable to this possible license violation, and it’s not a good thing.

And for who’s wondering about the status of the rest of xine-lib code, well, we have some problems we need to fix, thankfully the only thing that seems difficult to fix is nosefart code, but that I’ll write more about tomorrow, the rest will probably be cleaned up in the next weeks after all the original authors are contacted, and some code is synced from MPlayer.

Thanks a lot to Vir who gave me the sparkle to check the licensing of xine-lib to make sure we are compatible with GPL-3 (that might going to be used for libcdio/libvcd, and is used now by Samba). I wouldn’t have looked so deeply if he didn’t.

xine gets 64-bit Real binary codecs support

Yesterday I was talking with siretart on #xine about the lack of coordination between distributions when it comes to xine packages. Beside him and me, there aren’t many packagers writing on xine-devel, and even between us it is often difficult to coordinate, for instance we talked just yesterday that Debian is not using external libraries, as their policy would be, probably because when the control files were written there was no support for using them. As I’ve changed this for a while now – needing the same in Gentoo – it will now switch to use external whenever possible.

Talking of that, I decided to take a look to FreeBSD ports, renown for not submitting the patches to upstream whatever upstream it is, to see if they had any useful fix to merge. Beside a ton of fixes for FreeBSD 4, quite invasive because they need all the PRId64 and similar macros defined manually, that I’m not going to merge also because they involve crippling a lot of FFmpeg code, I’ve seen a couple of interesting and maybe useful things.

One was a fix for the OSS driver to work on 64-bit FreeBSD systems, as the second parameter (request) of ioctl() function is unsigned long typed rather than int, and there was a temporary used that had the wrong sign. The patch for that was quite invasive and ended up changing a lot of variables from int to long, even some that were passed to the further parameters of ioctl() (which expected them to be int*); the reported problem was fixed, but more likely problems were introduced. I ended up cleaning this patch up first using a __FreeBSD__ conditional, and afterward checking the type of the request parameter during configure stage (and for who’s interested, modern Linux ioctl() also uses unsigned long for the request parameter.. I’m not sure how this didn’t cause problems on 64-bit systems, probably Linux doesn’t use the MSB, so there is no overflow, and sign extension on call; the manpage is wrong on this).

the other set of interesting patches was to add support for Real binary codecs to FreeBSD. I say interesting even if I don’t like binary codecs, I never used them before, and I will probably not use them anymore in the future beside from testing purposes. The reason why I was interested in these was that the Real binary codecs plugins (two of them, for some reason, one for audio and one for video) were being built on every system, without way to disable them, even if they are totally 64-bit unsafe.

I decided this was a good start for cleaning that up. I first started looking to share as much code as possible between the two plugins, so that even merging the two plugins would be possible (working on that right now). Then I put the extra symbols needed for FreeBSD on it.. but I didn’t like the approach of just fake stderr and __environ symbols as it was, so I decided to make them weak aliases to the symbols used in FreeBSD.

As this was far from extensible, though, I ended up, during the night, writing some configure checks so that if the symbols are not found, one by one, they get get weak-aliased with the nearest approximation. I find that autoconf check quite nice, as it allows me to do the minimum checks needed.. when running on GNU/Linux (not sure uclibc/Linux) only the checks for the strict needed symbols are linked, they all pass, and nothing else is tested.

Now, this covers FreeBSD support, but I started this entry talking about 64-bit. Well, when I was looking at the Real plugins themselves, I decided to see how MPlayer fixed the problem with 64-bit.. copied their code, and tested it.. it worked fine.

This means that 1.1.5 release will have AMD64 Real binary codecs support (as well as Alpha) in an improvde real plugin. For Gentoo users, I’ll see to provide a xine-lib snapshot soon, as I need to flesh out a few details with the ebuild, as I changed a few --enable/--disable parameters to --with/--without again, and added an --enable/--disable for real too, so that it’s not build if you don’t want it or can’t run it (we don’t have the Real PPC or Alpha codecs in Gentoo, so the only arches where the real useflag would be used would be x86 and amd64).

So basically the changes coming yesterday are useful for both who wants to use the Real binary codecs and for who doesn’t want to, the first class of users will have a cleaner Real plugin which just works even for Real Audio (instead of failing miserably because FFmpeg tries to take priority over the Real Audio decoder), and one less plugin to load, the latter class will be glad to know they can skip building and loading two plugins :)

Bribes and gifts accepted if you were waiting for these binary codecs :P (no, just kidding).

FFmpeg-compatible demux_real will come, eventually.. the problem is that the data needed by the Real audio decoder and the data needed by FFmpeg aren’t the same, so I have to hack around to provide data for both.. and the problem of this is that I would end up having to copy the same buffer twice, which is far from being fast.. the alternative is to make FFmpeg’s decode aware of the buffer type… oh well, I’ll see what I can do about that in the next days.

Rip, rip, rip

Well, today I have a pretty long TODO list that I’d like to accomplish, although at the top of my list was also to watch Ice Age 2, while relaxing on the couch, that would probably stop me from finishing any other entry on that list… so I’ll remove that one for now. After all yesterday I as lucky, and tired, enough to watch Pirates of the Carribean (which was, by the way, a quite good movie, especially for someone like me so fascinated with swords of all kinds, thanks Alberto for «forcing» me to watch it ;) ).

I also tried to get pan (not PAM this time) to compile on Gentoo/FreeBSD, but after fixing enchant, I found myself stuck with gmime, as that needs a patch too but I’m not sure how to handle it yet (there’s an extra _POSIX_SOURCE that breaks on FreeBSD). I’ll fix this later on today.

Also Bruce M Simpson from FreeBSD contacted me about the nss-mdns port, so I’ve resumed looking for it, I’ll tonight if I can apply what I found today in the sources to get it working somehow. I don’t think it’s too difficult, it’s just scarcely documented.

But in this whole lot of stuff to do, I was hoping to relax by listening to some music.. something I wanted to try, was to rip the 5.1 tracks out of the Rhapsody’s Live in Canada 2005 to compress to WavPack so that I could listen to it through Amarok (one of the reasons I had was also to test the WavPack 5.1 support in xine/ffmpeg and the ability for PulseAudio to play 5.1 files from xine). Unfortunately, even if there are tricks with mencoder to dump the audio of a track through mplayer to an ac3 file (no way to get it through a GUI it seems), I can’t find a way to pass it to wavpack, as both ffmpeg and a52dec produce a stereo wav file, rather than a multichannel one, and WavPack does not work with AC3. I suppose my best bet would be to code myself a bridge between the two, but I don’t have enough time. Sigh.