Again about .la files (or why should they be killed off sooner rather than later)

I’ve been working as an experiment on rewriting xclip to use XCB rather than Xlib. This is mostly because I always have been interested in XCB but I never had time to learn the internals too much.

To make my task easier I ended up using some funcitons that are not available in the currently-released version of xcb-util, the side-package of XCB that contains some higher-level functions that make it easier to replace Xlib.

Beside the fact that xcb-util still haven’t bumped its version, which makes it impossible to check for the right version with pkg-config, there is one interesting point in using the latest available version through the x11 overlay.

Letting alone some problems with being able to actually fetch and install the packages I need (Donnie, I’ll send you the patches later if I can polish them a bit), over the actual GIT tree there are a few patches applied, coming from Jamey Sharp (an XCB developer) from March 2008. These remove one library (libxcb-xlib) and change the locking method used to make Xlib use the same socket as XCB. These changes not only break ABI (without changing the soname, alas!) but also make it impossible to build the old libX11 against the new libxcb. Using the live version of libX11 (that is also patched to use the new hand-out mechanism) fixes this problem, but the result is a way bigger trouble.

First of all, this is a perfectly good example of what I said about preserve-libs. If you are not using --as-needed, and you had libX11 built with xcb USE flag enabled, you’ll have libxcb.so.1 links on almost all X-using binaries in your system; after rebuilding the new libxcb and libX11 (which respectively would install libxcb.so.2, in theory, and let libX11 link to that), all the binaries will have in their process space both the old and the new libxcb. With different ABIs. And that’s a huge problem on itself.

Then there is the other problem, that is related to the .la files I discussed a few months back. As a huge amount of KDE modules (and not limited to) linked to Xlib, they also had libxcb-xlib listed in their .la files dependencies. Which causes everything to fail linking with libtool as it’s looking for the missing libxcb-xlib.la file.

I suppose it’s time to spend time to get a script to fix this situation, but I admit I’m not much motivated at the moment. Especially since my system is pretty slow when it comes to rebuild stuff for testing, and my employer is not going to pay me anytime soon to allow me getting a newer box.

Once the script is available, it should probably be much much easier to get rid of .la files in ebuilds, as we could just say to the users to run the fixing script and be done with that..

But I admit I was planning on doing some different things in the next days, I had little time for myself lately to begin with, and I’m following way too many things at once. Sigh.

This work by Diego Elio Pettenò is licensed under Creative Commons Attribution-ShareAlike 3.0 Unported License.
Based on a work at flameeyes.blog.

If you want to bitch, get your b****y facts straight at least

So already two fellow devs bitched at me because I enabled the xcb plugin in xine-lib-1.1.9 by default. Enabling those by default was not a fluke, and was totally intentional. And I probably know way better than they do what the USE flag does for that package.

First, the reason why xcb is enabled by default is that the next version of Kaffeine, as well as KDE4, will require XCB support. This because it’s the only way to be able to embed a xine-lib frontend inside another application (Kaffeine inside Konqueror for instance), without having to deal with two Xlib connections. Expecting more than a couple of users wanting to check out KDE4 in the near future, I think that enabling xcb by default is a good idea; I’d rather enable xcb support in Kaffeine by default too, as without xcb, opening a video in Konqueror often leads to the browser to crash, together with all the tabs.

Second, enabling xcb in xine-lib will not break non-xcb-using frontends. Christoph wrote separate plugins, for XShm over XCB and Xv over XCB; they use a different setup than X11 plugins, so the frontends will know what they can load and what they cannot. So an XCB-based plugin (and I mean xcb, not libX11 wrapping XCB) will use the XCB plugins, and a “classic” frontend will use the libX11-based plugins. Just like fbxine (the framebuffer xine frontend) will never try to use an X11 plugin, or aaxine will never try to use the OpenGL plugin.

Third, libxine does not get any linking to libxcb directly; and as the xine plugins does not install any .la file, the libxcb dependency will not be carried around.

Fourth, even if Kaffeine had xcb enabled by default, it does not install any library that other packages link to (well beside kaffeine-mozilla-plugin, but I don’t think if that’s actually developed at all), so libxcb can’t be carried around there either.
The only way libxcb can be carried around is by enabling it in libX11.

Fifth, libxcb and libX11 has no common symbol: you can load those on the same address space without causing any collision. libxcb was designed that way.

Sixth, you can have in your system both libxcb and libX11, they don’t collide, software using libX11 won’t magically be ported to libxcb, otherwise we’d be all happy.

Seventh, you can use at the same time libX11 and libxcb applications. This is the nice thing about X11: it’s a network-oriented protocol, so as long as you use the same protocol, you can mix and match client implementations of it. You can easily run an x86 Linux application using libxcb with a SPARC Solaris X11 server as DISPLAY.

Eighth, you don’t need libX11 built as a libxcb wrapper (USE=xcb for libX11) to have xine-lib built with USE=xcb. The plugins are separated, which is the whole idea of it. You could as well have a frontend that only uses libxcb, without libX11 at all.. I was actually going to work on a similar port of xine-ui.

Ninth, you don’t even need libX11 built as a libxcb wrapper to run Kaffeine (which uses libX11 through Qt) with xcb output. libxcb was designed this way; you have two connections between the client (Kaffeine) and the X11 server, one for libX11, one for libxcb. Of course if you have libX11 built as a wrapper for libxcb, you’d get just one connection, and the code from libX11 would be disappearing. But this does not mean that a pure libX11 would break Kaffeine; it would just use more resources. On the bright side, it wouldn’t crash your Konqueror with 20 tabs open if you end up in a site that has embedded video.

And for what concerns “installing libxcb without reason”, the reason I gave you already, it’s not like I’m forcing it on anybody, the xcb USE flag can still be disabled if you don’t want to use it. It’s no more and no less as any other default USE flags. Or should I say that having the cairo USE flag enabled by default in the profile is making me install cairo without reason, as I don’t want to use it?

(And no, I don’t want cairo disabled by default; I actually have the cairo USE flag enabled in my system, and I wanted to look at cairo more deeply one day, I just wanted to take an example from some GNOME-needed technology, just because it seems to me like at least one of the two complainers might not be interested in xcb because it’s simply used by KDE applications at the moment.)

(Actually, gxine in Mercurial should also supports XCB, because, mind what, it works also better…)

P.S.: if you just say “it’s broken and I don’t want to try again”, then I’m just going to ignore you. From now on. Forever.

Serving community

I’m not dead yet (if I continue using this phrase, when I’ll be dead, someone will have to write on my tombstone “he’s dead, finally”), but I’ve been quite swamped out with my job. It’s quite nice when you work on two months on a thing, without test data, and when you get test data and you get to produce the test suite you found that everything you’ve being doing was based on a wrong assumption… nice, really, not.

Anyway, since I left Gentoo, I’ve decided to change a few other things in my style. Although I did report a few bugs on the bugzilla for a couple of issues I had, most of my work lately has been upstream. Yesterday accounted for a few fixes in xine, three patches for Valgrind (that you can find in my overlay, if you want to look at them), and one for FFmpeg (that will pass unnoticed as usual). As working downstream works nice only if you can actually be helpful to users by taking action immediately, it’s less a priority for me to work on that side, although it’s obviously not entirely gone, so you can see my work on an init.d script for rbot (always in my overlay), that I needed so that ServoFlame doesn’t need to be started by hand – and with the new connection handling code in SVN, it’s actually coping nicely even with network failures – and on an updated xine-ui snapshot that is now in portage, solving the obnoxious bug with the doubleclick to go fullscreen. In all this, I stopped writing down the timetable for stable for the packages though.

Today instead I tried working on the Rust project a bit more, mainly by writing a new page (not yet linked on the site, so I won’t link it here just yet) with some step by step guide on how to set up a developer’s repository for Rust, complete with CIA-on-push and mail-on-push so that the rest of the project (users and other devs) would be able to review the change (this was suggested in a nice video from Google’s TechTalks — you can watch the video with your favourite media player, that I hope will be xine, by downloading it for the PSP/iPod; the mp4 H264/AAC file played nice for me on xine even with the green blobs.. see later on this entry); unfortunately the default update script for git installed in the hooks directory does not mail the actual diffs as far as I can see, and it doesn’t really provide a decent subject, so I’ll have to work on it a bit tomorrow.

I wish to thank David who started working on Rust with me, he’s probably the first person who was ever interested in working in a project of mine :) This is why ost of my projects ended up dead, being the only one working on them, I often felt liek they are of no interest to anyone else.

Anyway, let’s move on a bit on the green blobs I named above. If you ever tried to play an H264 stream with xine, you might have experienced this problem: green rectangles touching hte left border of the video appearing through all the stream. It doesn’t happen on all streams, but it does on some. With Valgrind’s help I was able to find the «leaf cause» of this problem, as it reports an “invalid read of size 4” in a section of the MMX code, and I can report that by disabling MMX code in FFmpeg itself the problem is solved, but why this happens it’s still a mystery to me, someone else should look into the issue… Mike that would be you, possibly :)

Anyway, tonight I was finally able to stop worrying about Windows’s crazy behaviour when a symlink is stored in a 7zip file (the extracted file is only a few bytes long, but being named .exe is enough for Windows to try running it… luckily it only produced an Invalid Instruction kind of error, rather than killing the system altogether), so tomorrow a new segment of work will start for me, and that might give me more time between one battery of tests and the other.

One of the things I am considering to relax, is to replace my current setup of xmodmap, xbindkeys and xvkbd by writing a new application that binds mouse buttons press with a keysym press, so that I can finally stop patching evdev with Olivier’s patch to use my LX700 keyboard, and finally close that odissey. With XCB the task actually seems way easier than with Xlib, I probably can write an xbindkeys workalike in very little time, but there is more involved in this as I want to replace three programs and not just one.

Right now I have evdev patched so that some of the “buttons” of the keyboard (the ones that are actually on the keyboard rather than on the mouse) are reported as keys again, then I xmodmap their keycode to a keysym (that is also bound on KDE’s shortcuts) and I can use those; for the forward/next buttons on my mouse, I instead use xbindkeys to bind the mouse press to a call to xvkbd to send the keysym, this work with a fork() call which is far from being lightweight. What I want to do is to use a vanilla evdev, no xmodmap, no xbindkeys and no keysym, but a xmouse2sym background process that reads a simple file containing a series of pairs «button = keysym» (similarly to .xmodmaprc), find if the keysym is already registered, in which case the proper keycode would be used, for the keysym that are not already registered, it would take a free slice of unused cods to allocate the key.

I’ve looked up the XCB methods to do this, and it’s mostly straightforward, and it should also be quite fast to implement with XCB compared to Xlib, but the big problem is that the string form of the keysym cannot be used internally, you need to use the number form, which usually is returned by XStringToKeysym() function, client-side… this function is not present in XCB itself, so I have either to use Xlib to get that function, or I’ll need to write my own function to get this data. Such a function would be nice in xcb-util so I might look into it… maybe I’ll be able to finally learn lexers and parsers ^_^

Okay so it’s now quite late and I should be sleeping.. it’s just that I don’t really feel that nice lately after sleeping, probably an aftermath of leaving Gentoo, or just my life that’s trying to tell me I’m not immortal, and I should employ it better, who knows…

A new step forward for XCB

If you remember, I always considered libxcb a good idea, especially considering the size of the symbols exported by libX11 and thus the time needed for the binding of those. I spent some days to make libxcb working on FreeBSD, and the result was pleasant after that; I’ve been using libX11 as a compatibility layer to XCB for a while now, and beside Java not working, nothing had problems up to now.

And now, thanks to Christoph & Christophe (Pfister and Thommeret), there’s another step forward for XCB support: there are now two more output plugins for xine: xcbxv and xcbxshm, that provides the same output of XV and XShm, but with XCB instead of X11. They are not yet merged into xine-lib, but I’ll probably commit them in the next days; in the mean time I’m testing them locally.

What is so important about this? Well, first of all next version of Kaffeine, 0.8.4, will require xcb, and the reason for this is that finally the KPart support can be used safely, without everything breaking loose when you close Konqueror, for instance.

I’m quite happy with this, but I’m not going to commit it just yet; tomorrow I’ll probably see to put it in my overlay instead, that would be a start for the testing.

Thanks Christop{,e}s for this! :)

Edit: I’ve just committed the ebuilds for patched xine-lib and patched Kaffeine in my overlay, that is available through GIT.

Please give it some test, and report to [omissis] if you have troubles with that; if nobody reports problems, I’ll probably try to commit those two to portage under package.mask, and see what happens there.

XCB on FreeBSD: case closed!

No I’m not referring to the American title for Detective Conan (although in the past I used to misquote titles of episodes and animes for my blog titles), but just saying that finally a released libxcb works fine on FreeBSD (specifically, Gentoo/FreeBSD).

Today XCB 1.0 was released (and thanks to Donnie is already in portage – beside, thanks for getting the accent right :). This version works out of the box on FreeBSD, as it uses the new libpthread-stubs to get the stubs needed for it to run.

Now, Florent, I’d like to ask you the favour to double check what you post on comments: you got at least one contact from a FreeBSD derivative project, okay not for X11, but you were :) And not always when tarballs are released by XOrg project they were already checked and fixed by Eric, like it happened for libxcb’s release candidates.

And yes, as you workaround a lot of issues from within ports, I’m mostly not contacting you, but instead trying to get it working out of the box, although that often is not as easy as one could hope (like now with Firefox).

Slowness

Today, I decided to try again looking at why Firefox 2.0 fails to build on Gentoo/FreeBSD whereas the 1.5 version built out of the box and worked fine.

The problem was that by linking to libc_r, the implicit link to libc was dropped (it’s one of the mangling of defaults by GCC I suppose, and that is supposed to happen anyway, for things like libc_p.a); the result was that functions like strlen() were undefined (obviously) and thus the link couldn’t continue.

The ports just forced -lc to every link, which is kinda rough way to get away with it, especially since libc_r is told to be deprecated in recent versions of FreeBSD, being the userland implementation of threads, taht is, yes, the only way to run 32-bit applications under 64-bit kernels, but also is almost the slowest implementation available.

As libxcb’s pthread problem is now solved upstream (thanks to Jamey and Josh who created pthread-stubs, that’s a really cool thing, it would be even better if gettext used that instead of coming with yet another set of weak links), I’ve decided to try fixing this issue too.

The first thing is that mixing libc_r and libpthread (that is used for instance by glib) is likely to produce bad results, as the symbols gets mixed together, so I’ve tried to make sure that libc_r was not used and libpthread was used instead.. one can always use libmap.conf to change the selection of threading libraries to use, the choice at link time is mostly a backward compatibility.

Unfortunately to fix this, I had not only to remove -lc_r from being used, but also to relax the checks, allowing undefined references in shared objects (as they are left by -pthread flag). So now Firefox 2.0 builds on Gentoo/FreeBSD!

But it seems like one of the changes I did in the mean time (removed -Bsymbolic, thinking it was just for pthread problems) broke Firefox during runtime, so I’m now waiting for a new build. Unfortunately on the Duron, building Firefox takes something like two hours, especially since it only has enterprise to rely on (I haven’t upgraded farragut’s compiler to 6.2 yet). The SPARC is really faster at this point..

SPARCing around

While I’m listening to Interview to pkgsrc developer Johnny Lam (yes, I do like the bsdtalk podcast), I decided to write some little update on what I’m during on this “less work on Gentoo” week.

A part continuing my job with PHP, MySQL and JavaScript (brr), I’ve also continued using Klothos to keyword some of my preferred tools, like for instance metalog, and avahi, that I use to monitor my network to see which boxes are up.. unfortunately the latter does not complete the registering, and gdb seems to be useless to find where it gets stuck, so I’ll try to debug that better later on. I then decided to shift to something more useful like trying to get catalyst working, but even that ended up as a cul-de-sac because I got a kernel panic as soon as it tried to bind the directories for the chroot, the problem is probably either a bug in the kernel, or a misalignment between the userland (6.2-RC1) and the kernel (6.2-BETA3), I’ll solve that once I’ve got the CF memory working (I also received the memory today, it’s a geek’s dream to get a machine to boot within such a little device..

Tonight I also tested the new Audacious plugins package(s) on FreeBSD to get WavPack and MIDI decoding working, it’s kinda cool to have Audacious playing on a box connected to just a KVM and an ethernet cable, and listening it through the main box’s amplifier, viva PulseAudio! :)

I’m now thinking of branching xine-lib to work on a possible 1.2 version with FFmpeg imported without changes, maybe that would help to sync it more often, but I’m not really sure if I have the time to continue it on right now. I just can’t stand here doing nothing leaving xine dying, I simply can’t.

On good news, XCB problems with FreeBSD are now being solved, thanks to Jamey and Josh the solution will be a standalone library that would stub the needed functions for pthread, being a no-op for GLIBC and other systems where the functions are all stubbed in the C library (I think I should send a PR to FreeBSD about the missing pthread_equal stub, but I’m not yet sure).

On bad news, cdrkit is a desperate case on FreeBSD, gcvt, ecvt and fcvt are missing, and the ebuild currently hard-deps on libcap. I’ll try to see to fix it, but I’m not optimistic.

And now last but not least, a big thank to Mike (Arthur) for Fahrenheit 451, I read that one when I was 11, an old Italian copy of my teacher (class’s library) and I’m really looking forward for reading original Bradbury’s word for it, it’s still a current reading, unfortunately (not that’s a bad reading at all, but it would be surely better if there wasn’t the risk of ending up that way).

Working on a few things

Let’s start by saying that at this very moment my throat hurts, badly, either caused by cold or by the smoke from my boiler (it’s still a wood-fed boiler), my foot also hurts a bit, because of the whole sunday passed with my good shoes on (should have taken the other ones I supposed), and I don’t want to get started on how my personal life is lately.

But I haven’t been taking naps all day, although I’d like to. First of all, since yesterday I got XCB and libX11/XCB working on Gentoo/FreeBSD. The patch is still to be polished before being merged, I’ll do so later on tonight or tomorrow.

I’ve also been working on xine’s FLAC support so to fix both the OggFlac problem and the refusal of FLAC files starting with an IDE3 block rather than the Streaminfo one (that are anyway invalid FLAC files, but that’s beside the point).

I have to say, I never thought a container format would be that messed up as Ogg… every stream type has its own headers, in the case of FLAC, it uses FLAC’s own metadata blocks. And FLAC is a masochist format for audio… It’s a lossless codec, so the size of the encoded files is usually not that low, but the metadata blocks tries to save a few bytes by not aligning the data structures to bytes.. I gone crazy trying to read a value of size 20 bits using bitfields or other similar approach, there’s no way: FLAC’s metadata blocks are not mappable against a machine-independent C structure, which makes parsing quite hard.

Anyway, I’ll try to complete these works as soon as I can, also because I need to return to my paid job as soon as I feel better.. and because when I cannot do something because of the feverish mood I’m in I start to be frustrated and depress myself… so if I’m not able to get something working soon, I’ll start screaming by rage :P

On the bright side, John Palmieri announce D-Bus 1.0 release that hopefully will work out of the box on FreeBSD :)

XCB on *BSD: many pthread-related problems

I’m copying here a mail I just sent to the XCB mailing list with a few things I’ve just got by trying to find the cause of the xserver not starting on Prakesh:

So, I’ve been trying to find why X didn’t start after having it built fine
with xcb enabled… and I’ve seen there’s a little big mess with it on
FreeBSD (and DragonFly and OpenBSD at least, not sure on NetBSD, but it’s
likely to be the same).

Background: on *BSD systems there’s more than one library providing pthread
functions, the result is that the -pthread flag (as well as -lpthread) does
not link the threading libraries when producing a shared object, but only
when the final executable is generated, so that you don’t end up with two
different threading implementations in a single executable (that will result
in a crash); when linking a non-threaded program to a threaded library, you
need to pass -pthread to it to let it build correctly.

libX11 wasn’t threaded previously, but libxcb and libxcb-xlib are now. The
result is that programs linking to libX11 will now need to use -pthread to
finalise the link, or they won’t work. The patches I’ve sent today will fix
this problem for the packages using pkg-config to locate libX11 (mostly
modular-x packages right now, I suppose), but for everything else, a lot of
patching will be needed for xcb to be usable on FreeBSD (unless of course
Ports will try again to cut the edge and tinker with packages without fixing
the issue properly).

I’ll try to provide some quick ways to work around the issue, fixing the
packages I can find broken on my system, but I’d sure welcome some help :) Or
even just an acknowldgement that I’m not doing this pointlessy.

This means, more things to fix… oh yes, most of KDE is fine already because Qt is threaded, so it “works around the issue”, but not everything, at least KControl and KDM needs to be slightly reworked.

The situation with XCB

So, as soon as XCB was in Portage I did give it a try… unfortunately Kaffeine refused to work as intended so I had to get away from it because I really need it working during the day (I play movies or films almost all day long when I’m trying to concentrate on a problem).

Now I was thinking of getting more seriously at it to find the issue, but I don’t want to break Enteprise… so I wanted to get it working on Prakesh… unfortunately xorg-server failed because libxcb does not state correctly its dependency over pthread.

I’ve now fixed the issue, or at least I hope so, but one question is still on my mind. Eric Anholt is an active Xorg developer, and a FreeBSD developer, why doesn’t he maintain the sources in sync? Why he’s more apt to workaround the stuff in Ports rather than fixing it for good in the sources? When modular X was in its first betas, I’ve been trying to report the issues with it on FreeBSD, and I was just dismissed with “Gentoo/FreeBSD is not supported, go play with your hacks somewhere else” kind of approach… after a few months I waited there, Anholt stumbled across the same errors and fixed them almost in the same way.

I wonder why people still consider us a problem rather than an help, especially considering that in the past year and a half we helped making more than a few projects FreeBSD-aware… PulseAudio for instance is now fully operational on FreeBSD from the repository as I’ve been working on porting it; D-Bus is now having more momentum on FreeBSD directly upstream, thanks to Timothy, Steev, Cardoe and J5; Binutils now supports FreeBSD’s ld-elf and static SPARC64 and AMD64 binaries; we even fixed a few bugs in FreeBSD sources themselves (in particular 6.2_beta3 release has two kernel patches prepared by Alex and Javier)… and we’re still considered harmful?