Bundling libraries for trouble

You might remember that I’ve been very opinionated against bundling libraries and, to a point, static linking of libraries for Gentoo. My reasons have been mostly geared toward security but there has been a few more instances I wrote about of problems with bundled libraries and stability, for instance the moment when you get symbol collisions between a bundled library and a different version of said library used by one of the dependencies, like that one time in xine.

But there are other reasons why bundling is bad in most cases, especially distributions, and it’s much worse than just statically linking everything. Unfortunately, while all the major distribution have, as far as I know, a policy against bundled (or even statically linked) libraries, there are very few people speaking against them outside your average distribution speaker.

One such a rare gem comes out of Steve McIntyre a few weeks ago, and actually makes two different topics I wrote about meet in a quite interesting way. Steve worked on finding which software packages make use of CPU-specific assembly code for performance-critical code, which would have to be ported for the new 64-bit ARM architecture (Aarch64). And this has mostly reminded me of x32.

In many ways, there are so many problems in common between Aarch64 and x32, and they mostly gear toward the fact that in both cases you have an architecture (or ABI) that is very similar to a known, well-understood architecture but is not identical. The biggest difference, a part from the implementations themselves, is in the way the two have been conceived: as I said before, Intel’s public documentation for the ABI’s inception noted explicitly the way that it was designed for closed systems, rather than open ones (the definition of open or closed system has nothing to do with open- or closed-source software, and has to be found more into the expectancies on what the users will be able to add to the system). The recent stretching of x32 on the open system environments is, in my opinion, not really a positive thing, but if that’s what people want …

I think Steve’s reports is worth a read, both for those who are interested to see what it takes to introduce a new architecture (or ABI). In particular, for those who maintained before that my complaining of x32 breaking assembly code all over the place was a moot point — people with a clue on how GCC works know that sometimes you cannot get away with its optimizations, and you actually need to handwrite code; at the same time, as Steve noted, sometimes the handwritten code is so bad that you should drop it and move back to plain compiled C.

There is also a visible amount of software where the handwritten assembly gets imported due to bundling and direct inclusion… this tends to be relatively common because handwritten assembly is usually tied to performance-critical code… which for many is the same code you bundle because a dynamic link is “not fast enough” — I disagree.

So anyway, give a read to Steve’s report, and then compare with some of the points made in my series of x32-related articles and tell me if I was completely wrong.

Same-ABI and any-ABI dependencies

Two really-desired features of Portage, that are important for, respectively, desktop and embedded use cases, are multilib and cross-compilation. Both of these, to be properly implemented, require Portage to discern between same-ABI and any-ABI dependencies.

This concept has been called in the past Linked-in and Executed dependencies, but I don’t like that name at all as it makes sense only for those who actually know what the two concept expects. Actually, I don’t like this name either because it confuses the term ABI as the calling convention of an architecture, and the term ABI as the compiled interface of a software library. If anybody can make sure we find a simple term for the calling convention type of ABI, it would be quite nice in my opionion.

Another good reason to get rid of the terms Linked-in and Executed dependencies is that abstracting well enough the concept, one can easily see the same code and mechanisms to be used to describe the dependencies of extension modules like Python’s and Ruby’s, that depend on the version of the interpreter they are built and ran against.

With good enough support, this would allow to handle dependencies for multiple Python and Ruby versions at once without needing strange and silly hacks like the ones present in the ruby eclasses. And would have solved the problem of PHP extension versions that caused to split them in dev-php4 and dev-php5.

Why is this distinction needed for both multilib and crosscompile? Well, let’s start with the multilib case, and see how it works.

Let’s say you want to install mozilla-firefox-bin, which is 32-bit. It will require a series of libraries built 32-bit like GTK+ and similar, but it will also require the launcher script. That launcher script does not need to be built with 32-bit ABI, as it does not interface directly with Firefox. The libraries will be a same-ABI dependencies, while the script will be an any-ABI dependency.

Even better, the script, being a script, could be declared to not have an absolute ABI, something alike RPM’s “noarch”. This would make it much easier to handle dependencies on data packages (icons, documentation, scripts).

But any-ABI dependencies can easily apply to non-scripts, that is binary packages. Let’s take for instance CUPS and the software that Canon makes available for Pixma printers. The software is only available for 32-bit systems, as it relies on some binary files that Canon does not give the sources of. On the other hand, it does not interface with CUPS through library calls, it’s instead a filter that gets executed to convert the PostScript data coming from CUPS to a format that the printer will understand. The Pixma software would then have an any-ABI dependency on CUPS, and a same-ABI dependency on the libraries it links to, like GTK+.

On cross-compiled environments, this merged with the ROOT variable support would make it much easier to build a system’s root with the minimal amount of software needed, and would solve the problem related to Perl’s cross-compiling.

A few packages need autoconf to be present in the system, but none of them need anything more than being able to execut them. When cross-compiling, there is no need to have a copy of autoconf built for the ABI of the system you’re building, you just need a copy of it that you can execute. This makes it an any-ABI dependency. Not asking a CHOST-built autoconf, Portage won’t be asking for a CHOST-built Perl either, and that would save you the headache, because Perl cannot be cross-compiled.

This of course will require a lot of changes, especially EAPI related, and finding a new, decently good syntax, for ABI-specific dependencies. In particular, there are a few issues that needs to be addressed before one can even think of implementing this:

  • there has to be a way for profiles to tell Portage the ABIs a system can handle; an amd64 multilib profile will advertise support for x86_64 and x86 ABIs for instance, an x86 Gentoo/FreeBSD profile would advertise support for x86/FreeBSD and x86/Linux ABIs (through the Linux compatibility layer);
  • there has to be a way to let packages inject further ABI specifications, like I said above, Ruby and Python are two perfect examples; a system that has Ruby 1.8 and Ruby 1.9 installed, will advertise support for both ABIs, while a system with just Python 2.5 will advertise that single ABI;
  • cross-compiling still would require some work, maybe a replacement for crossdev could use a prefix like cross- or something like that, it would need someway to express cross-compiling tools and libraries versus cross-compiled tools and libraries;
  • by default, the packages should be available on any native ABI, maybe a meta-ABI “native” would allow to say “whichever ABI the profile accept”; it could somehow cross-over with the meaning of KEYWORDS, especially for cross-compiled environments;
  • different ABIs will have different paths for installation: on amd64 multilib systems, the 32-bit libraries are currently installed by the 64-bit sides on /lib32 directories, while the binaries are mixed in /bin; on a Gentoo/FreeBSD system, you most likely want to install everything prefixed in a /linux directory, while for cross-compiling support, you’d install in the SYSROOT (/usr/$CHOST prefix); as you can guess, here it would be very nice to import at least part of the extensions the Prefix project have to allow handling rbitrary prefixes; it might not need further enhancements to support Solaris and the rest (but I could write something about that), but at least it would work to have support for the arbitrary prefix in tree.

There are probably a tons other problems, and I have no clear idea on how this should be implemented as it is, but at least this could be a starting point for some discussion in this sense, and maybe Zack or Marius might catch on this and start prototyping some interaction.

Crazy idea… an alternative to CPU features useflags

So, I’ve been having tonight the craziest idea possible :P Deprecating mmx, 3dnow, sse, and similar useflags. Let’s try to get a reason for that tho :)

So, mmx, 3dnow and sse useflags are used to enable special CPU-specific features that people might not have on their CPU. It’s not a good idea to enable them everywhere, but it’s often a bad idea do not enable them at all if you support them.

Unfortunately, it’s not always clear when to enable them and when not, and it’s difficult to handle them all considering that on amd64 they are usually all enabled… but 3dnow is not present on nocona arch (Intel EM64T arch).

I was tonight thinking of something like this:

has_mmx() {
  echo | $(tc-getCC) ${CFLAGS} -E -dM - | grep -q "#define __MMX__ 1"
}

with that function, you can check if the user asked to support a processor with MMX enabled, by using the user-supplied CFLAGS.

When you use -march you tell GCC to enable different extensions to the instruction set, so if you ask for an athlon-tbird arch, you’ll get MMX and 3DNow!, while if you ask for nocona you won’t get 3DNow! at all.

To disable one feature even when we know the arch we’re building for does have that extension, or to enable one even if we have a lesser arch (does not make much sense, but you never know), you can use -mno-mmx or -mmmx flags, that in 99% of the cases are misused by ricers.

This works for all the cpu-extensions flags we have, even altivec, and is safe to use with every GCC version. So why not? This will also reduce the arch tests, you can even use this function

is_x86() {
  echo | $(tc-getCC) ${CFLAGS} -E -dM - | grep -q "#define __i386__ 1"
}

to check if you’re building for x86 systems, no need either to think of ${ARCH} and similar…

I think I’ll be proposing this officially tomorrow, it might help :)