Tag: Portability

Yes we still needs autotools

One of the most common refrains that I hear lately, particularly when people discover Autotools Mythbuster is that we don’t need autotools anymore.

The argument goes as such: since Autotools were designed for portability on ancient systems that nobody really uses anymore, and that most of the modern operating systems have a common interface, whether that is POSIX or C99, the reasons to keep Autotools around are minimal.

This could be true… if your software does nothing that is ever platform specific. Which indeed is possible, but quite rare. Indeed, unpaper has a fairly limited amount of code in its configure.ac, as the lowest level code it has, it’s to read and write files. Indeed, I could have easily used anything else for the build system.

But on the other hand, if you’re doing anything more specific, which usually includes network I/O, you end up with a bit more of a script. Furthermore, if you don’t want to pull a systemd and decide that the latest Linux version is all you want to support, you end up having to figure out alternatives, or at least conditionals to what you can and cannot use. You may not want to do like VLC which supports anything between OS/2 and the latest Apple TV, but there is space between those extremes.

If you’re a library, this is even more important. Because while it might be that you’re not interested in any peculiar systems, it might very well be that one of your consumers is. Going back to the VLC example, I have spent quite a bit of time in the past weekends of this year helping the VLC project by fixing (or helping to fix) the build system of new libraries that are made a dependency of VLC for Android.

So while we have indeed overcome the difficulties of porting across many different UNIX flavours, we still have portability concerns. I would guess that it is true that we should reconsider what Autoconf tests for by default, and in particular there are some tests that are not completely compatible for modern systems (for instance the endianness tests were an obvious failure when MacIntel arrived, as then it would be building the code for both big endian (PPC) and little endian (Intel) — on the other hand, even these concerns are not important anymore, as universal binaries are already out of style.

So yes, I do think we still need portability, and I still think that not requiring a tool that depends on XML RPC libraries is a good side of autotools…

Multi-architecture, theory versus practise

You probably remember the whole thing about FatELF and my assertion that FatELF does nothing to solve what the users supporting it want to see solved: multiple architecture support by vendors. Since I don’t want to be taken for one of those people who throw an assertion and pretend everybody falls in line with it, I’d like to explain somewhat further what the problem is in my opinion.

As I said before, even if FatELF could simplify deployment (at the expense of increasing exponentially the complexity of any other part of the operating system that deals with executables and libraries), it does nothing to solve a much more important problem, that has to be solved before you can even think of achieve multi-architecture support from vendors: development.

Now, in theory it’s pretty easy to write multi-architecture code: you make no use of any machine-dependent feature, no inline assembly, no function call outside the scope of a standard. But is it possible for sophisticated code to keep this way? It certainly often is not for open source software, even when it already supports multiple architecture and multiple software platforms as well. You can find that even OpenOffice require a not-so-trivial porting to support Linux/HPPA and that’s a piece of software that, while deriving from a proprietary suite (and having been handled by Sun which is quite well known for messy build systems), has been heavily hacked at by a large community of developers, and includes already stable support for 64-bit architectures.

Now try to be a bit imaginative, and find yourself working at a piece of proprietary code: you’ve already allocated money to support Linux, which is for many points of view, a fringe operating system. Sure it starts to increase in popularity, but then again a lot of those using it won’t run proprietary application nonetheless… or wouldn’t pay for them. (And let’s not even start with the argument but Chrome OS will bring a lot more users to Linux since that’s already been shown as a moot point). Most likely at this point you are looking at supporting a relatively small subset of Linux users; it’s not just a matter of difference between distributions, it’s just a way to cut down testing time; if it works on unsupported distributions is fine, but you won’t go out of your way for them; the common “enterprisey” distributions are fine for that.

Now, at the end of the nineties or at the beginning of the current decade, you wouldn’t have to think much in term of architectures either: using Linux on anything but x86 mostly required lots of effort (and lead to instability). In all cases you had to “eradicate” the official operating system of the platform, which meant Windows for x86, Solaris for SPARC and Mac OS for PPC; but while the former was quite obvious, the other still required more work since they were developed by the developer of the hardware in the first place.

Nowadays it is true that things changed, but how did they change exactly? The major change is definitely the introduction of the AMD64 (or x86-64 if you prefer) architecture: an extension to the “good” old x86 that supports 64-bit addresses. This alone created quite a few problems: from one side, since it allows for compatibility with the old x86 software, proprietary, commercial software didn’t flock to support it so fast: after all their software could still be used, even though it required further support from the distributions (multilib support that is), on the other side, multilib was before something that only a few niche architectures like mips looked out for, so support for it wasn’t as ready for most distributions.

And, to put the cherry on top, users started insisting for some software to be available natively for x86-64 systems, so that it would be more compatible, or at least shinier in their eyes; Java, Flash Player, and stuff like that had to be ported over. But here we’re reaching the point where theory (or, if you’re definitely cynical – like me – fantasy) clashes against practise: making Flash work on AMD64 systems didn’t just involve calling another compiler, as many people think, partly because the technologies weren’t all available to Adobe to rebuild, and partly because the code made assumption about the architecture it was working on.

Let be honest: it’s hypocrite to say that Free Software developers don’t make such assumption; it’s more like porters and distributions fixed their code long time ago; proprietary software does not have this kind of peer review, and they are, generally, not interested on it. It takes time, it takes effort and thus it takes money. And that money does not generally come out of architectures like Alpha, or MIPS. And I’m not calling out for the two of them without reason here: they are the two architectures that have actually allowed for some porting to be done for AMD64 before its time. The former was probably the previously most available 64-bit system Linux worked decently on (SPARC64 is a long story), and had code requirements very similar to x86-64 in term of both pointer size and PIC libraries. The latter had the first implementations of multilib around.

But again, handling endianness correctly (and did you know that MIPS, ARM and PowerPC exist in multiple endian variations?), making sure that the pointers are not assumed to be of any particular size, and never use ASM-only routines is simply not enough to ensure your software will work on any particular architecture. There are many problems, some of which are solvable by changing engineering procedures, and some others which are simply not solvable without spending extra time debugging that architecture.

For instance, if you’ve got an hand-optimised x86-only assembly routine, and a replacement C code for the other architectures, that code is unlikely to get tested as much as the x86 code, if your development focuses simply on x86. And I’m not kidding you when I say that it’s not such a rare thing to happen, also with Free Software projects. Bugs in that piece of code will be tricky to identify unless you add to your development process the test and the support for that particular architecture; which, trust me, is not simple.

Similarly, you can think of the strict aliasing problem: GCC 4.4 introduced further optimisations that can make use of strict aliasing assumption on x86 as well; before, this feature was mostly used by other architectures. Interestingly enough, the amount of strict-aliasing bug is definitely not trivial, and will cause some spurious bugs at runtime. Again, this is something that you can only fix by properly testing, and debugging, on different architectures. Even though some failures now happen on x86 too, this does not mean that the same problems happen, no more no less, on anything else. And you need to add your compiler’s bug to the mix, which is also not so simple.

And all of this is only covering the problems with the code itself, and comes nowhere near the problems of cross-compilation, it does not talk about the problems and bugs that can be in your dependencies’ code for the other architectures, or the availability of stable-interface distributions for those architectures (how many architectures is RHEL available for?).

After all this, do you still think that the only problem keeping vendors from supporting multiple architectures is the lack of a “Universal Binary” feature? Really? If so, I have some fresh air to sell you.

Dealing with missing functions

While I’m still expanding Autotools Mythbuster and tying up some loose ends before going on vacation I’ve been thinking of one particular problem that has hit me in feng and that is: how do you deal with missing functions in various operating systems?

Let me explain; one of the tricks I always stress with autoconf is to reduce the number of tests since that not only reduces the complexity of the configure.ac file for the developer, but also reduces the time wasted by the ./configure run.

To do this, I usually assume the systems I’m building on are standard; this is something that I derived from FFmpeg, that in general assumes that it’s building with a standard-compliant C99 compiler on a POSIX system; this works fine with all the modern Linux systems (especially since POSIX became GLIBC in the latest incarnation), and most other operating systems, once you exclude Microsoft and, partially, OS X (I don’t know about Snow Leopard yet, since I’m not subscribed as a developer). The problem is that often you have to support older operating systems, or systems that are not standard; or on the other hand, you might want to use extensions that are not available on all the operating systems, but are on most.

For these cases, usually you make use of autoconf checks, and then write your own replacement function; this unfortunately leads to nasty side-effect when they are added and at the same time they require duplication of a huge amount of code. Nowadays, on Linux, this can easily be noted in the strlcpy() and strlcat() functions, since they are born on OpenBSD, ported to most *BSD systems, but not (yet) available on the GNU C library: from my latest collision report, I count 66 and 47 copies of those exported (including by KDE libraries, libmagic and others).

To find alternative implementations, you can also look at the gnulib project by GNU itself, that allows to add the proper replacement functions for most of them to your project; I even wrote about that and I do sustain the project, but while it reduces the amount of duplicated source code, it doesn’t help to reduce the redundancy of compiled copies of the same functions over and over.

What we need is a single library where all the replacement functions can be compiled in, and then it can be linked against binaries and other libraries needing functions that are not available on the system. Thanks to options like --as-needed and autoconf auto-discovery of which library holds a given function (the AC_SEARCH_LIBS macro), this shouldn’t really be an excessive problem. The problem here is that you have a non-trivial amount of work for such a library: you need to be compatible with the current interfaces, you might want to have the functions optimised, you might have to deal with license issues.

Ideally, such a library should be released either under BSD license or LGPL, so that it could be used (but not embedded) on a the vastest possible range of software projects. Picking up some functions from gnulib, and others from the BSD C libraries might be a start. Unfortunately some of the functions in gnulib are only GPL-licensed, not LGPL, so we still cannot go with just reusing that code.

But seriously, this is something you got to consider on the long run: supporting older systems where functions are unavailable tend to get more difficult by the year, since more interfaces are added; and unless you get all your interfaces from convenience libraries such as glib, you end up either with a lot of replacement functions, or simply supporting a smaller subset of systems that you might, “if just” you had a way to make use of some extra functions on them.

Using Gnulib to improve software portability

This article was originally published on Linux.com.

Many, if not most, free and open source software projects are developed primarily on Linux-based systems using the GNU C Library (glibc). Projects that use glibc are likely to depend on functions that are not available on systems that use different C libraries, such as the different BSD flavors. When packages are built on systems that don’t use glibc they often fail, because the other C libraries are missing functions found in glibc. The GNU Portability Library can help developers with cross-platform programming needs.

In the past there were many different libraries, such as publib, that tried to provide alternatives to the functions that are missing in the main C library. Unfortunately, handling compatibility libraries proved to be difficult. The additional libraries would require additional tests when running configuration scripts prior to compilation, and add dependencies for non-glibc systems.

As the number of new functions provided by the glibc increased, the GNU project started looking at the requirements for portability of programs on operating systems based on different libraries, and eventually created the GNU Portability Library (Gnulib) project.

Normally, a library is code that is compiled as a shared or static file, and then linked into the final executable. Gnulib is a source code library, more similar to a student’s collection of notes than a usual compiled library. Gnulib can’t be compiled separately; the code in it is intended to be copied into the projects using it.

###Using Gnulib

Two requirements limit Gnulib use, one technical and the other legal. First, the software you use it with must also use GNU Autotools, as Gnulib provides tests for replacements of functions and headers written in the M4 language, ready for usage with GNU Autoconf.

Second, it has to be licensed under the GNU General Public License (GPL) or the GNU Lesser General Public License (LGPL), as the code inside Gnulib is mostly (but not entirely) released under the GPL itself. Some of it is also released under the LGPL, and some of it is available as public domain software.

If you’re working on software released under other licenses, such as the BSD license or Apple Public Source License (APSL), it’s better to avoid the use of functions that are not available in a library licensed with more open terms. For example, you could take the libraries present in a BSD-licensed C library to replace missing functions in the current library, whichever license the software is using. Alternative, you could find replacement functions in other BSD-licensed software or create a “cleanroom” implementation without copying code from GPLed software, leaving the external interface the same but using different code.

It’s usually easy to re-implement functions or just copy missing functions from another project when they are not available through another C library, especially when they are simple functions that consist of less than 10 lines of code. Unfortunately, many projects depend firmly on GNU extensions, and won’t build with replacement functions, or the code is already so complex that adding cases to maintain manually is an extra encumbrance for developers.

What Gnulib provides is not only the source code of the missing functions, but an entire framework to allow a project to depend on GNU extensions, while retaining portability with non-GNU based systems.

The core of the framework is the gnulib-tool script, which is the automated tool for extracting and manipulating source code from Gnulib. Using gnulib-tool, you can see the list of available modules (gnulib-tool –list) or test them (one by one, or all together, using the –test or –megatest options), but more importantly you can automatically maintain the replacement functions for a source tree.

A practical example should help explain the concept. Let’s say that there’s a foofreak package that uses the strndup() function (not available on BSD systems for instance) and the timegm() functions (not available on Solaris). To make the source portable, a developer can run gnulib-tool –import strndup timegm from the same directory of the source code, and the script will copy (in the default directories) the source code and the M4 autoconf tests for strndup(), timegm(), and their dependencies – for example, strndup() depends on strnlen().

After running, gnulib-tool tells you to make a few changes to your code to allow the replacement to be checked and used when needed. It requires the Makefile in lib/ to be generated from the configure script, so it has to be added to AC_OUTPUT or AC_CONFIG_FILES. At the same time the lib/ subdirectory has to be added to the SUBDIRS variable in the Makefile.am. The M4 tests are not shipped with other packages, so they must be copied in the m4/ directory, and that has to be added as an include directory for aclocal. Finally, two macros have to be called within the configure.ac to initialize the checks (gl_EARLY and gl_INIT).

You can specify the name of the subdirectories and the prefix of the macros by running gnulib-tool with the parameters –source-base, –m4-base, and –macro-prefix, respectively. It’s also important to note that the replacement functions are built in an auxiliary library called libgnu (by default, but the name can be overridden by using the –lib parameter), so the part of the software using those functions has to be linked against this too.

If later on your project also wants to use the iconv() function, gnulib-tool can detect the currently imported modules and add the required iconv module without rewriting everything from scratch. This makes it simple to add new modules when you use new functions.

The different replacement functions are called “modules” by gnulib-tool, and they consist of some source code, some header files, and an M4 macro test. As some functions depend on the behavior of other functions, the modules depend one on the other, so adding a single module can generate quite a few additional checks and replacements, which make sure that the behavior is safe.

As some modules are licensed under the GPL, while other are licensed under the LGPL, a package licensed under the latter might want to make sure that no GPL modules are pulled in, as that would break the license. To avoid adding GPLed modules, you can use gnulib-tool’s –lgpl option, which forces the use of LGPL modules.

You can also use alternative code to provide a replacement function instead of using the Gnulib modules, and to avoid problems with dependencies. Gnulib-tool has an –avoid option that prevents specified Gnulib modules from being pulled in.

Following the previous example, if foofreak already contains a strnlen() function, used when the system library doesn’t provide one, it would be possible to use that, instead of importing the strnlen() module from Gnulib, by issuing the command gnulib-tool –import strndup timegm –avoid strnlen. With this syntax the strnlen module will be ignored and the function already present in foofreak will be used. While this option is provided, it’s usually not advisable to use it if you don’t really know what you’re doing. A better alternative would be dropping strnlen() from the code where it was used, and using the replacement provided by Gnulib instead.

###Summary

Gnulib is an interesting tool for people working with GPL- or LGPL-licensed software that needs to be portable without dropping the use of GNU extensions, but it has some drawbacks. The major drawback is the license restrictions, which requires non-(L)GPL-licensed software to look elsewhere for replacements. It also requires the use of the GNU toolchain with Autotools, as it would be quite difficult to mimic the same tests with something like SCons or Jam.

Finally, the source code sharing between projects breaks one of the basic advantages in the use of libraries: the reuse of the same machine code. When the same function, required by 10 or 20 programs, has to be built inside the executable itself as the system does not provide it, there will be 10 or 20 copies of the same code in memory and on disk, and they may behave in different ways, leading to problems if they are linked inside a library used by third-party software.

Gnulib is worth a try, but you should not use it in critical software or software that might have a limited audience. In those situations, avoid the use of extension functions when possible, and add replacement functions only when they’re actually needed. There’s no point in having a replacement function for something that is works on 90% of modern systems and breaks only on obsolete or obscure operating systems or C libraries, especially if the software is written to be run on modern machines.

Best practices for portable patches

This article was originally published on Linux.com.

One of the things I usually take care of as a Gentoo packages maintainer is sending patches to upstream developers. If a patch is applied upstream, we can remove it from future versions of a package so we have less work to do to maintain the package. Unfortunately, it seems that other distributions and packagers don’t always do the same. This is true not only for Linux distributions such as Debian, Fedora Core, and SUSE, but also for maintainers of packages in places like FreeBSD’s Ports, DarwinPorts or Fink. Here are some tips for developers on making things easier for yourself and everyone who has to touch your code. When upstream developers are unaware of the problems their software has on platforms they can’t test (perhaps because they use another distribution, another environment, another operating system, or another hardware platform) they can’t fix them and are likely to introduce more problems in further versions, if they assume that things are good as they are. Letting upstream developers know of the problems, filing bugs, and in general reporting problems is one of the best ways to help an open source project, and it’s something even users with no technical skills can do.

When you have the technical ability to fix a bug, though, you should try to provide to the upstream developers a patch. However, not all patches can be applied unconditionally upstream, and that makes it harder to fix a problem in the short term. Some of the errors people make while preparing a patch and sending it upstream can be fixed in a reasonably simple way, but I still see bad patches in many packaging systems (including Gentoo on occasion).

The first thing to take into account when writing a patch is that the environment in which you’re working can be different from other environments. By “environment” I mean all the factors that can affect the behaviour of a program, such as the operating system and its version; the distribution, if an operating system has more than one; the drivers used when there is more than one kind; the version of the libraries or of the tools; and so on. One of the most common assumptions developers make while creating patches is that the environment they’re using is the “right” one and everything else should follow it; however, even when the environment used is the “right one,” a patch should always be general enough to be applicable also to “broken” environments.

Using GNU’s Autotools is a good way to allow a C/C++ program to adapt itself to multiple environments. Although they have many problems, and are hated by most of their users, Autotools are currently the best way to handle a multi-platform, adaptable building system. They have facilities to check for headers, libraries, functions, and a lot more. Unfortunately Autotools are quite difficult to learn, and I’m not going into details on how to use them or how to write macros in Autotools’ M4 language, but here’s a brief overview.

In an Autotools-based project, you usually write a configure.ac script in M4, which defines the checks needed on the host machine (the machine which where the build is going to be done); this script is then translated into a shell script by autoconf. You also write makefile.am files which are used by automake to create the templates for makefiles that are created by the configure script after the checks. Usually the configure script can define conditionals for makefiles and create a config.h file where you can find some C preprocessor’s macros useful to create conditional code for a C or C++ source file (using #ifdefs).

For instance, you can’t always suppose that a given header is there, even if it’s part of a standard defined for Unix systems. Libraries change, including system libraries, and something you’re writing now can change in the future. Try to make sure that all the system and non-conventional headers you use are present before using them. You can usually do that with an AC_CHECK_HEADERS() call in configure.ac and using the config.h generated on the host machine to check whether they are present. Sometimes you get warnings during configure execution, stating that a given header “can be preprocessed but fails to compile”: while this is a warning at this time, it is going to be an error in the future, so try to fix it as soon as you can. The fix usually involves adding a couple of needed headers in the prerequisite argument. With this check, for example, you can usually avoid the infamous error of malloc.h on FreeBSD systems (however, malloc.h is actually deprecated; you can use stdlib.h in its place without problems).

System headers aren’t portable, so you should always avoid them if you’re not going to use a kernel service you can’t get in other ways. Try instead to get the information or the services you need; it’s usually more portable also between different version of the same operating system. It’s not unlikely that different Linux (kernel) versions have incompatible headers that can make some software fail to work.

Libraries, too, can be a problem. Glibc, used by every Linux distribution (apart from the ones for embedded usage, which normally use uclibc or dietlibc), provides not only the base functions a normal Clibrary should provide (for example, to interact with kernel) but also provides a complete iconv() implementation, basic gettext implementation, and a getopt_long() function used to get long parameters when a program uses getopt to parse the arguments given to it by the user. However, other system libraries, like FreeBSD’s, don’t provide all that; iconv() is provided on FreeBSD systems by GNU libiconv, while gettext is entirely provided by GNU gettext; for getopt_long(), you need another library just on 4.x series of FreeBSD, on DragonFly BSD, and probably on other non-Linux systems. Autotools provide the AC_CHECK_LIB() macro to check for presence of a given function in a library, allowing the packages to check if they must link to libiconv, libdl, libintl, and so on.

Size of basic types, like integer sizes, can vary among hardware platforms, as can pointer size. This problem is getting bigger lately, as x86_64 systems can now be considered the first 64-bit hardware platform for mainstream desktop users. Having to care about 64-bit cleanness can be a bit tricky for software developed assuming the usual x86 platform, but the fixes are usually easy enough to make in a couple of minutes. One of the most important ways to ensure that the variables’ sizes are right is to use the “standardized integers” which can be found on sys/types.h or stdint.h headers. These are renamed integer types, with a name in the form (u)intSIZE_t — so you have int8_t for 8-bit signed integers and uint64_t for 64-bit unsigned integers. Pointers can’t use those fixed-length types, as they depend on the architecture; they usually have the same length as long integers, but if you can, try ptrdiff_t before using long to store them. Other types, such as size_t and off_t, have their dimension fixed in all the platforms, so they are safe to be used as they are. You should never mix fixed-length and “named” types like unsigned int and long, and you should be sure that you’re using the right %-code when passing integers to printf or scanf. You can usually find macros defined that allow you to get the right code with the size, so PRId64 would be the equivalent of %d for 64-bit integers, while PRIu32 would be the equivalent of %u for 32-bit integers.

Assembler code has also been a problem lately. Before the introduction of x86_64 processors every hardware platform had its own assembler code, so there weren’t too many problems porting it. It was just a matter of having a non-assembler version of the code, which was maybe slower but portable. With the new architecture, instead, you can have assembler code shared between x86 and x86_64 systems, and this is especially true for multimedia applications that make use of extended instructions like MMX, SSE, or 3DNow!. Although the syntax of assembler is usually the same, there are a few things to think about, such as the size of the registers and the operations run on them. Using “base” operations on 32-bit registers on an x86_64 processor will fail, so you usually have to select the register to use as operands with conditionals.

The discussion about hardware compatibility is of course longer than this and deserves a book of its own, as there are a lot of other tricky conditions to take care of. Those working on non-x86 architectures already know how to fix eventual problems and are likely to provide patches in cases where the software is broken.

Returning to software compatibility issues: you should take into account the possibility of crosscompiling the software. Letting Autotools-based projects build in crosscompile is usually an easy task, if you have all the dependencies already crosscompiled (and, obviously, you have a working crosscompiler for the target architecture). Unfortunately, some configure scripts are broken by design and fail to work as they should in crosscompile. The most common error is to use the output of the uname command to select the architecture or platform to compile for; this will tell you about the current system, not the target platform. Instead of using that, you can rely on ${host} and ${target} variables, which contains a CHOST-like string defining the host system (the one you’re compiling on) and the target one (the one you’re compiling for). The CHOST-like strings are structured as tuples, either three components (arch-vendor-os) or four (arch-vendor-kernel-libc), although the latter is used only when the operating system has no one “native” libc. The arch part of the string defines the hardware platform used: i386, i686, x86_64, ppc, ppc64, sparc, and so on; the vendor part used to refer to the hardware vendor (for example, ibm for their mainframes) but lately is being used to refer to the software vendor, so you can find there redhat, debian, mandrake, gentoo (note that Gentoo uses it only on uclibc and Gentoo/*BSD systems) and others, but usually it’s just “pc” for x86 and “unknown” for other architectures. The os part is the most tricky one, as it defines the operating system used. Usually it’s “linux-gnu” for GNU/Linux systems, but it can be “linux-uclibc” or just “linux” for other Linux-based systems, or can be “freebsdX.Y” to refer to a given version of FreeBSD (just “freebsd” is not a valid value). Generally, this value is what you use to check which OS-specific code to enable.

I hope that this introductory article will inspire patch authors to ensure that the changes they make are not going to break other systems. By using the right combination of Autotools and conditional compilation to check for the right environment where the patch is needed, developers can allow upstream maintainers to fix their software, saving maintainers from having to reinvent the wheel every time something in the “common” environment changes.

The importance of portability

As anybody who read my blog before could have seen, one of the things I’m more concerned with is portability. Not only between different OSes (Linux and FreeBSD) but also between different architecture (AMD64).
The portability is a simple process actually, you just need to care a bit more about the size of the datatypes and of their internal structure, not relying on the byte disposition (little endian or big endian), also because, as many developers pointed out for me in #gentoo-dev, we can’t be sure of the endianness neither on architectures which usually uses a given endianness, like PPC which can be different from implementation to implementation.

What I’m trying to say here is that portability is going from now on to be always more and more important: not only Linux and *BSD runs on a wide variety of hardware platforms, but now also Apple with their MacOS is going to change platform. What that means for us? Well most of the multi-platform projects will need to restructure the code if they was assuming that OSX == PPC == Big Endian. But how they can do that? There are out there some development kit but they aren’t ensured to be the final product; actually we can’t ensure what is going on in the market two months for now, how can we predict what will be the final hardware platform Apple will choose? We actually know that it will be an Intel chip and won’t be compatible with PPC, but which will be the internal structures is something we don’t know yet.

So what’s the deal then? What can software producers do now to avoid the hell of portability when it will come to the new platform?
The first problem is with size of integers and with printf arguments: fortunately there are the standard integer types intX_t uintX_t, which can usually be trusted without problems, and then there are the macros like PRId64 to select the right type for the output, just a bit more attention to this can avoid having to mess around with #ifdef conditionals to have the things working well.

The other problem is with endianness of types, and more specifically with data structures written in binary form on disk (in files). The solution to this is to specify their structure and endianness in a complete form, making sure that they are loaded with special functions used to preserve their endianness; also if the most widespread hardware platform currently available (Intel IA-32 and derived) and the one which is imposing itself in 64-bit desktop market (AMD x86_64 and derived) are littleendian, probably the most natural way to define data structures on disk is big endian. I know this can seem stupid, but the use of bigendian data structures on disk means that they can be seamlessy streamed over the network: quite all the binary protocols currently used uses big endian as endianness, that’s why it’s sometime called Network Endianness. It’s also the more logical natural form for numbers, for humans.

Not relying on data structure internal format can be sometime a bit slower than accessing them directly, as you needs to use shifts and bitwise and operations to get the part of the structure you need, but with today’s hardware such an overhead is probably not significative for the complessive performance of the system.

Then just remember: developing something portable from scratch is going to be a lot less of a pain than need to port something to a new architecture when the mainstream OS you are using is changing its platform, and this is true not only for Linux, *BSD, OS X and Windows programs but also for all the programs which wants to run on “strange” or new operating systems, or for the ones who wants to run on every operating system of the world… that can seem to be something too difficult to do, but looking for the right services of the operating systems, it’s not impossible to work on something which is so much abstract to run everywhere, without need to use something like Java or MONO/.Net and various virtual machines.

Anyway I really hope that with the new 4.x series of GCC a lot of the old, fooledup, really really bad styled code is going to become deprecated, and I’m going to do my part as much as I can.