Mailbox

I’m back discussing the backtrace guide that I wrote some time ago. Fredrik asks by mail:

[…] after reading the bt-article I’m starting to think that the debug USE, more or less simply enables the developers own debugging stuff (like noisy extra output during run-time, assertions (which, afaik, are removed by -DNDEBUG which isn’t that uncommon in “release mode” cf. “debug mode”), etc).

So, my question is simply: if I want debugging symbols I should stay away from the debug USE and only add “-ggdb” to my CFLAGS/CXXFLAGS and put splitdebug in my FEATURES?

The answer is not as simple as I’d like it to be. Fredrik is definitely right into assessing that the debug USE flag, at least in most cases, is used to enable “debug mode” builds. Assertions, debug verbosity, extra safety checks, … all this kind of stuff, that is useful for developers during debug session, and totally useless (or even harmful) for production builds, can be tied to the debug USE flag. But, since we’re Gentoo we cannot do things this cleanly. Unfortunately.

While the QA team, as far as I know, agrees on the meaning of debug USE flag, not all developers seem to accept that, so you may well have multiple meanings of it right now:

enable the debug code, like I stated above: logs, assertions and so on;
enable debug symbols generation appending -g or -ggdb to the build flags;
disable optimisation by ignoring the CFLAGS variable.

Or most likely, a combination of the above. Note that the official meaning has been, for a long time, the following:

debug – Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see http://www.gentoo.org/proj/en/qa/backtraces.xml

The reason why the QA team would like for the USE flag to only handle the first case is that the other two make little to no sense in the context of Gentoo. Even though you ask the build system to emit debug symbols with -ggdb, without setting the proper features into portage, they’ll be stripped off before merge to live filesystem (and don’t even think of using RESTRICT=strip!); toying with CFLAGS is also a very bad idea because you’re second-guessing the user (and we have enough ways to handle per-ebuild CFLAGS to deal with that). Enabling the debug code is instead something that only an USE flag can do cleanly.

Unfortunately it’s not just the opinion of the QA team or of the maintainers that are on the table here: some upstream developers, like the Drizzle developers, seem not to understand that backtraces can be valid even when using optimisation (at least up to -O2), or that you can have useful information out of a backtrace even without enabling assertions and similar problems. Actually, to find out why something crashed in production, you want the backtrace to be produced without the debug code, otherwise you’re debugging two different programs!

Now, I sincerely think that the description for the debug USE flag is definitely too generic; but you cannot give more details about it without becoming specific to one package or one set of packages; fortunately me and Doug solved that some time ago – over two and a half years ago to be precise – by allowing per-package extended USE flags descriptions in metadata.xml files. And indeed there are quite a few packages that do document it. In one case (app-portage/eix) even admitting they are doing it wrong.

I could write a whole post about how I would expect app-portage packages to be a notch over the rest of the tree, given that are packages written specifically for Gentoo, and should know and comply with the policies we set out for packages. The fact that at least two fails to do that, is a bad sign.

It is important to write some notes about the -g flags here, before people get confused. You can neither always disable them nor always enable them in Gentoo itself. The latter is a common assumption I read people thinking on: “since Gentoo already strips the debug information, why not always emitting it and leave the user just to tinker with the stripping features?”, well the answer is actually easy to understand; emitting the debug information is not a trivial task, as the compiler needs to write more data to the assembler that needs to compile it into the DWARF code (and Arnaldo from the dwarves/perf projects has shown me how messed up that is, trust me, it is a lot). It might sound very easy when you look at very simple source files, but it becomes tremendously complex when you have thousand-lines source files or, worse, C++ source files with templates. Not only it’s a lot of work for the assembler (and linker) but it also increases the size of the temporary object files (the products of compile+assembly) and of the final shared objects and executable files, before stripping; if you build in RAM, this means you’ll need more memory to store the build directory; if you build on disk, it increases the amount of stuff that gets written to and read from the disk, slowing down the process quite a bit in some cases.

At the same time, we cannot use (or suggest the use of) -g0 to disable the whole debug information generation. While the idea looks nice, and I also thought about it a bit, I wouldn’t be surprised if some packages actually made use of basic debug information; of course in those cases you’d also have RESTRICT=strip (and there are a number of those packages in the tree already). So you might want to try it by yourself but.. you’re on your own if you shoot yourself in the foot with that.

One could ask here why the debug USE flag is so package-specific, and why one shouldn’t just use -DNDEBUG to disable all the asserts to begin with. The answer is that, unfortunately, a huge amount of software fails to work when assertions are disabled, because they consider their asserts as the main error handling feature, while asserts should really just be used to verify situations that will not happen. Unfortunately, once you disable those asserts, they have no graceful handling of errors, so the software might crash badly or – much worse – corrupt the data before crashing. So before deciding whether you can add the debug USE flag, you got to find out whether the package supports doing so.

Also, you might have to note that not all the assertions are disabled by defining NDEBUG; for instance, g_assert() from glib is disabled via another macro (which if I recall correctly should be G_ASSERT_DISABLE or something along those lines); but even that might not be a good idea to disable, unless you know what you’re doing).

I sincerely try my best to have both the upstream packaging and the ebuild abiding by this rule: --enable-debug in feng adds some further debug messages, and adds function that fully free the resources before exiting (useful to remove false positives from valgrind), while --disable-debug removes all of that and all the assertions.

At any rate, the bottom line is that, as usual, Gentoo lacks consistency in its design and decisions; we really should start applying this kind of consistency to new packages as they are added, but I fear most reviewers, including the Sunrise reviewers, will forget about these details. I’m not asking perfection here, but at least having a clue?

Jeremy asks via email:

I’m curious about the mention of gcc -O0 producing broken code in this article: do you have a reference to a particular bug for that? I think the article would benefit from being more specific about which versions of gcc produce broken code for -O0, otherwise readers will be scared off from using it for ever.

What he refers to is the Gentoo backtracing guide I wrote that gives some indications about using Portage features and ebuild support to get proper backtraces (stack traces), and how to avoid debug code instead.

If you really need a precise backtrace, you should use -O1 (using the optimisations of level 0 is not recommended as there are known issues with gcc building invalid code).

I’ll update the guide today if I can find the time, so you’ll probably find a more correct phrase up there, soonish.

The issue here is actually very complex, which is why I think a reply on the blog is worth the hassle. The first note here to do is GCC itself does not really miscompile with -O0 optimisations. The problems lie in the code of the software itself, which makes it much more difficult to deal with — when the problem is in the compiler you can isolate the problem and wait till it’s fixed to make sure you can use the feature; if the problem is in the code that gets compiled, then we cannot really be sure it all works, not unless you build and test everything and even then it might not be easy.

Anyway, -O0 does a little more than just disabling some optimisations, it changes a lot in the way code is handled, first of all by changing the preprocessor definitions:

flame@yamato ~ % diff -u <(echo | gcc -O0 -dM -E -) <(echo | gcc -O1 -dM -E -)
--- /proc/self/fd/11    2010-06-14 02:59:32.973351250 +0200
+++ /proc/self/fd/12    2010-06-14 02:59:32.974351316 +0200
@@ -31,6 +31,7 @@
 #define __UINTMAX_TYPE__ long unsigned int
 #define __linux 1
 #define __DEC32_EPSILON__ 1E-6DF
+#define __OPTIMIZE__ 1
 #define __unix 1
 #define __UINT32_MAX__ 4294967295U
 #define __LDBL_MAX_EXP__ 16384
@@ -89,7 +90,6 @@
 #define __UINT16_MAX__ 65535
 #define __DBL_HAS_DENORM__ 1
 #define __UINT8_TYPE__ unsigned char
-#define __NO_INLINE__ 1
 #define __FLT_MANT_DIG__ 24
 #define __VERSION__ "4.5.0"
 #define __UINT64_C(c) c ## UL

Beside showing the second big problem that I’ll talk about (disabling inline functions), it should remind you of my corner case testing from two years ago: disabling optimisation explicitly causes headers to change as they hide or show different interfaces, “proper” or “fast”. As you might guess already, this makes it slightly harder to actually track down a crash, or get an absolute stack trace, because it’s the (already pre-processed) code that changes itself; the crash might very well be only happening in the codepath taken when the optimised functions are used.

This gets combined with no inlining of functions: some of the system headers will replace definitions of some functions with inline wrappers when it is possible. These inlines often are faster, but they also take care of adding the warnings with _FORTIFY_SOURCE (the feature of recent GCC and GLIBC that adds warnings at build-time for probably-wrong behaviour of source code, ignoring return values, providing wrong parameters, or causing buffer overflows).

The two combined make using the system library (GLIBC) quite different between optimised and non-optimised builds; it’s almost like having two different basic C libraries, as you might guess, it’s important, to know where something’s crashing, to use the same C library to get the trace. In quite a few cases, rebuilding with -O0 can stop something from crashing, but it’s almost never caused by the compiler miscompiling something, it’ s usually due to the code only working with the “proper” interfaces, but failing when using “less proper” interfaces that are faster, but rely on the rules to be followed literally.

One example of this are the functions ntohs and ntohl: they are present as functions in the C library but they are also present as macros in the GLIBC includes, so that they can be expanded at build-time (especially when used on constants). This causes quite some change on the handling of temporary variables, if you play with pointers and type-punning (very long story now) so the behaviour of the ntohs or ntohl calls differs, and upstream is likely to have tested only one behaviour, rather than both.

Finally, there are possible build failures tied with the use of -O0, because, among other things, the Dead Code Elimination pass is hindered. Dead Code Elimination is a safe optimisation, so it’s executed even at -O0, its objective is to remove code that is never going to be executed, like an if (0) code. Using if (0) rather than adding #ifdef all over the code is a decent way to make sure that the syntax of the code is correct, even for those parts that are not enabled (as long as you have declarations of functions and the like). Unfortunately, without optimisations enabled, DCE will only work over explicit if (0) blocks, and not indirect ones. So for instance if you got code like this:

int bar() {
#if ENABLE_FOO
  int a = is_foo_enabled_runtime();
#else
  int a = 0;
#endif

  if ( a )
     process_foo_parameters();

  process_parameters();

  if ( a )
   foo();
  else
   not_foo();
}

DCE will only drop the if (a) branches when constant propagation is also applied, and will produce code with undefined references to process_foo_parameters() and foo() when -O0 is used for the build. This trick is often used in the wild to write code that is readable and validated for syntax even when features are not enabled, without using a huge quantity of #ifdef statements. And as I just shown, it can easily break when built with -O0. Among the others, FFmpeg uses this kind of code, so it cannot really be compiled at an absolute -O0.

Note: this mostly applies to GCC, as other compilers might do constant propagation even at -O0 given that it is a fairly stable and safe optimisation. Clang, though, behaves the same way.

While it can be argued that software shouldn’t rely on specific features of the C compilers but just on the C language specification, it is quite impossible for us to vouch that all the software out there will build and work properly when built with no optimisation at all, for the reasons I just explained. Gentoo of course is ready to accept bugs for those packages, but don’t expect most of us to go out of our way to fix it in the tree if you don’t provide a patch; alternatively, talk with upstream and get them to fix their code, or work it around somehow. If you find a package that fails to build or (worse) fail to work properly when built with -O0, please open a bug against the tracker after tracking down where the issue here. If you give us a patch we’ll likely merge it soon, otherwise it would still be feasible to track down the packages with problems.

Mailbox: about the debug USE flag

Mailbox: does GCC miscompile at -O0?