This Time Self-Hosted
dark mode light mode Search

Mailbox: about the debug USE flag

I’m back discussing the backtrace guide that I wrote some time ago. Fredrik asks by mail:

[…] after reading the bt-article I’m starting to think that the debug USE, more or less simply enables the developers own debugging stuff (like noisy extra output during run-time, assertions (which, afaik, are removed by -DNDEBUG which isn’t that uncommon in “release mode” cf. “debug mode”), etc).

So, my question is simply: if I want debugging symbols I should stay away from the debug USE and only add “-ggdb” to my CFLAGS/CXXFLAGS and put splitdebug in my FEATURES?

The answer is not as simple as I’d like it to be. Fredrik is definitely right into assessing that the debug USE flag, at least in most cases, is used to enable “debug mode” builds. Assertions, debug verbosity, extra safety checks, … all this kind of stuff, that is useful for developers during debug session, and totally useless (or even harmful) for production builds, can be tied to the debug USE flag. But, since we’re Gentoo we cannot do things this cleanly. Unfortunately.

While the QA team, as far as I know, agrees on the meaning of debug USE flag, not all developers seem to accept that, so you may well have multiple meanings of it right now:

  • enable the debug code, like I stated above: logs, assertions and so on;
  • enable debug symbols generation appending -g or -ggdb to the build flags;
  • disable optimisation by ignoring the CFLAGS variable.

Or most likely, a combination of the above. Note that the official meaning has been, for a long time, the following:

debug – Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see

The reason why the QA team would like for the USE flag to only handle the first case is that the other two make little to no sense in the context of Gentoo. Even though you ask the build system to emit debug symbols with -ggdb, without setting the proper features into portage, they’ll be stripped off before merge to live filesystem (and don’t even think of using RESTRICT=strip!); toying with CFLAGS is also a very bad idea because you’re second-guessing the user (and we have enough ways to handle per-ebuild CFLAGS to deal with that). Enabling the debug code is instead something that only an USE flag can do cleanly.

Unfortunately it’s not just the opinion of the QA team or of the maintainers that are on the table here: some upstream developers, like the Drizzle developers, seem not to understand that backtraces can be valid even when using optimisation (at least up to -O2), or that you can have useful information out of a backtrace even without enabling assertions and similar problems. Actually, to find out why something crashed in production, you want the backtrace to be produced without the debug code, otherwise you’re debugging two different programs!

Now, I sincerely think that the description for the debug USE flag is definitely too generic; but you cannot give more details about it without becoming specific to one package or one set of packages; fortunately me and Doug solved that some time ago – over two and a half years ago to be precise – by allowing per-package extended USE flags descriptions in metadata.xml files. And indeed there are quite a few packages that do document it. In one case (app-portage/eix) even admitting they are doing it wrong.

I could write a whole post about how I would expect app-portage packages to be a notch over the rest of the tree, given that are packages written specifically for Gentoo, and should know and comply with the policies we set out for packages. The fact that at least two fails to do that, is a bad sign.

It is important to write some notes about the -g flags here, before people get confused. You can neither always disable them nor always enable them in Gentoo itself. The latter is a common assumption I read people thinking on: “since Gentoo already strips the debug information, why not always emitting it and leave the user just to tinker with the stripping features?”, well the answer is actually easy to understand; emitting the debug information is not a trivial task, as the compiler needs to write more data to the assembler that needs to compile it into the DWARF code (and Arnaldo from the dwarves/perf projects has shown me how messed up that is, trust me, it is a lot). It might sound very easy when you look at very simple source files, but it becomes tremendously complex when you have thousand-lines source files or, worse, C++ source files with templates. Not only it’s a lot of work for the assembler (and linker) but it also increases the size of the temporary object files (the products of compile+assembly) and of the final shared objects and executable files, before stripping; if you build in RAM, this means you’ll need more memory to store the build directory; if you build on disk, it increases the amount of stuff that gets written to and read from the disk, slowing down the process quite a bit in some cases.

At the same time, we cannot use (or suggest the use of) -g0 to disable the whole debug information generation. While the idea looks nice, and I also thought about it a bit, I wouldn’t be surprised if some packages actually made use of basic debug information; of course in those cases you’d also have RESTRICT=strip (and there are a number of those packages in the tree already). So you might want to try it by yourself but.. you’re on your own if you shoot yourself in the foot with that.

One could ask here why the debug USE flag is so package-specific, and why one shouldn’t just use -DNDEBUG to disable all the asserts to begin with. The answer is that, unfortunately, a huge amount of software fails to work when assertions are disabled, because they consider their asserts as the main error handling feature, while asserts should really just be used to verify situations that will not happen. Unfortunately, once you disable those asserts, they have no graceful handling of errors, so the software might crash badly or – much worse – corrupt the data before crashing. So before deciding whether you can add the debug USE flag, you got to find out whether the package supports doing so.

Also, you might have to note that not all the assertions are disabled by defining NDEBUG; for instance, g_assert() from glib is disabled via another macro (which if I recall correctly should be G_ASSERT_DISABLE or something along those lines); but even that might not be a good idea to disable, unless you know what you’re doing).

I sincerely try my best to have both the upstream packaging and the ebuild abiding by this rule: --enable-debug in feng adds some further debug messages, and adds function that fully free the resources before exiting (useful to remove false positives from valgrind), while --disable-debug removes all of that and all the assertions.

At any rate, the bottom line is that, as usual, Gentoo lacks consistency in its design and decisions; we really should start applying this kind of consistency to new packages as they are added, but I fear most reviewers, including the Sunrise reviewers, will forget about these details. I’m not asking perfection here, but at least having a clue?

Comments 10
  1. And I have used the verb “emit” in the proper way. If you couldn’t tell, please re-read the article.

  2. You describe two different things here, and I would like to separate them and comment on each of them:1. Some useflags like “debug” or “doc” do not have a completely consistent meaning. One reason for this is that the meaning must essentially depend on the project and what upstream provides, and the other reason is that for many packages, these flags could be much more fine-grained, e.g. “doc” could be splitted in “user-doc”, “developer-doc”, “interface-doc”, and these in turn be subdivided into manpages, info, ascii-doc, pdf-doc, ps-doc, html-doc, …; similarly for “debug” which could be subdivided into several verbosity levels and some other measures you mention above. So in the absence of this diversity some default choice must be made, and probably the best default choice for “debug” is to follow upstreams experience with bug reports: In fact, when a user really needs “debug” he is usually about to contact upstream anyway. If the user can easily fine-tune it manually (like in the mentioned case of eix where he can manually add the debug-flags if he does not like the choice caused by the flag) the better. Of course, it is important that it is documented what the flag actually does.2. The relation between the choice of code-paths (e.g. omitting certain optimizations) and backtraces is in its nature two-sided: You are right that it can happen that for a different code path a problem does not appear or appears in a different manner. However, these situations are relatively rare, and moreover, also the information that this changes things is often useful for debugging. On the other hand, it can happen that the optimized program has such another structure that a backtrace is almost useless. Also this situation is relatively rare. So, in my experience, usually the choice is not important, but when it is, it is not possible to predict which is the “right” choice: I do not agree with your opinion that one choice is always better than the other.

  3. “I could write a whole post about how I would expect app-portage packages to be a notch over the rest of the tree, given that are packages written specifically for Gentoo, and should know and comply with the policies we set out for packages. The fact that at least two fails to do that, is a bad sign.”Being the current tools-portage lead, I would be interested in that post.Additionally, I would like to point out the following message that is still applicable today.…What I find interesting is that nobody has really taken me up on the open-ended offer to touch/fix/change stuff in the tools-portage herd. Which leads me to believe that very few developers are really interested in them. (The one exception to the above is idl0r who has joined the herd and actively working on gentoolkit-dev.)

  4. Paul, to be honest, as far as I can see it’s always third party packages that have problems. Am I wrong that eix is not part of the team’s work?

  5. You are correct for eix, that package does not belong to the herd. Although due to its popularity, if it quits having an active maintainer it will probably migrate to being in the herd.

  6. > I sincerely try my best to have both the upstream packaging and the ebuild abiding by this rule: > –enable-debug in feng adds some further debug messages, and adds function that fully free the resources before exiting (useful to remove false positives from valgrind),> while –disable-debug removes all of that and all the assertions.Mmm if I’m reading this correctly, it means there is a third state where you don’t specify –enable nor –disable and it results in assertions enabled but other debug stuff disabled? How do you express this with an USE flag that’s either enabled or disabled and never unspecified?

  7. Nope, it’s all isolated by @#ifndef NDEBUG@ so it’s all or nothing there.

  8. I’ve come across one situation where I think it’s acceptable for the debug flag to mess with CFLAGS – to disable any special CFLAGS that the build system normally uses but may interfere with debugging information.For example, dev-util/codeblocks normally builds with -ffast-math, as the code is written specifically to work with it. With –enable-debug that flag isn’t used.I don’t have a problem with the debug flag dropping flags that the build system added in the first place. What do you think?

  9. Yes, sorry Ryan, I should have explicited that… I agree, in that case the debug USE flag is fine with playing with the flags.. it’s the one the _user_ asks for that shouldn’t be touched.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.