Remote debugging with GDB; part 1: Gentoo

In the last couple of days I had to focus my attention on just one issue, a work issue which I’m not going to discuss here since it wouldn’t make much sense either way. But this brought me to think about one particular point: remote debugging, and how to make it easier with Gentoo.

Remote debugging is a method commonly used when you have to debug issues on small, embedded systems, where running a full blown debugger is out of question. With GDB, this consists of running on the target system a tiny gdbserver process, which can be controlled by a master gdb instance via either serial port or TCP connection. The advantages don’t stop at not needing to run the full blown debugger: the target system may also be equipped with stripped programs with no debug information, and keep the debug information entirely on the system where the master gdb instance is being executed.

This is all fine and dandy for embedded systems but it also does not stop to this, you can easily make use of this technique to debug issues on remote servers, where you might not want to upload all the debug symbols for the software causing the trouble, it then becomes a nice Swiss Army knife for system administrators.

There are a few issues you got to work with when you use gdbserver though, some of which I think should be improved in Gentoo itself. First of all, to debug a remote system that is running a different architecture than yours, you need a cross-debugger; luckily crossdev can build that for you, then you need the actual cross-built gdbserver. Unfortunately, even though the server is small and self-sufficient, it is currently only being built by sys-devel/gdb which is not so nice for embedded systems; we’d need a minimal or server-only USE flag for that package or even better a dev-util/gdbserver standalone package so that it could be cross-built and installed without building the full blown debugger which is not useful at all.

Then there is the problem of debug information. In Gentoo we already provide some useful support for that through the splitdebug feature, which takes all the debug information from an ELF file, executable or library, and splits it out in a .debug file that only contains the information useful during debugging. This split does not really help much on a desktop system, since the debug information wouldn’t be loaded anyway, my reasoning to have it separated is to make sure I can drop them all at once if I’m very short on space, without breaking my system. On the other hand, it is very useful to have it around for embedded systems for instance, although it currently is a bit clumsy to use.

Right now one common way to achieve proper archiving of debug information and stripping them in production is using the buildpkg feature together with splitdebug, and set up an INSTALL_MASK variable for /usr/lib/debug when doing the build of the root. The alternative is to simply remove that directory before preparing the tarball of the rootfs or stuff like that. This works decently, since the binary packages will have the full debug information, and you’d just need to reinstall the package you need debug information for without the INSTALL_MASK. Unfortunately this will end up replacing the files from the rest of the package, which is not so nice because it might change the timestamps on the filesystem, as well as wasting time, and eventually flash too.

This also does not play entirely nice with remote server administration. The server where this blog is hosted is a Gentoo vserver guest, it was installed starting from a standard amd64 stage, then I built a local chroot starting from the same stage, setting it up exactly as I wanted it to be; finally, I synced over the Portage configuration files, the tree and the binary packages built of all it, and installed them. The remote copy of the packages archive is bigger than the actual used packages, since it contains also the packages that are just build dependencies, but the overhead of this I can ignore without too much problems. On the other hand, if I were to package in all the debug information, and just avoid installing it with INSTALL_MASK, the overhead wouldn’t be this ignorable.

My ideal solution for this would involve making Portage more aware of the splitdebug feature, and actually split it out on package level too, similarly to what RPM does with the -dbg package. By creating a -debug or -dbg binpkg to the side of each package that would otherwise have /usr/lib/debug in it, and giving the user an option on whether to install the sub-package, it would be possible to know whether to merge on the root filesystem the debug information or ot, without using INSTALL_MASK. Additionally, having a common suffix for these packages would allow me to just ignore them when syncing them to the remote server, removing the overhead.

Dreaming a bit more, it could be possible to design multiple sub-package automatic generation, to resemble a bit what binary distributions like Debian and RedHat have been doing all these years, to split documentation in its -doc package, the headers and static libraries in -dev and so on. Then it would just require to give the user ability to choose which subpackages to install by default, and a per-package override. A normal desktop Gentoo system would probably want to have everything installed by default, but if you’re deploying Gentoo-based systems, you probably would just have a chroot on a build machine that does the work, and then the system would just get the subset needed (with or without documentation). Maybe it’s not going to be easy to implement, and I’m sure it’s going to be controversial, but I think it might be worth looking into it. Implementing it in a non-disruptive way (with respect to the average user and developer workflow) is probably going to make it feasible.

Tomorrow, hopefully, I’ll be writing some more distribution-agnostic instructions on how to remotely debug applications using GDB.