Remote debugging with GDB; part 2: GDB

When I first wrote the original part 1 of the tutorial I was working on a job project for a customer of mine; time passed without me writing the second part, and this actually went this way for over an year. Talk about long-term commitment. On the other hand, since this is going to be used almost literally for documentation for the same customer (different project, though), I’m resuming the work writing this.

On the bright side, at least one of the Gentoo changes that I have described as desired in the previous post is now solved: we have a dev-util/gdbserver ebuild in the tree (maintained by me, written because of the current project, so can be said to be contributed by the current customer of mine). So from the device side, you only need to install that to have it ready for debugging.

On the build-machine side, you need the cross-debugger for the target architecture and system; you can install it with crossdev like you do for the cross-linker and cross-compiler, but with a twist: for GDB 7.0 or later you need to install with expat support (because it seems like they now use XML as a protocol to talk between the client/master and the server/slave). So for instance you run this command: USE=expat crossdev -t $CHOST --ex-gdb.

If you’re not using Gentoo on either side, then you’re going to have to deal with both builds by hand… tough luck. On the other hand, the gdbserver executable is very small in size, on the latest version it’s just a bit over 100KiB, so building it takes very little time and takes up very little space even on embedded Linux systems.

On the remote system, there are three modes of operation: single program, attach to running process, and multi-execution daemon; the latter is my favourite mode of operations, as you just need to start gdbserver like a standard Unix daemon, waiting for incoming requests and handling them one by one as long as it’s needed (yes, I guess it would make sense to have an init script for gdbserver when used on not-too-embedded systems). In all three cases, it needs a communication media to work; on modern, and less embedded, systems that media is almost always a TCP/IP connection (just give it a ip:port address format), but the software supports using serial ports for that task as well.

The executable for gdbserver in my system is just 72KiB, so it’s very quick to upload it to the target, even if it’s an embedded Linux system. After uploading, you have to start it on the target; you can either start a program directly, attach to a process already running or, my favourite, use it for multiple debugging at once. In this latter mode, gdbserver acts a lot like a standard Unix daemon, waiting for incoming requests and handling them one by one for as long as it’s needed. This actually makes me consider the idea of setting up an init script to use for debug and test systems.

To start the daemon-like instance, just use the --multi option:

arm # gdbserver --multi

Now you can connect to the server through the cross-debugger built earlier:

% arm-linux-gnu-gdb
(gdb) target extended-remote

This drops us inside the server, or rather inside the target board, ready to debug. So let’s upload this crashy program:

% cat crash.c

int main() {
        char *str = NULL;
        return strcmp(str, "foo");
% arm-linux-gnu-gcc -ggdb crash.c -o crash
% arm-linux-gnu-strip crash -o crash_s
% wput crash_s ...

At this point you can note one very important point: we’re building the the program with the -ggdb option to produce the debug information for the program, so that GDB can tell which variable is which and which address correspond to which function; this is very important to have meaningful backtrace and is even more important if you plan on using further features including, for instance, break- and watch-points. You could technically even use -g3 to embed the source files into the final executable, which is particularly useful if you plan on having multiple firmwares around, but that’s not always working correctly so leave it as that for now.

But even if we need the debug information in the compiled executable object, we don’t need the (bigger) binary to be present on our target system; the reason is that the debug information is only composed of extra sections, and does not add or subtract any code or information to the runtime-executable code. This is why -ggdb does not make your system slower, and why stripping an executable of its debug information is enough to make it slimmer. At the same time, the extra sections added by -ggdb and stripped by strip are never going to be mapped from disk into memory anyway, at least when executing them with, so there is no extra memory usage caused by having non-stripped binaries. The only difference is in the size of the file, which might still be something if you’re parsing it for some reason, and might fragment it into multiple pieces.

Anyway, since the debug information, as just stated, is not needed, nor used at all, at runtime, there is no reason why the information has to stay on the target: the gdbserver program is just going to execute whatever the master GDB instance will ask it to, and has no reason to have a clue about the executable file or its debugging information. So you can just copy a stripped version of the file and upload it on the target; you can then use it normally, or run it through gdbserver.

After uploading the file you need to set up GDB to run the correct executable, and to load the symbols from the local copy of it:

(gdb) symbol-file crash
(gdb) set remote exec-file /uploaded/to/crash_s

and you’re completely set. You can now use the standard GDB commands (break, run, handle, x, kill, …) like you were running the debugger on the target itself! The gdbserver program will take the action, and the main GDB instance will direct it as per your instructions.

Have fun remote debugging your code!

One thought on “Remote debugging with GDB; part 2: GDB

  1. I’m sure you’ve never heard of me, but I’ve been reading your posts for years now. There may be a wave of posts from people saying “don’t resign”, and you probably don’t want to listen to a mass of folks expressing their sentiments the same way, but please hear me out.Your posts have been some of the most thorough, information-dense material I’ve read. You’ve tested/played with/reviewed a wide variety of software systems. Whenever you’ve touched something I know something about, your conclusions have been very similar to mine. I’ve also tended to agree with your commits and, while slightly less adamant in supporting free software over proprietary, your general philosophy.Because of this, I’ve come to trust your opinions on things I’ve not had the opportunity to test myself. I don’t know how many decisions I’ve made based on your posts, but it’s certainly dozens. I’ve never been let down.I use Gentoo on several systems. I’m not one of the users who particularly cares about CFLAGS or bleeding-edge up-to-dateness. I was extremely distraught when a client called me in to repair a server that mysteriously stopped working and I discovered it had a script which ran a forced world update. I’m in that unspoken mass of people who use Gentoo for features like config-protect or how it doesn’t install programs into the default runlevels or start them up when they’re first placed on the system.I know it seems frustrating now, but please take a step back and look at the situation. Don’t make a decision to punish others, the Gentoo team, whatever. If you resign from the Gentoo project the people you’ll most be damaging will be people like me: the people who read your posts, the people who find answers from you on Google, and the people who may have never heard your name but use the software you maintain.What you produce in your spare time has value. You’re changing the world for the better. You have no obligation to continue doing so. When you stop (and, naturally, we all must stop some time), you probably won’t hear much like this – it’s more likely just the usual background patter of spam and content-free posts. But please do think hard about it, and when you do don’t think that your work has gone unappreciated. Most of the prayer in a church is never spoken, most of the impact of art is never expressed, and most good deeds go unrewarded. But there is worth in continuing to do them nonetheless.There will always be other people in the way. Don’t let them grind you down.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s