When I first wrote the original part 1 of the tutorial I was working on a job project for a customer of mine; time passed without me writing the second part, and this actually went this way for over an year. Talk about long-term commitment. On the other hand, since this is going to be used almost literally for documentation for the same customer (different project, though), I’m resuming the work writing this.
On the bright side, at least one of the Gentoo changes that I have described as desired in the previous post is now solved: we have a dev-util/gdbserver ebuild in the tree (maintained by me, written because of the current project, so can be said to be contributed by the current customer of mine). So from the device side, you only need to install that to have it ready for debugging.
On the build-machine side, you need the cross-debugger for the target architecture and system; you can install it with crossdev like you do for the cross-linker and cross-compiler, but with a twist: for GDB 7.0 or later you need to install with expat support (because it seems like they now use XML as a protocol to talk between the client/master and the server/slave). So for instance you run this command: USE=expat crossdev -t $CHOST --ex-gdb.
If you’re not using Gentoo on either side, then you’re going to have to deal with both builds by hand… tough luck. On the other hand, the gdbserver executable is very small in size, on the latest version it’s just a bit over 100KiB, so building it takes very little time and takes up very little space even on embedded Linux systems.
On the remote system, there are three modes of operation: single program, attach to running process, and multi-execution daemon; the latter is my favourite mode of operations, as you just need to start gdbserver like a standard Unix daemon, waiting for incoming requests and handling them one by one as long as it’s needed (yes, I guess it would make sense to have an init script for gdbserver when used on not-too-embedded systems). In all three cases, it needs a communication media to work; on modern, and less embedded, systems that media is almost always a TCP/IP connection (just give it a ip:port address format), but the software supports using serial ports for that task as well.
The executable for gdbserver in my system is just 72KiB, so it’s very quick to upload it to the target, even if it’s an embedded Linux system. After uploading, you have to start it on the target; you can either start a program directly, attach to a process already running or, my favourite, use it for multiple debugging at once. In this latter mode, gdbserver acts a lot like a standard Unix daemon, waiting for incoming requests and handling them one by one for as long as it’s needed. This actually makes me consider the idea of setting up an init script to use for debug and test systems.
To start the daemon-like instance, just use the --multi option:
arm # gdbserver --multi 192.168.0.38:12345Code language: CSS (css)Now you can connect to the server through the cross-debugger built earlier:
% arm-linux-gnu-gdb
(gdb) target extended-remote 192.168.0.38:12345Code language: CSS (css)This drops us inside the server, or rather inside the target board, ready to debug. So let’s upload this crashy program:
% cat crash.c
#include <string.h>
int main() {
char *str = NULL;
return strcmp(str, "foo");
}
% arm-linux-gnu-gcc -ggdb crash.c -o crash
% arm-linux-gnu-strip crash -o crash_s
% wput crash_s ...Code language: PHP (php)At this point you can note one very important point: we’re building the the program with the -ggdb option to produce the debug information for the program, so that GDB can tell which variable is which and which address correspond to which function; this is very important to have meaningful backtrace and is even more important if you plan on using further features including, for instance, break- and watch-points. You could technically even use -g3 to embed the source files into the final executable, which is particularly useful if you plan on having multiple firmware images around, but that’s not always working correctly so leave it as that for now.
But even if we need the debug information in the compiled executable object, we don’t need the (bigger) binary to be present on our target system; the reason is that the debug information is only composed of extra sections, and does not add or subtract any code or information to the runtime-executable code. This is why -ggdb does not make your system slower, and why stripping an executable of its debug information is enough to make it slimmer. At the same time, the extra sections added by -ggdb and stripped by strip are never going to be mapped from disk into memory anyway, at least when executing them with ld.so, so there is no extra memory usage caused by having non-stripped binaries. The only difference is in the size of the file, which might still be something if you’re parsing it for some reason, and might fragment it into multiple pieces.
Anyway, since the debug information, as just stated, is not needed, nor used at all, at runtime, there is no reason why the information has to stay on the target: the gdbserver program is just going to execute whatever the master GDB instance will ask it to, and has no reason to have a clue about the executable file or its debugging information. So you can just copy a stripped version of the file and upload it on the target; you can then use it normally, or run it through gdbserver.
After uploading the file you need to set up GDB to run the correct executable, and to load the symbols from the local copy of it:
(gdb) symbol-file crash
(gdb) set remote exec-file /uploaded/to/crash_sCode language: JavaScript (javascript)and you’re completely set. You can now use the standard GDB commands (break, run, handle, x, kill, …) like you were running the debugger on the target itself! The gdbserver program will take the action, and the main GDB instance will direct it as per your instructions.
Have fun remote debugging your code!