Tag: Embedded

Inside the life of an embedded developer

Now that the new ultrabook is working almost fine (I’m still tweaking the kernel to make sure it works as intended, especially for what concerns the brightness, and tweaking the power usage, for once, to give me some more power), I think it’s time for me to get used to the new keyboard, which is definitely different from the one I’m used to on the Dell, and more similar to the one I’ve used on the Mac instead. To do so, I think I’ll resume writing a lot on different things that are not strictly related to what I’m doing at the moment but talk about random topic I know about. Here’s the first of this series.

So, you probably all know me for a Gentoo Linux developer, some of you will know me for a multimedia-related developer thanks to my work on xine, libav, and a few other projects; a few would know me as a Ruby and Rails developer, but that’s not something I’m usually boasting or happy about. In general, I think that I’m doing my best not to work on Rails if I can. One of the things I’ve done over the years has been working on a few embedded projects, a couple of which has been Linux related. Despite what some people thought of me before, I’m not an electrical engineer, so I’m afraid I haven’t had the hands-on experience that many other people I know had, which I have to say, sometimes bother me a bit.

Anyway, the projects I worked on over time have never been huge project, or very well funded, or managed projects, which meant that despite Greg’s hope as expressed on the gentoo-dev mailing list, I never had a company train me in the legal landscape of Free Software licenses. Actually in most of the cases, I ended up being the one that had to explain to my manager how these licenses interact. And while I take licenses very seriously, and I do my best to make sure that they are respected, even when the task falls on the shoulder of someone else, and that someone might not actually do everything by the book.

So, while I’m not a lawyer and I really don’t want to go near that option, I always take the chance of understand licenses correctly and, when my ass is on the line, I make sure to verify the licenses of what I’m using for my job. One such example is the project I’ve been working since last March, of which I’ve prepared the firmware, based on Gentoo Linux. To make sure that all the licenses were respected properly, I had to come up with the list of packages that the firmware is composed of, and then verify the license of each. Doing this I ended up finding a problem with PulseAudio due to linking in the (GPL-only) GDBM library.

What happens is that if the company you’re working for has no experience with licenses, and/or does not want to pay to involve lawyers to review what is being done, it’s extremely easy for mistake to happen unless you are very careful. And in many cases, if you, as a developer, pay more attention to the licenses than your manager does, it’s also seen as a negative point, as that’s not how they’d like for you to employ your time. Of course you can say that you shouldn’t be working for that company then, but sometimes it’s not like you have tons of options.

But this is by far not the only problem. Sometimes, what happens is a classic 801 — that is that instead of writing custom code for the embedded platform you’re developing for, the company wants you to re-use previous code, which has a high likelihood of being written in a language that is completely unsuitable for the embedded world: C++, Java, COBOL, PHP….

Speaking of which, here’s an anecdote from my previous experiences in the field: at one point I was working on the firmware for an ARM7 device, that had to run an application written in Java. Said application was originally written to use PostgreSQL and Tomcat, with a full-fledged browser, but had to run on a tiny resistive display with SQLite. But since at the time IcedTea was nowhere to be found, and the device wouldn’t have had enough memory for it anyway, the original implementation used a slightly patched GCJ to build the application to ELF, and used JNI hooks to link to SQLite. The time (and money, when my time was involved) spent making sure the system wouldn’t run out of memory would probably have sufficed to rewrite the whole thing in C. And before you ask, the code bases between the Tomcat and the GCJ versions started drifting almost immediately, so code sharing was not a good option anyway.

Now, to finish this mostly pointless, anecdotal post of mine, I’d like to write a few words of commentary about embedded systems, systemd, and openrc. Whenever I head one or the other side saying that embedded people love the other, I think they don’t know how different embedded development can be from one company to the next, and even between good companies there are so big variation, that make them stand lightyears apart from bad companies like some of the ones I described above. Both sides have good points for the embedded world; what you choose depends vastly on what you’re doing.

If memory and timing are your highest constraints, then it’s very likely that you’re looking into systemd. If you don’t have those kind of constraints, but you’re building a re-usable or highly customizable platform, it’s likely you’re not going to choose it. The reason? While if you’re half-decent in your development cross-compilation shouldn’t be a problem, the reality is that in many place, it is. What happen then is that you want to be able to make last-minute changes, especially in the boot process, for debugging purposes, using shell scripts is vastly easier, and for some people, doing it more easily is the target, rather than doing it right (and this is far from saying that I find the whole set of systemd ideas and designs “right”).

But this is a discussion that is more of a flamebait than anything else, and if I want to go into details I should probably spend more time on it than I’m comfortable doing now. In general, the only thing I’m afraid of, is that too many people make assumption on how people do things, or take for granted that companies, big and small, care about doing things “right” — by my experience, that’s not really the case, that often.

Remote debugging with GDB; part 2: GDB

When I first wrote the original part 1 of the tutorial I was working on a job project for a customer of mine; time passed without me writing the second part, and this actually went this way for over an year. Talk about long-term commitment. On the other hand, since this is going to be used almost literally for documentation for the same customer (different project, though), I’m resuming the work writing this.

On the bright side, at least one of the Gentoo changes that I have described as desired in the previous post is now solved: we have a dev-util/gdbserver ebuild in the tree (maintained by me, written because of the current project, so can be said to be contributed by the current customer of mine). So from the device side, you only need to install that to have it ready for debugging.

On the build-machine side, you need the cross-debugger for the target architecture and system; you can install it with crossdev like you do for the cross-linker and cross-compiler, but with a twist: for GDB 7.0 or later you need to install with expat support (because it seems like they now use XML as a protocol to talk between the client/master and the server/slave). So for instance you run this command: USE=expat crossdev -t $CHOST --ex-gdb.

If you’re not using Gentoo on either side, then you’re going to have to deal with both builds by hand… tough luck. On the other hand, the gdbserver executable is very small in size, on the latest version it’s just a bit over 100KiB, so building it takes very little time and takes up very little space even on embedded Linux systems.

On the remote system, there are three modes of operation: single program, attach to running process, and multi-execution daemon; the latter is my favourite mode of operations, as you just need to start gdbserver like a standard Unix daemon, waiting for incoming requests and handling them one by one as long as it’s needed (yes, I guess it would make sense to have an init script for gdbserver when used on not-too-embedded systems). In all three cases, it needs a communication media to work; on modern, and less embedded, systems that media is almost always a TCP/IP connection (just give it a ip:port address format), but the software supports using serial ports for that task as well.

The executable for gdbserver in my system is just 72KiB, so it’s very quick to upload it to the target, even if it’s an embedded Linux system. After uploading, you have to start it on the target; you can either start a program directly, attach to a process already running or, my favourite, use it for multiple debugging at once. In this latter mode, gdbserver acts a lot like a standard Unix daemon, waiting for incoming requests and handling them one by one for as long as it’s needed. This actually makes me consider the idea of setting up an init script to use for debug and test systems.

To start the daemon-like instance, just use the --multi option:

arm # gdbserver --multi 192.168.0.38:12345

Now you can connect to the server through the cross-debugger built earlier:

% arm-linux-gnu-gdb
(gdb) target extended-remote 192.168.0.38:12345

This drops us inside the server, or rather inside the target board, ready to debug. So let’s upload this crashy program:

% cat crash.c
#include 

int main() {
        char *str = NULL;
        return strcmp(str, "foo");
}
% arm-linux-gnu-gcc -ggdb crash.c -o crash
% arm-linux-gnu-strip crash -o crash_s
% wput crash_s ...

At this point you can note one very important point: we’re building the the program with the -ggdb option to produce the debug information for the program, so that GDB can tell which variable is which and which address correspond to which function; this is very important to have meaningful backtrace and is even more important if you plan on using further features including, for instance, break- and watch-points. You could technically even use -g3 to embed the source files into the final executable, which is particularly useful if you plan on having multiple firmwares around, but that’s not always working correctly so leave it as that for now.

But even if we need the debug information in the compiled executable object, we don’t need the (bigger) binary to be present on our target system; the reason is that the debug information is only composed of extra sections, and does not add or subtract any code or information to the runtime-executable code. This is why -ggdb does not make your system slower, and why stripping an executable of its debug information is enough to make it slimmer. At the same time, the extra sections added by -ggdb and stripped by strip are never going to be mapped from disk into memory anyway, at least when executing them with ld.so, so there is no extra memory usage caused by having non-stripped binaries. The only difference is in the size of the file, which might still be something if you’re parsing it for some reason, and might fragment it into multiple pieces.

Anyway, since the debug information, as just stated, is not needed, nor used at all, at runtime, there is no reason why the information has to stay on the target: the gdbserver program is just going to execute whatever the master GDB instance will ask it to, and has no reason to have a clue about the executable file or its debugging information. So you can just copy a stripped version of the file and upload it on the target; you can then use it normally, or run it through gdbserver.

After uploading the file you need to set up GDB to run the correct executable, and to load the symbols from the local copy of it:

(gdb) symbol-file crash
(gdb) set remote exec-file /uploaded/to/crash_s

and you’re completely set. You can now use the standard GDB commands (break, run, handle, x, kill, …) like you were running the debugger on the target itself! The gdbserver program will take the action, and the main GDB instance will direct it as per your instructions.

Have fun remote debugging your code!

And sometimes it’s my fault

Picking up from my previous post I finally was able to install a boot loader on the compact flash card, yuppie! Indeed it was quite easy to do so with grub, after Zeev made me notice that yeah there is nothing wrong with it, it’s just a matter of configuring it properly.

Unfortunately starting up everything properly wasn’t really a cakewalk, but it was mostly my fault:

  • first of all, I created grub.conf and not menu.lst, which of course wasn’t picked up by grub;
  • then I used the wrong device for the root parameter (/dev/hda1 rather than /dev/sda1);
  • I built-in ext3 support but formatted the card as ext4 (non-journaled);
  • I forgot to create /dev/console so I couldn’t see OpenRC startup output (on a related note: if you pass twice the console parameters to the kernel, like I do on Yamato, /dev/console only outputs to the second one not the first, why? Either I forgot to configure it properly, or I should look into fixing this into the kernel);
  • I set up the serial getty with the wrong line speed (9600 while both grub and the kernel use 115200);
  • I forgot to add a password for root (and also a little note: ROOT= merges don’t create users and groups properly, but this is not Gentoo’s fault, rather shadow, that manages Linux authentication databases, don’t support accessing files that are on a different root — I think FreeBSD utilities do handle that though);
  • hostapd failed to start because it couldn’t find libnl.so (broken DEPEND/RDEPEND assignment in the ebuild, I fixed that for hostapd and a bunch of other ebuilds making the same mistake; I also opened a repoman check request so maybe in the future it can be avoided)

Now I finally got somewhat closer to what I need to do, but it’s still not yet complete. I hope to be able to get it as main router soonish. I’m also going to reduce the load on hostapd in the future since I finally went around buying some cat5e cable to wire up my house (FTP, since I need to use the same connections as the electrical system, sigh). And on that note I added some of the hardware I still need to complete the task to my wishlist.

Getting this to work is giving me quite a bit of insight in what we should probably be doing to support more properly embedded system cases like mine, and I don’t think this is anything extremely complex, like some want us to think; it only requires starting to accept some different changes, for instance by spitting the meaning of system from “what is inside the stage3” so that we can actually have stuff in the stage3 that is not in system and that can be removed with a simple -C (or --depclean after removing what uses it), instead of having to deal with profile-mucking files. If that was the case and the system set was really minimal (like Reiman suggested, a minimal POSIX set), then we could be saying “okay, that package is in system so I don’t really have to depend on it”.

Preparing a router; improving Gentoo

I’m finally working on setting up my home gateway with a Gentoo-based system, as I wrote about some time ago, since I didn’t find anything that looked even barely like what I wanted, off the shelf.

But since the router is going to be working on CF flash drives, I’m also using some tricks to get it up, and this is binging me to find even more problems within Gentoo, but that’s as usual and to be expected, isn’t it?

The first problem is that I don’t want to have superfluous stuff in the router: Portage (I don’t want to live update it), Perl, Python, autotools, other build-related tools, all that stuff should not be there at all. So the obvious is to buil dup my own profile to reduce the system set. This is actually quite boring and a problem in my opinion. Given the stage3 is system itself, to have autotools in the stages (which we do want) we need to have them in system… but they shouldn’t be.

The second problem is that not having either Portage nor autotools in the final system means that the dependencies over Python and Perl of the rest of the packages need to be perfect: no more and no less. Unfortunately sys-apps/file depends on Python unconditionally and util-linux installs a Perl script without depending on it (or having a perl USE flag). If the runtime dependencies are not properly expressed, my router won’t work as intended and that’s going t make me sad, depressed, and probably very angry with those who don’t express dependencies fully. And C++ dependencies will also be a problem.

I’ll be posting some tips and tricks soonish, for now I’m still working on defining what should I add to my router, and how. In particular I’ve been able to do stuff like enabling graphite n the GCC used for building, but not enabling it on the gcc needed on the system, so that it won’t bring in ppl and cloog-ppl, in a quite decent fashion.

Again, being able to prepare this kind of systems programmatically needs more work than just a few tricks and a lot of fighting so you probably will read more about this, and find more bugs about this in the future.

And as a bottom-line: since this is working mostly on stable tree, I also noticed that a few packages that should have been stable already weren’t… so I started opening a few stable requests for my packages; I really need to find a way to make sure they don’t excessively slack off…

Silly projects: Gentoo on a WRT54GL

I have one silly project I should probably look forward to work on this weekend to vent off some steam: getting my WRT54GL router to run with Gentoo/MIPS. I know it’s probably going to fail because I know near to nothing about MIPS, I know nothing about Gentoo/MIPS, and I remember being told that the mipsel target that WRT54GL are is not well supported by Gentoo. Either way, I’m going to try.

You could probably be wondering why I would be trying something as silly as this, and the reasons are actually a bit of a series. The first problem is that I lack an IPMI agent for accessing Yamato remotely (for remote light out and other things) and I’m sure that’s going to be useful to me soonish. The second one is that I need to set up again the routing of my office with a single wireless client, instead of the current setup which I prepared last year with Yamato having a wireless card.

The problem with IPMI is that I plan on not being at home all day every day in the future; I actually hope to be able to get a driving license this year and make good use of it by finding a job out of home (especially important for my mental health lately!) and in that case I’d be likely to need a way to access Yamato remotely if it gets messed up. Having a low-power system like the WRT to work as a jump box is acceptable I’d say.

With routing, the issue is at the same time simpler and more complex. Simpler because I just need a router to route between the general wireless network (which is accessible by almost anybody) and the wired network that I use for the office and my bedroom. More complex because the original setup used the WRT54GL, then I decided to move to just a single network card, but now I’m in a setup that is quite messed up: Yamato routes all the traffic of PlayStation 3, AppleTV, and iMac, as well as the eventual computers I need to fix and other stuff like that which actually translates the whole thing into a real mess, especially because I ended up splitting the network to such extents that applying NFS ACLs simply by IP masks is impossible.

Of course I could just be using OpenWRT like I did before, but since every upgrade of OpenWRT has been a real mess to deal with (with all their changes into setup, interfaces, nvram and so on), I’m actually thinking that Gentoo would be easier to maintain for me, given I’d just have to update the image once in a blue moon hopefully. I could also just start the router through a TFTP-provide image and then leave it to be with that. At any rate, it would also be a nice experiment and a way to learn a bit more about embedded systems, so…

Right now I only found how to take it apart and I noticed that I have to solder in myself the pins for serial console access, unfortunately it’s almost 1am so soldering them in right now is out of question (I also have to find the pins, which I’m not sure I have at home, worse case scenario I’m going to desolder them from somewhere). I should probably go taking a look to whether the Linus kernel tree can boot on this thing. When I last used it, OpenWRT only supported 2.4 on it; while 2.6 was being worked on it didn’t work on this model, and the wireless network was supposed never to work on it, since it’s using the infamous Broadcom wireless chip (which nowadays might actually work out of the Linus tree via the b43 driver).

If anybody (especially the colleagues actually working on Gentoo/MIPS) have a clue about what I’m to expect out of this, I’d be quite happy to hear it, even if it’s “You’re crazy, it’s never going to work” or “Leave it alone, it’s too much hassle to bother with”.

Oh and yah, I know this thing is not powerful enough to build and it’ll have to go through cross-compiling, and cross-compiling with Portage is not nice, and all the stuff like that. I guess the point is that I intend to work a bit more on that matter, even if currently I’m not paid to do so (I was for a while some time ago). There are more than a few things that I’m interested in looking at to find a solution, actually. It’s very low priority (unless someone bribes me to pick it up) but maybe I can be of help to the broader picture somehow.

Remote debugging with GDB; part 1: Gentoo

In the last couple of days I had to focus my attention on just one issue, a work issue which I’m not going to discuss here since it wouldn’t make much sense either way. But this brought me to think about one particular point: remote debugging, and how to make it easier with Gentoo.

Remote debugging is a method commonly used when you have to debug issues on small, embedded systems, where running a full blown debugger is out of question. With GDB, this consists of running on the target system a tiny gdbserver proces, which can be controlled by a master gdb instance via either serial port or TCP connection. The advantages don’t stop at not needing to run the full blown debugger: the target system may also be equipped with stripped programs with no debug information, and keep the debug information entirely on the system where the master gdb instance is being executed.

This is all fine and dandy for embedded systems but it also does not stop to this, you can easily make use of this technique to debug issues on remote servers, where you might not want to upload all the debug symbols for the software causing the trouble, it then becomes a nice Swiss Army knife for system administrators.

There are a few issues you got to work with when you use gdbserver though, some of which I think should be improved in Gentoo itself. First of all, to debug a remote system that is running a different architecture than yours, you need a cross-debugger; luckily crossdev can build that for you, then you need the actual cross-built gdbserver. Unfortunately, even though the server is small and self-sufficient, it is currently only being built by sys-devel/gdb which is not so nice for embedded systems; we’d need a minimal or server-only USE flag for that package or even better a dev-util/gdbserver standalone package so that it could be cross-built and installed without building the full blown debugger which is not useful at all.

Then there is the problem of debug information. In Gentoo we already provide some useful support for that through the splitdebug feature, which takes all the debug information from an ELF file, executable or library, and splits it out in a .debug file that only contains the information useful during debugging. This split does not really help much on a desktop system, since the debug information wouldn’t be loaded anyway, my reasoning to have it separated is to make sure I can drop them all at once if I’m very short on space, without breaking my system. On the other hand, it is very useful to have it around for embedded systems for instance, although it currently is a bit clumsy to use.

Right now one common way to achieve proper archiving of debug information and stripping them in production is using the buildpkg feature together with splitdebug, and set up an INSTALL_MASK variable for /usr/lib/debug when doing the build of the root. The alternative is to simply remove that directory before preparing the tarball of the rootfs or stuff like that. This works decently, since the binary packages will have the full debug information, and you’d just need to reinstall the package you need debug information for without the INSTALL_MASK. Unfortunately this will end up replacing the files from the rest of the package, which is not so nice because it might change the timestamps on the filesystem, as well as wasting time, and eventually flash too.

This also does not play entirely nice with remote server administration. The server where this blog is hosted is a Gentoo vserver guest, it was installed starting from a standard amd64 stage, then I built a local chroot starting from the same stage, setting it up exactly as I wanted it to be; finally, I synced over the Portage configuration files, the tree and the binary packages built of all it, and installed them. The remote copy of the packages archive is bigger than the actual used packages, since it contains also the packages that are just build dependencies, but the overhead of this I can ignore without too much problems. On the other hand, if I were to package in all the debug information, and just avoid installing it with INSTALL_MASK, the overhead wouldn’t be this ignorable.

My ideal solution for this would involve making Portage more aware of the splitdebug feature, and actually split it out on package level too, similarly to what RPM does with the -dbg package. By creating a -debug or -dbg binpkg to the side of each package that would otherwise have /usr/lib/debug in it, and giving the user an option on whether to install the sub-package, it would be possible to know whether to merge on the root filesystem the debug information or ot, without using INSTALL_MASK. Additionally, having a common suffix for these packages would allow me to just ignore them when syncing them to the remote server, removing the overhead.

Dreaming a bit more, it could be possible to design multiple sub-package automatic generation, to resemble a bit what binary distributions like Debian and RedHat have been doing all these years, to split documentation in its -doc package, the headers and static libraries in -dev and so on. Then it would just require to give the user ability to choose which subpackages to install by default, and a per-package override. A normal desktop Gentoo system would probably want to have everything installed by default, but if you’re deploying Gentoo-based systems, you probably would just have a chroot on a build machine that does the work, and then the system would just get the subset needed (with or without documentation). Maybe it’s not going to be easy to implement, and I’m sure it’s going to be controversial, but I think it might be worth looking into it. Implementing it in a non-disruptive way (with respect to the average user and developer workflow) is probably going to make it feasible.

Tomorrow, hopefully, I’ll be writing some more distribution-agnostic instructions on how to remotely debug applications using GDB.

Minimising alsa-lib

As I said before, I’ve been working on trying to improve the ALSA packages in Gentoo, so you can see in 1.0.14 series that the ALSA_CARDS values are shown by portage so that you know which one you can enable, and at least for some of them you get a failure if you try to enable them without the correct configuration options in the kernel.

Today, I wanted to try minimising the footprint of alsa-lib, as that is a library that almost every system has loaded in a few instances.. the trick I had in mind (and I have locally realised) is to allow users to choose the PCM plugins they want to compile in the library… the PCM plugins are things like dmix and dsnoop (used for software mixing of, respectively, output and input), iec958 (passthrough digital audio) and plug (resampling), but they are not the only ones of course. They are the bricks that compose the asound.conf files.

Of course a normal system rarely uses all of the plugins that are usually built in alsa-lib, and for some particular setups (mine included), the used plugins are very very limited. That’s why attacking the amount of plugins built was my first choice, so I created an ALSA_PCM_PLUGINS variable and used that to decide what had to be built and what not; in my case just plug and iec958, as I use PulseAudio for software mixing (I also have some space for hardware mixing if I need to bypass PulseAudio) and plug is pretty much a Swiss knife.

Also, there’s the ALISP interpreter used for very complex setups, and there is the whole support for MIDI stuff (instruments, sequencers, sample loaders, rawmidi output) that is unlikely to be used on most of modern systems (MIDI has a pretty much limited use nowadays, considering the kind of sound cards we have available; even software like Audacious that play .mid files is simply using software synthesis rather than hardware instruments).

My pretty much limited setup shows now this:

[ebuild   R   ] media-libs/alsa-lib-1.0.14_rc1  USE="-alisp -debug -doc -midi"
ALSA_PCM_PLUGINS="-adpcm -alaw -asym -copy -dmix -dshare -dsnoop -empty
-extplug -file -hooks iec958 -ioplug -ladspa -lfloat -linear -meter -mulaw -multi
-null plug -rate -route -share -shm -softvol" 0 kB

and the size of the package (built with -ggdb) is down from 6.4MB to 3.4MB now, that’s almost half of the previous size. Considering that a big part of that package is composed of configuration files, the size of the library decreased sensibly.

I’m waiting to hear comments on gentoo-dev before proceeding with adding the variable, the defaults and the ebuild itself, but if all goes as I hope, alsa-lib from Gentoo would be also simpler to use on embedded devices.