Tag: Cython

USB Captures Yak Shaving

Months ago I complained about the state of USB captures solutions in 2020. One of the issues it that you can’t easily provide a capture filter to libpcap, because they don’t want to implement user-mode capturing, and Linux does not provide BPF-based filtering for usbmon.

While I do still find it an interesting idea to add BPF filtering there, my kernel-fu is still fairly limited, and I thought I would start with something easier: filtering in userspace with a custom capture program. This also got me a bit more comfortable with the actual capture API, that I have been ignoring for the most part.

As I said before, languages are tools, and I could have tried implementing the tool in a different programming language. But on the other hand, I’m trying to get this done to integrate with the rest of the chatter-extraction tools I released as part of usbmon-tools, so why straying away (too much) from the path? Well, turns out that the usbmon interface is a bit too complicated to implement in pure Python, but Cython makes for a good extended language for it, and it’s something I’m familiar enough with — including for something fairly similar with the SGIO implementation.

It was yet another interesting exercise in Yak Shaving though. Beside the documentation being obtuse at times, and trying to explain the interfaces in their chronological order, with the most useful once last, I found myself partially stumped when I realised that the ioctl() constants you have to use to get any useful information are not available on any userspace header of the Linux kernel! Indeed, it seems the main implementation of usbmon, as part of libpcap, just copies enough of the structures to be able to read the information — and, by the way, does not actually follow the documented process: it sets a value for the buffer size, rather than getting the one that is already set.

I’ve now engaged to make sure that the structures and constants are available to userspace, because at the very least that needs to be addressed properly. I’ve also added unrolled constants for the two ioctl calls that are needed to set the capture up, which keep the amount of copy-paste from kernel headers to a minimum.

While I have committed a monitoring tool that allows printing the output of packets, this is far from the end. It only outputs text format right now, it doesn’t do URB re-tagging, and it only does naïve filters. My next few steps will likely involve getting python-pcapng write support merged in, and start writing pcapng file with the new tool. Then I can start looking at a more common, more interesting filtering set.

Once the capturing is properly taken care of, I have two main needs that I need to address, in the toolset: one is to be able to unpack PL2303 serial protocols — because the programmer that is failing me is using PL2303 and I would like to see how the conversation with the bootloader is going. While the stcgal tool has debug output, having a general chatter printer feels like it would be useful in the future. The other is USB Mass Storage parsing and inspecting, because I need it for the beurer, but also because I would like to turn some of my past reverse engineering blog posts into a talk, and I would like to have some more examples of how the tools make it easier to find the meat of the information.

So yeah that’s where a Sunday went for me…

Book Review: Learning Cython Programming

Thanks to PacktPub I got a chance to read an interesting book this week: Learning Cython Programming by Philip Herron (Amazon, Packt). I was curious because, as you probably noticed, after starting at my current place of work the past April, I ended up having to learn to use Python, which ended up with me writing my glucometer utilities in that language, contrarily to most of my other work, which has been done in Ruby. But that’s a topic for a different post.

First of all, I was curious about Cython; I heard the name before but never looked much into it, and when I saw the book’s title and I quickly checked what it was, my interest was definitely picked. If you haven’t looked into it either, at a quick summary it’s a code generator bridge between Python and good plain old C, wrapping the latter such that you can either make it run Python callbacks, or generate a shared object module that Python can load, and offload the computation-intensive code to a more performant language. And it looks a lot like a well-designed and well-implemented version of what I hoped to get in Ruby with Rust — no connection with Mozilla’s language with the same name.

The book is a quick starter, short and to the point, which is an approach I like. Together with the downloadable source code, it makes it a very good solution to learn Cython, and I recommend it if you’re interested. Not only it covers the obvious language itself, but it covers a wide range of use cases that show how to make good use of the options provided by Cython. It even goes on to show how to integrate it in a build system (although I have some reserves on the Autotools code in there, which I think I’ll send Philip a correction for).

I seriously wish I had Cython and this book when I was working on Hypnos, an Ultima OnLine «server emulator» for which I wanted to add Python-based scripting — other emulators at the time used either a very simple, almost basic-like scripting language, Perl or C#. This was before I tried to use Python for real, which turned me to hate its whitespace-based indentation. I did write some support for it but it was a long and tedious process, so I never finished it. Not only Cython would make that work much less tedious, but the book shows exactly how to add Python scripting capabilities to a bigger, C program using tmux as the example.

The book does not ignore the shortcomings of Cython of course, including the (quite clumsy) handling of exceptions when crossing the foreign language barrier. While there are still a bunch of issues to be straightened out, I think the book is fairly good at setting proper expectation for Cython. If you’re interested in the language, the book looks like the perfect fit.