The routed network broadcast problem

You might remember my network diagram that has shown you the absurd setup I have at home to connec tall the rooms where computers are located. Since then, something was reduced, and indeed now the network section between my bedroom and the office is over the usual Ethernet (should be Gigabit, but something doesn’t look right) cable. This actually should also reduce the power consume at home since the old Powerline adaptors were still an extra powered appliance; the main reason why I replaced the, though, was that the green LEDs definitely bothered me while trying to sleep, and at the same time, speed was quite an issue with some files’ streaming.

The result is that only two media are used here: WiFi and cabled Ethernet; unfortunately, I still lack a way to connect Yamato and Deep Space 9 (the router) via Ethernet directly, so they are connected via a standard infrastructure WiFi. This is not really exceptional, in the sense that the connection between them is not very stable (I use an ath9k card on Yamato, with 2.6.32rc7 kernel), and when I’m downloading stuff with bittorrent or similar, I need to restart the network connections about once every five minutes to keep it going properly, which you can guess is not that fun.

Now unfortunately there is one problem here, which I ignored for quite a while but I cannot ignore any longer (because I finally got the table I needed to play with Unreal Tournament 3 with my PlayStation 3!): the cabled ethernet segments fail to get UPnP support.

The whole network inside a single Class B IP range (172.28.0.0/16), fractioned into four main subnetworks (direct, and behind Yamato, known and unknown computers, they have different filters on the firewall) by the DHCP server running on Deep Space 9 (for simplicity, Yamato is the only box in the network to have a static IP address, in an unused subnetwork range together with Deep Space 9, beside the router/DHCP server itself). Yamato has two interfaces enabled: wlan0 which connects to the AP and then to DS9, and br0 which is the bridge of the remaining interfaces (eth0 and eth1 for the cabled network segments – the latter I only bring up when I need more ports for work devices – and vde0 for the virtual networks). Here start the problem: while a WiFi network is usually akin to a switched network, and of course my cabled segment is also switched, the two together are not switched but routed together, by Yamato which is a second router in the network.

Of course I built DS9 to reduce the load of Yamato (even though my original planning involved linking the cabled with that through a another, very long, cable), so the services are currently mostly running on DS9 rather than Yamato: DHCP server, DNS server, UPnP server and so on. The problem is that almost all the “zeroconf” kind of services, which include not only Apple’s Bonjour protocol, but UPnP and DHCP as well use the UDP transport and the broadcast address to look for the servers. And UDP broadcast only works within switched networks, not routed ones.

The obvious solution in these cases, which is more or less the only solution you’ll ever read proposed around when people ask about broadcast repeaters, is to use bridging instead of routing to merge the two networks together; a switch is, after all, just a multi-port bridge, so the result is again a switched network. Unfortunately this brings two issues with it: the first is that you effectively lose the boundary between the two networks, even when that was very transparent, like I’d like it to be, the filtering can still be useful for some things; the latter is that bridging WLAN interfaces is complex and pretty much suboptimal.

The problem with bridging WLAN is that putting the network card in promiscuous mode is not enough: the access point by default only sends over the air the PDUs whose destination is an associated mac address. And telling the access point to send all the PDUs might not be good either; while in my setup the problem is relatively small (the only two devices connected via Ethernet to DS9 are the AP and the Siemens VoIP phone — the Linux bridge software will still understand to only send the VoIP phone data to the connected network card and the rest to the AP), it doesn’t look like a very good long-term solution.

To solve part of the problem, at least the most common part of it, both ISC DHCP and Avahi provide support for transparently join two routed networks that would otherwise be isolated: dhcrelay and Avahi’s refector. The former is not just a simple repeater of DHCP requests, but it also adds a “circuit-id” to the requests, so that requests coming from behind it are tagged and can be treated differently (this is how I handle differently the clients behind Yamato — of course those have to get to a subnet that is routed through Yamato); the latter just picks up the service broadcasts and copy them to the various interfaces it listens on… but neither is perfect.

With dhcrelay the problem is deep inside the way it has been implemented: it has to listen on both the interface the requests will come from, and that where the responses come from… and it doesn’t discriminate between them; this means in the case of Yamato that I have to listen to both br0 and wlan0, but then the requests sent by the clients on WiFi will still reach the relay and would be sent back to DS9 through the relay; for this reason the “circuit-id” contains the interface the request came from, so I only check for that id to be br0 instead of just checking if it exists, before deciding how to divide the clients. The alternative is using iptables to filter the requests from the wlan0 interface, but let’s leave that for a moment.

The problem with Avahi seems more to be a bug, or rather an untested corner case; I have found no way to stop Linux from issuing link-local IPv6 addresses to the interfaces that result “up”; this unfortunately means that eth0, vde0 and br0 all have their IPv6 address… so the broadcasts coming from wlan0 are reflected on all three of them, and all the clients connected to the cabled (or virtual) segment will receive the broadcast twice. This wouldn’t be much of an issue if Apple’s compuers didn’t decide to rename themselves to “Whatever (2)” when they felt somebody else was using their hostname in the network. I should speak with Lennart about it but I haven’t had time to deal with that just yet.

There remains a third protocol there that I found no solution for yet: UPnP; with UPnP the problem is relatively easy: SSDP uses UDP broadcasts on port 1900 to find the router, before talking directly with it, so the only thing that I’d be needing is a repeater over that particular port. The best solution to me would have been using iptables directly, but since that’s not implemented for what I can see, I guess I’ll end up either writing my own UDP repeater, or look for something working, and properly written. If somebody has a clue about that, I’d be happy to hear the solutions.

Interestingly enough, UPnP during my analysis proven to be the only protocol I’m interested in that actually could be just re-broadcasted with a generic repeater; for DHCP, I need to discern proxied requests to assign them to properly routed subnetworks; for Bonjour, the port wouldn’t be free for a repeater since Avahi itself would be using it to begin with.

So bottom-line, I’d have three needs that somebody might want to help me with: get a better dhcrelay the current implementation sucks in more ways than a few, starting for the not being able to specify which is the input and which the output interface, or the lack of a configurable circuit-id string; fix the Avahi IPv6 reflector over bridged network, although I have no idea how (alternative: find a way to tell Linux/OpenRC not to issue a link-local IPv6 address to the interfaces); write a generic UDP broadcast repeater so that UPnP can work with a routed network — the last one is what I’ll probably work on tomorrow so I can get the PS3 to pass through the ports with DS9.