It’s a rare situation in a small networking environment that having two subnetworks on one broadcast domain can be an issue. I would normally avoid such a scenario (and it’s usually easy to do so) but I recently got AT&T’s U-verse, and the do-it-all device that it requires (a 2Wire 3800HGV-B “residential gateway”) has forced me to put both my private (NAT’d) subnetwork on the same broadcast domain as my public (DMZ’d) subnetwork. While undesirable, this isn’t usually a problem, except that my dual-homed Linux box had trouble behaving with the 2Wire gateway.
The Two-Interface Linux Box
As you can see in the diagram above, there’s a Linux box on this network that has two network interfaces, one for the Internet (let’s say its MAC address is 00:aa:bb:cc:22 and its IP is “a.b.c.2”), and one for the intranet (00:10:00:00:22 and 10.0.0.2). Despite that both interfaces are on the same broadcast domain, there are still benefits to having two interfaces. Among them are: most network traffic is still segmented (private traffic hits only private switches), I only need to use gigabit hardware where it matters (private network segments), and I get two ways to log into the machine (if there’s a problem with one interface, I still have another).
Here’s the problem with the Linux box: it, by default, responds to Address Resolution Protocol (ARP) requests for an address of either of its interfaces on both interfaces, each with its own MAC address. That is, if someone else on the network asks “Who has (What is the MAC address of) IP address 10.0.0.2?” (an ARP request), and both interfaces receive this request, the Linux box’s interfaces will respond (ARP responses):
- from the “private” interface: “IP 10.0.0.2 has MAC address 00:10:00:00:22” (this is correct)
- from the “public” interface: “IP 10.0.0.2 has MAC address 00:aa:bb:cc:22” (this is not correct)
In most cases, this is OK. The other computers on the network that want to send a packet to 10.0.0.2 will end up sending it to either interface (perhaps the wrong one), but this is okay because Linux doesn’t care that it got a packet for 10.0.0.2 on its a.b.c.2 interface – the packet still gets delivered to the software on the machine that expected it.
This doesn’t Fly with the 3800HGV-B
So here’s where Linux’s behavior created a problem. The Residential Gateway’s routing software keeps track of which MAC address owns an IP so that it knows where to send a packet destined for that IP.
There’s a subtle quirk in this gateway’s routing software that distinguishes it from how the other systems on the network behaves. Its table maps a MAC addresses to an IP, and both MAC addresses and IPs in the table must be unique. Linux, Windows, and most other devices, on the other hand map IPs to MAC addresses, where only IPs must be unique (thus allowing multiple IPs to point to one MAC address).
Let’s take an example where an ARP request is first made for ‘a.b.c.2’, then another is made for ‘10.0.0.2’. Linux will produce four responses (two for each request out of each interface). The four responses could arrive in this order:
- a.b.c.2 → 00:10:00:00:22 (incorrect)
- a.b.c.2 → 00:aa:bb:cc:22 (correct)
- 10.0.0.2 → 00:10:00:00:22 (correct)
- 10.0.0.2 → 00:aa:bb:cc:22 (incorrect)
After all four responses, the Windows machine’s Internet-to-MAC address table would look like this:
- a.b.c.2 → 00:aa:bb:cc:22 (the latest response about a.b.c.2 — correct)
- 10.0.0.2 → 00:aa:bb:cc:22 (the latest response about 10.0.0.2 — incorrect)
This may go unnoticed because the Windows machine will send a packet bound for either a.b.c.2 or 10.0.0.2 to 00:aa:bb:cc:22, and Linux will accept it on either interface and send it to the listening software — although “incorrectly”, it works just fine.
Between the 3rd and 4th response, the Residential Gateway’s table would be:
- a.b.c.2 → 00:aa:bb:cc:22 (the latest response about a.b.c.2 — correct)
- 10.0.0.2 → 00:10:00:00:22 (the latest response about 10.0.0.2 — correct)
which is accurate until a moment later when the fourth ARP response is received that says “IP 10.0.0.2 has MAC address 00:aa:bb:cc:22,” at which point the table looks like:
- 10.0.0.2 → 00:aa:bb:cc:22 (the latest response about 10.0.02 — incorrect)
The key here is that now there’s only one entry (which happens to be wrong) because MAC addresses must be unique in the Residential Gateway. Now the routing software in the gateway doesn’t think that there’s a device on the network that has the IP address ‘a.b.c.2’ and any packets destined to it are discarded. This causes a very strange problem. If the table happened to contain ‘a.b.c.2’, I would be able to establish a connection with that machine from the Internet. However, when an ARP response hit the wire that eliminated ‘a.b.c.2’ from the table (usually within 5-60 seconds), I would lose my connection because the gateway would simply stop routing my packets to that address.
ARP Flux
The gateway is a victim of what the “Guide to IP Layer Network Administration with Linux” calls ARP flux, “confusion, or worse yet, non-deterministic population of the ARP cache … can lead to the possibly puzzling effect that an IP migrates non-deterministically through multiple link layer addresses.”
Solution: arp_filter
The solution in this situation is to enable arp_filter on the Linux box, a sysctl that prevents interfaces from responding to ARP requests that aren’t for an IP address assigned to that interface.
This can be done easily-but-temporarily (until the next reboot) by running the command:
sysctl net.ipv4.conf.eth[num].arp_filter=1
for each interface, where [num] is the number of the interface.
For this to have a permanent effect, edit /etc/sysctl.conf, adding a line for each interface: “net.ipv4.conf.eth[num].arp_filter = 1“, where [num] is the number of the interface. For example, these lines are in my /etc/sysctl.conf:
# interfaces shouldn't respond to ARPs that aren't theirsnet.ipv4.conf.eth0.arp_filter = 1
net.ipv4.conf.eth1.arp_filter = 1
An annoying problem caused by a lame default
Linux… what are you doing responding to ARPs on the wrong interface with the wrong MAC address? The configuration paradigm communicated by the configuration utilities (whether via ifconfig or system-config-network) is: “10.0.0.2 is assigned to eth0 and a.b.c.1 is assigned to eth1.” None of the information gives me any reason to believe that not only will eth0 take packets intended for eth1’s address, but that it will actively tell others that it will do so (ARP responses). This behavior should not be the default — Windows and MacOSX/BSD, who don’t do this, seem to agree.
Leave a Reply