The Two-Interface Linux Box

As you can see in the diagram above, there’s a Linux box on this network that has two network interfaces, one for the Internet (let’s say its MAC address is 00:aa:bb:cc:22 and its IP is “a.b.c.2″), and one for the intranet (00:10:00:00:22 and 10.0.0.2). Despite that both interfaces are on the same broadcast domain, there are still benefits to having two interfaces. Among them are: most network traffic is still segmented (private traffic hits only private switches), I only need to use gigabit hardware where it matters (private network segments), and I get two ways to log into the machine (if there’s a problem with one interface, I still have another).

Here’s the problem with the Linux box: it, by default, responds to Address Resolution Protocol (ARP) requests for an address of either of its interfaces on both interfaces, each with its own MAC address. That is, if someone else on the network asks “Who has (What is the MAC address of) IP address 10.0.0.2?” (an ARP request), and both interfaces receive this request, the Linux box’s interfaces will respond (ARP responses):

• from the “private” interface: “IP 10.0.0.2 has MAC address 00:10:00:00:22″ (this is correct)
• from the “public” interface: “IP 10.0.0.2 has MAC address 00:aa:bb:cc:22″ (this is not correct)

In most cases, this is OK. The other computers on the network that want to send a packet to 10.0.0.2 will end up sending it to either interface (perhaps the wrong one), but this is okay because Linux doesn’t care that it got a packet for 10.0.0.2 on its a.b.c.2 interface – the packet still gets delivered to the software on the machine that expected it.

This doesn’t Fly with the 3800HGV-B

So here’s where Linux’s behavior created a problem. The Residential Gateway’s routing software keeps track of which MAC address owns an IP so that it knows where to send a packet destined for that IP.
There’s a subtle quirk in this gateway’s routing software that distinguishes it from how the other systems on the network behaves. Its table maps a MAC addresses to an IP, and both MAC addresses and IPs in the table must be unique. Linux, Windows, and most other devices, on the other hand map IPs to MAC addresses, where only IPs must be unique (thus allowing multiple IPs to point to one MAC address).

Let’s take an example where an ARP request is first made for ‘a.b.c.2′, then another is made for ’10.0.0.2′. Linux will produce four responses (two for each request out of each interface). The four responses could arrive in this order:

• a.b.c.2 → 00:10:00:00:22 (incorrect)
• a.b.c.2 → 00:aa:bb:cc:22 (correct)
• 10.0.0.2 → 00:10:00:00:22 (correct)
• 10.0.0.2 → 00:aa:bb:cc:22 (incorrect)

After all four responses, the Windows machine’s Internet-to-MAC address table would look like this:

• a.b.c.2 → 00:aa:bb:cc:22 (the latest response about a.b.c.2 — correct)
• 10.0.0.2 → 00:aa:bb:cc:22 (the latest response about 10.0.0.2 — incorrect)

This may go unnoticed because the Windows machine will send a packet bound for either a.b.c.2 or 10.0.0.2 to 00:aa:bb:cc:22, and Linux will accept it on either interface and send it to the listening software — although “incorrectly”, it works just fine.

Between the 3rd and 4th response, the Residential Gateway’s table would be:

• a.b.c.2 → 00:aa:bb:cc:22 (the latest response about a.b.c.2 — correct)
• 10.0.0.2 → 00:10:00:00:22 (the latest response about 10.0.0.2 — correct)

which is accurate until a moment later when the fourth ARP response is received that says “IP 10.0.0.2 has MAC address 00:aa:bb:cc:22,” at which point the table looks like:

• 10.0.0.2 → 00:aa:bb:cc:22 (the latest response about 10.0.02 — incorrect)

The key here is that now there’s only one entry (which happens to be wrong) because MAC addresses must be unique in the Residential Gateway. Now the routing software in the gateway doesn’t think that there’s a device on the network that has the IP address ‘a.b.c.2’ and any packets destined to it are discarded. This causes a very strange problem. If the table happened to contain ‘a.b.c.2’, I would be able to establish a connection with that machine from the Internet. However, when an ARP response hit the wire that eliminated ‘a.b.c.2’ from the table (usually within 5-60 seconds), I would lose my connection because the gateway would simply stop routing my packets to that address.

ARP Flux

The gateway is a victim of what the “Guide to IP Layer Network Administration with Linux” calls ARP flux, “confusion, or worse yet, non-deterministic population of the ARP cache … can lead to the possibly puzzling effect that an IP migrates non-deterministically through multiple link layer addresses.”

Solution: arp_filter

The solution in this situation is to enable arp_filter on the Linux box, a sysctl that prevents interfaces from responding to ARP requests that aren’t for an IP address assigned to that interface.
This can be done easily-but-temporarily (until the next reboot) by running the command:


sysctl net.ipv4.conf.eth[num].arp_filter=1


for each interface, where [num] is the number of the interface.

For this to have a permanent effect, edit /etc/sysctl.conf, adding a line for each interface: “net.ipv4.conf.eth[num].arp_filter = 1“, where [num] is the number of the interface. For example, these lines are in my /etc/sysctl.conf:


# interfaces shouldn't respond to ARPs that aren't theirs

net.ipv4.conf.eth0.arp_filter = 1

net.ipv4.conf.eth1.arp_filter = 1

An annoying problem caused by a lame default

Linux… what are you doing responding to ARPs on the wrong interface with the wrong MAC address? The configuration paradigm communicated by the configuration utilities (whether via ifconfig or system-config-network) is: “10.0.0.2 is assigned to eth0 and a.b.c.1 is assigned to eth1.” None of the information gives me any reason to believe that not only will eth0 take packets intended for eth1’s address, but that it will actively tell others that it will do so (ARP responses). This behavior should not be the default — Windows and MacOSX/BSD, who don’t do this, seem to agree.

What made this setup interesting to me (from a networking perspective) was that it forced me to compromise on keeping my private network separate from my public network…

Typical Network

This first diagram, below, represents how I would typically set up small network with public IPs.

Just as most people do, I have my workstations behind a typical residential gateway (viz., a “wireless router”). What’s just a bit atypical is that I have two servers that each have two network interfaces and participate in both the Internet and intranet networks.

Because all devices that participate in both networks are smart enough (operate on the third network layer or above) a packet never makes it from one network to the other unless it needs to, which is pretty much only when a workstation is communicating with the Internet (through the residential gateway).

Thus, the Internet subnetwork and intranet subnetwork are on different broadcast domains. Anything broadcast onto the intranet (whether by a workstation or server), such as an ARP request, will only be heard by the other devices on the intranet.

Two Networks on One LAN

Things got a bit more complicated when I got AT&T’s U-verse. The key difference that this Residential Gateway introduced is that it joined my two networks into one broadcast domain. This is because the Gateway’s internal network interface has an intranet IP address, 10.0.0.1/24, so that it can send IPTV to the private network TV set-top-boxes that are hooked up via Ethernet, and the Gateway is routing the public IP addresses (“a.b.c.x”) for the servers, so the Gateway’s internal network interface also has an IP address of a.b.c.7.

Both the public and intranet networks are connected to the 2Wire 3800HGV-B's built-in switch

Topology with the 2Wire 3800HGV-B with public/static IPs

So, my 3800HGV-B Gateway is participating on both networks on one interface (technically, on two ports of its built-in layer-2 switch). Now, all devices, whether public or private, are able to communicate with each other. They usually ignore each other, but they will all hear broadcasts from any device. For example, if Workstation1 sends out a broadcast packet (such as an ARP request), it will reach the Linux server on both its Internet and intranet interfaces (because switches, including the Gateway’s built-in switch, will propagate it out to all attached networks. This is not usually a problem, except that the Linux server was confusing the Gateway.

Network summary for the overly curious

(actual IPs partially masked)

• AT&T U-verse’s local router (my Gateway’s gateway) is: 99.27.aa.bb
• My Gateway’s external interface IP is: 99.27.aa.cc (intranet devices NAT’d traffic appears to come from this IP)
• My Gateway’s internal interface IP for intranet: 10.0.0.1 (workstation and other devices have this as their default gateway)
• My Gateway’s internal interface IP for Internet servers: 99.65.dd.ee (servers with public/static IPs have this as their default gateway)
• Server IPs look like: 99.65.dd.ff/29
• workstation and other device IPs look like 10.0.0.gg/24