Dual-WAN/ISP Router on Hardened Gentoo Linux - Part 2: Marking Traffic by Interface

We will quickly highlight 3 primary dual-WAN dual-ISP routing challenges, and configure Linux, and Netfilter (iptable/ip6tables) to address 2 of the challenges. This will build a solid foundational framework for Advanced Linux Policy-Based Routing that will enable us to address the 3rd challenge in Part 3 of this series.

Linux does not have an easy-button that will enable dual-WAN functionality. Various seemingly unrelated settings and configurations are built up one upon the other to facilitate dual-WAN functionality. There are many different possible options and configurations one may select when building their perfect dual-WAN solution. The solution described herein is likely the easiest for most people because they may already be familiar with some of the command-lines and concepts.

The primary Linux Kernel features we will utilize for this solution are,

• Linux Advanced Routing and Traffic Control;
• IP: advanced router;
• IP: policy routing;
• Advanced netfilter configuration;
• Connection mark tracking support;
• Connection tracking;
• "MARK" target support; and
• Packet mangling.

Dual Internet WAN Inbound Routing Challenges

In a normal Linux system traffic without an explicit route will be routed via the Default Gateway. This normally is exactly what we want because internal networks have explicit routes, and every other network is likely reachable via the Internet connection. This is why the Internet connection is set as the Default Gateway. I.e., traffic to internally unknown IP addresses/Networks are likely on the Internet.

There are 3 primary challenges precipitated when there are 2 Internet WAN connections:

1. Inbound traffic to the backup Internet connection (eth1) will have response traffic routed via the primary Internet connection (eth0) - the Default Gateway. This will not make any sense to the TCP/IP stack of the receiving user, and a connection will never be established.
2. Linux, by default, will view the backup Internet connection (eth1) only as a directly connected network and not route Internet traffic over the connection.
3. Which Internet connection should Internal-to-Internet traffic be routed via?

This series will address all 3 of these challenges and more.

Linux Kernel Solution Overview

We will use Netfilter, and Linux's Advanced Policy Based Routing features to build 4 important mechanisms:

1. Identify which interface inbound traffic was received on.
2. Store a number that represents the inbound interface the traffic was received on within Connection Tracking's "FW-mark". - I'm keeping this simple here.
3. Set the FW-mark of outbound traffic to the FW-mark's value associated with that traffic as stored within Connection Tracking.
4. Configure Linux's Advanced Policy Based Routing rules to interrogate the FW-mark and make appropriate routing decisions.

We will address the first 3 items in this Part; however, without all 4 of these items in place all outbound Internet traffic will be routed out the primary Internet connection's interface as it is the Default Gateway. While this configuration may not strictly be necessarily for a primary and only fail-over configuration the extra utility of being able to use both Internet connections for inbound traffic is highly beneficial especially if one VPNs in to access files, runs servers, conducts testing, etc. etc. Also, it's a good cliffhanger to entice people to read Part 3.

Using FW-Mark to Identify the Network Interface

TCP/IP is a higher-level protocol that works with IP addresses/networks, and is generally unaware of the lower-level protocols and interfaces. As such we need to augment the information Linux keeps about a TCP/IP connection so to include additional information about which network interface to use. Linux has a generic facility which is referred to as FW-mark ("firewall mark"). This facility allows for the storage of an arbitrary number (i.e., the mark), that exists only within Linux's TCP/IP Connection Tracking facility (I'm simplifying here), and has support for bit-masks.

There are several tools that one may use to manipulate the FW-mark. The tool most people would be familiar with is Linux's firewall, Netfilter (iptables/ip6tables). Netfilter also has an advantage here because it also provides easy access to Linux's Connection Tracking facilities. We will need both of these features.

Augmenting Connection Tracking with FW-Mark

We will use the FW-mark facility to store a number that represents the eth0 network interface, and another different number that will represent the eth1 network interface. In a simple setup, these numbers maybe arbitrary chosen:

• 10 is for eth0 - this isn't really needed as eth0 is the Default Gateway.
• 20 is for eth1

Pro-tip: The above are simple values that are not suitable for complex situations as they are not bit-aligned, but will work perfect for our needs.

Using Netfilter, we will mark inbound connections from eth0 with a FW-mark "10", and inbound connections from eth1 with a different FW-mark of "20". Remember to use both iptables and ip6tables as appropriate for your environment.

• iptables -t mangle -A PREROUTING -i eth0 -j MARK --set-xmark 10
• iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-xmark 20

Pro-tip: should one already be using FW-mark for something else it is recommended to use an appropriate value with an appropriate bit-mask.

Configure Netfilter to save the FW-mark within the Connection Tracking facility.

• iptables -t mangle -A PREROUTING -i eth0 -j CONNMARK --save-mark
• iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --save-mark

Restore/Set the FW-mark on outbound traffic to the FW-mark's value associated with that traffic as stored within Connection Tracking.

• iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

Pro-tip: should one already be using FW-mark for something else use an appropriate bit-mask value to identify the interface when restoring the FW-mark.

Enabling FW-mark Processing in the Linux Kernel

These often forgotten Linux Kernel settings should be set as indicated when using FW-mark.

Configure the Linux Kernel to set an FW-mark for kernel-generated reply packets.

• sysctl net.ipv4.fwmark_reflect=1
• sysctl net.ipv6.fwmark_reflect=1

Configure the Linux Kernel to consider the FW-mark when performing a reverse path route lookup.

• sysctl net.ipv4.conf.eth0.src_valid_mark=1
• sysctl net.ipv4.conf.eth1.src_valid_mark=1

Pro-tip: this is a per-interface IPv4 setting.

Summary and Part 3 Forward

We quickly highlighted 3 primary dual-WAN dual-ISP routing challenges, and configured Linux, and Netfilter (iptable/ip6tables) to address 2 of the challenges. This will build a solid foundational framework for Advanced Linux Policy-Based Routing that will enable us to address the 3rd challenge in Part 3 of this series.