BGP Multihop.

External BGP (eBGP) Multihop Support

Connections between BGP speakers of different ASs are referred to as External BGP (eBGP) connections. BGP enforces the rule that peer routers for eBGP connections need to be on a directly attached network. If the peer routers are multiple hops away from each other or if multiple links are between them, you can override this restriction by enabling the eBGP multihop feature. TCP connections between eBGP peers are tied to the addresses of the outgoing interfaces. Therefore, a single interface failure severs the session even if a viable path exists between the peers.

eBGP multihop support can provide redundancy so that an eBGP peer session persists even in the event of an interface failure. Using an address assigned to the loopback interface for the eBGP peering session ensures that the TCP connection stays up even if one of the links between them is down, provided the peer loopback address is reachable. Let’s examine this behavior by using this lab topology…

Let’s assume here that we want to setup a eBGP peering between router1 and router4.? To make matters a little more interesting, let’s also use the loopback interfaces on each router as the update-source for the BGP session.? I’ll be using OSPF as the IGP to get reachability between all of the networks.? The BGP config on router1 and router4 will then look like this…

Router1

Router4

Taking a look at the BGP summary, we see that both routers are in the IDLE mode…

Let’s take a quick look at the BGP packets being sent from router1 and examine their TTL…

A capture on the router1’s interface facing router2 curiously doesn’t show any BGP traffic at all.? This is rather strange.? I would have expected to see BGP packets being sent towards 172.64.32.2 with the default TTL of 1.? However, it appears that this isn’t occurring.? A quick look at the ‘show ip bgp neighbors’ commands yields this output…

Interesting.? So the router knows that the neighbor isn’t directly connected.? Apparently this means that the router knows it’s hop count is 1 and that since it isn’t directly connected it doesn’t even try to establish a connection.? We can tell the router to still attempt the the connection by issuing this command under the BGP configuration…

Now if we take a look at our capture, we should see some BGP packets…

If you look at the above output closely we can see that router1 is attempting to establish a TCP connection on port 179 with router4.? We can also see that router2 (10.0.0.13) is replying back to router1 informing it that the TTL has expired in transit.? An up close examination of the TCP packet shows the TTL of the packet to be 1 which is what we’d expect to see…

Let’s go ahead and re-enable the directly connected neighbor check on router1 and instead change the eBGP multihop setting…

A second look at the capture shows the TCP packets being generated with a TTL of a 2 and the ICMP TTL expired in transit packets being generated by router3 (10.0.0.10)…

Let’s take a look at the diagram again to make sure we understand where we’re at…

We are now telling router1 to attempt it’s connection to router4 so long as it is less than or equal to 2 hops away.? We saw the TCP SYN leave router1 destined for router4 with a TTL of 2.? Router2 receives the packet, processes it, decrements the TTL by 1, and sends it on its way to router3 with a TTL of 1.? Router 3 process the packet and realizes that it cant send a packet with a TTL of 0 (after it decrements it), and replies with a ICMP TTL expired in transit.?

Side Note:? This is probably a good time to discuss how Cisco routers handle TTL in general.? From my experience, Cisco network devices process TTL in this manner.? If the packet is destined for the router (control plane etc.) a packet coming across the wire to the device with a TTL of 1 is fine.? The router will decrement the TTL of the packet on interface ingress, and then pass the packet up to the control plane.? This implies that a TTL of 0 is fine on a local device EVEN if you are headed to a loopback interface on the router.? However, if the packet is not destined for the router the router will decrement the TTL, do a forwarding lookup, realize the packet is not destined for itself, and will then drop the packet and generate a ICMP TTL expired message back to the sender.

Now let’s change the eBGP multihop setting to 3 on router1 and see what happens…

We can see that router4 is sending a TCP reset back to router1…

Much like how router1 didn’t want to play ball when it knew it’s neighbor was directly connected, router4 is in the same boat.? It’s completely refusing the TCP session.? If we were to enable the ‘disable-connected-check’ on router4 we’d see that router4’s replies to router1 would be dropped on router3 since they only have a TTL of 1.? Let’s confirm that theory…

Just as we thought, lets look at the packets in more detail…

Here’s router1 reaching router4.? By this time the TTL of the packet is 1…

Here’s the reply from router4 back to router1.? Notice that the TTL of this reply is also 1…

And finally, here’s router3 telling router4 that the TTL expired in transit…

So now let’s turn the eBGP multihop on router4 to three and see what happens…

And the session is now up!? Now let’s look at some quick verification commands to determine how eBGP multihop is configured…

The allowed hop count is recorded in the ‘show ip bgp neighbor’ command.? Note that when you are using the default hop count of 1, this command won’t display anything.? The full output of the ‘show ip bgp neighbor’ command will show whether or not the neighbor is directly connected.

So now that we beat the snot out of eBGP multihop, let’s talk about the other option which is TTL-Security.? When I first read about TTL-Security, I was rather confused.? Not by the concept of it, but why I would want to use it rather than eBGP multihop.? It should be noted here that the two features are mutually exclusive.? AKA, you can only use one of them at a time.? So let’s start with the definition of TTL-Security from Cisco…

The TTL security mechanism (GTSM) relies on a simple concept to protect BGP infrastructure from spoofed IP packets. It recognizes the fact that the vast majority of EBGP sessions are established between directly-connected routers and therefore the IP TTL values in packets belonging to these sessions should have predictable values. If an incoming packet does not have the expected IP TTL value it is possible that it is coming from an unauthorized and potentially harmful source.

TTL security is enabled using the ttl-security command. This command requires a minimum TTL value to be specified. When TTL security is enabled on a BGP session the IP TTL values in packets that are supposedly coming from the peer are compared (in hardware) to the configured minimum value and if there is a discrepancy the packet is discarded and a log is generated. TTL security is used most often on single-hop EBGP sessions but it can be used on multihop EBGP and IBGP sessions as well.

This feature protects the eBGP peering session by comparing the value in the TTL field of received IP packets against a hop count that is configured locally for each eBGP peering session. If the value in the TTL field of the incoming IP packet is greater than or equal to the locally configured value, the IP packet is accepted and processed normally. If the TTL value in the IP packet is less than the locally configured value, the packet is silently discarded and no ICMP message is generated. This is designed behavior; a response to a forged packet is unnecessary

Sounds quite a bit like eBGP multihop in some ways doesn’t it?? Well, the key to understanding TTL-Security (at least for me) was these two sentences…

eBGP multihop configures the maximum number of hops in which a eBGP speaker can use to reach a eBGP peer.? TTL-Security assumes the default TTL of 255 is being used and ensures that the TTL of the received packet is greater than or equal to the minimum TLL (255 minus configured hop count).

Confused?? Let’s run through a quick example so you can see what I mean.? I’m going to remove the eBGP multihop command from router1 and router4 and reconfigure them with the TTL-Security command…

So let’s check out what we see on the wire…

Here we can see the packet from router4 headed to router1.? Check out the TTL of the packet…

Now check out the TTL of the packet coming from router1 headed to router4 as it is seen on the wire between router3 and router4…

As you can see, the initial packets being generated by each router are starting at 255.? The packet sent by router1 reaches a TTL of 253 by the time it hits the wire between router3 and router4.? So what’s going on here?? The TTL is more than sufficient for the TCP packets to arrive at each router.? Why isn’t the session coming up?

TTL-Security takes the number you supply in the ‘ttl-security hop’ command and uses it to determine a minimum incoming TTL.? The maximum TTL is assumed on Cisco devices to be 255.? That being said, specifying a TTL-security value of 2 means my TTL minimum is 253 and the TTL maximum is 255.? When router 1 sends the packet to router 4 we saw the packet leaving router3 headed to router4 with a TTL of 253.?

So if the value of the incoming packet was 253 why doesn’t the connection work when the minimum TTL is 253?? The bottom line is that Cisco router decrement the TTL of a packet on ingress to an interface (see side note above).? The instant that packet is received by the router, the TTL is decremented before it can reach the control plane.?

Making sense?? Let’s have some picture to clarify the point…

As we can see above, the TTL starts off with 255 and then decrements.? The red lines show the TTL ‘on the wire’.? That is, the TTL generated by the router that pushes the frame onto that segment.? So while the TTL of the packet between router3 and router4 is 253, router4 still has to do it’s job and decrement the TTL to 252.? The general concept is the same between eBGP multihop and TTL-Security, but the purpose of each is rather different.? Take these two examples for instance…

Consider we have a rogue router that’s many hops away.? The rogue router and router1 both appear to be within the correct distance for router4 to accept the session.? Granted, router4’s reply to the rogue router won’t work since he likely has en eBGP multihop setting of 3 (like router1 in this example) but router4 will still accept the initial TCP SYN for the BGP session.? If we change this to TTL-Security we get something that looks like this…

Considering we had TTL-Security configured for 3 hops on both router1 and router4, the rogue router wouldn’t be able to create a session to router4 because it’s arriving TTL is not in between the minimum(252) and maximum(255) range.? However, the connection from router1 would work just fine since it’s arriving TTL at router4 is 253 (252 once router4 processes it).? Once configured we’d see something like this on router4…

Why Use BGP Multi-Hop?

1. Peering Over Non-Direct Connections

• Network Design Flexibility: Sometimes, it isn't possible or desirable to have a direct link between BGP peers, especially in complex network topologies or when dealing with peering across multiple ASes. BGP multi-hop allows for greater flexibility by enabling these non-direct peerings.
• Logical Separation: Multi-hop BGP is often used in scenarios where physical links are logically separated. For example, BGP multi-hop can be employed in route reflector setups where clients do not need to be directly connected to the route reflector itself.

2. eBGP in Data Centers

• Connecting Data Centers: Data centers often employ BGP multi-hop to establish eBGP sessions across different locations, enabling data centers to function as a single, seamless network entity. This approach is useful for large enterprises or service providers managing multiple data centers.

3. Redundancy and Failover

• Enhanced Redundancy: BGP multi-hop allows the creation of redundant paths between BGP peers across multiple intermediate routers. This redundancy is crucial in ensuring high availability and failover capabilities.

4. Connecting to Remote ISPs

• Accessing Upstream Providers: Sometimes, an organization may need to peer with a remote ISP or upstream provider that isn't directly connected. In such cases, BGP multi-hop allows the establishment of this peering relationship, enabling the organization to access global routing tables.

Security Considerations

While BGP multi-hop is a powerful tool, it does open up potential security concerns. With an increased TTL, packets can traverse multiple routers, potentially exposing the BGP session to attacks such as spoofing or session hijacking. To mitigate this risk, it's advisable to:

1. Use MD5 Authentication: Secure BGP sessions with MD5 to ensure that peers are authenticated.
2. Limit TTL: Set the TTL to the minimal required value to reduce the attack surface.
3. Implement IP Filtering: Filter incoming and outgoing BGP traffic to only allow traffic from trusted peers.

Sources

https://orhanergun.net/ebgp-multihop

https://networklessons.com/bgp/ebgp-multihop

https://www.dasblinkenlichten.com/ebgp-multihop-vs-ttl-security/

https://sc1.checkpoint.com/documents/R80.40/WebAdminGuides/EN/CP_R80.40_Gaia_Advanced_Routing_AdminGuide/Topics-GARG/eBGP-Multihop.htm

https://ipcisco.com/lesson/bgp-multi-hop-command/