Azure ExpressRoute design

ExpressRoute enables us to connect on Premises to Azure services seamlessly. lets review some design decisions you will make before deploying an ExpressRoute circuit.

Workflow

The architecture consists of the following components.

• On-premises corporate network. A private local-area network running within an organization.
• ExpressRoute circuit. A layer 2 or layer 3 circuit supplied by the connectivity provider that joins the on-premises network with Azure through the edge routers. The circuit uses the hardware infrastructure managed by the connectivity provider.
• Local edge routers. Routers that connect the on-premises network to the circuit managed by the provider. Depending on how your connection is provisioned, you may need to provide the public IP addresses used by the routers.
• Microsoft edge routers. Two routers in an active-active highly available configuration. These routers enable a connectivity provider to connect their circuits directly to their datacenter. Depending on how your connection is provisioned, you may need to provide the public IP addresses used by the routers.
• Azure virtual networks (VNets). Each VNet resides in a single Azure region, and can host multiple application tiers. Application tiers can be segmented using subnets in each VNet.
• Azure public services. Azure services that can be used within a hybrid application. These services are also available over the Internet, but accessing them using an ExpressRoute circuit provides low latency and more predictable performance, because traffic does not go through the Internet.
• Microsoft 365 services. The publicly available Microsoft 365 applications and services provided by Microsoft. Connections are performed using?Microsoft peering, with addresses that are either owned by your organization or supplied by your connectivity provider. You can also connect directly to Microsoft CRM Online through Microsoft peering.
• Connectivity providers?(not shown). Companies that provide a connection either using layer 2 or layer 3 connectivity between your datacenter and an Azure datacenter.

Components

• Azure ExpressRoute. ExpressRoute lets you extend your on-premises networks into the Microsoft cloud over a private connection, with the help of a connectivity provider. With ExpressRoute, you can establish connections to Microsoft cloud services, such as Microsoft Azure and Microsoft 365.
• Azure Virtual Network. Azure Virtual Network (VNet) is the fundamental building block for your private network in Azure. VNet enables many types of Azure resources, such as Azure Virtual Machines (VMs), to securely communicate with each other, the internet, and on-premises networks.

Recommendations

The following recommendations apply for most scenarios. Follow these recommendations unless you have a specific requirement that overrides them.

Connectivity providers

Select a suitable ExpressRoute connectivity provider for your location.

ExpressRoute connectivity providers connect your datacenter to Microsoft in the following ways:

• Co-located at a cloud exchange. If you're co-located in a facility with a cloud exchange, you can order virtual cross-connections to Azure through the co-location provider's Ethernet exchange. Co-location providers can offer either layer 2 cross-connections, or managed layer 3 cross-connections between your infrastructure in the co-location facility and Azure.
• Point-to-point Ethernet connections. You can connect your on-premises datacenters/offices to Azure through point-to-point Ethernet links. Point-to-point Ethernet providers can offer layer 2 connections, or managed layer 3 connections between your site and Azure.
• Any-to-any (IPVPN) networks. You can integrate your wide area network (WAN) with Azure. Internet protocol virtual private network (IPVPN) providers (typically a multiprotocol label switching VPN) offer any-to-any connectivity between your branch offices and datacenters. Azure can be interconnected to your WAN to make it look just like any other branch office. WAN providers typically offer managed layer 3 connectivity.

ExpressRoute circuit

Ensure that your organization has met the?ExpressRoute prerequisite requirements?for connecting to Azure.

If you haven't already done so, add a subnet named?GatewaySubnet?to your Azure VNet and create an ExpressRoute virtual network gateway using the Azure VPN gateway service. For more information about this process, see?ExpressRoute workflows for circuit provisioning and circuit states.

Considerations

These considerations implement the pillars of the Azure Well-Architected Framework, which is a set of guiding tenets that can be used to improve the quality of a workload. For more information, see?Microsoft Azure Well-Architected Framework.

Scalability

ExpressRoute circuits provide a high bandwidth path between networks. Generally, the higher the bandwidth the greater the cost.

ExpressRoute offers two?pricing plans?to customers, a metered plan and an unlimited data plan. Charges vary according to circuit bandwidth. Available bandwidth will likely vary from provider to provider. Use the?Get-AzExpressRouteServiceProvider?cmdlet to see the providers available in your region and the bandwidths that they offer.

A single ExpressRoute circuit can support a certain number of peerings and VNet links. See?ExpressRoute limits?for more information.

For an extra charge, the ExpressRoute Premium add-on provides some additional capability:

• Increased route limits for private peering.
• Increased number of VNet links per ExpressRoute circuit.
• Global connectivity for services.

See?ExpressRoute pricing?for details.

ExpressRoute circuits are designed to allow temporary network bursts up to two times the bandwidth limit that you procured for no additional cost. This is achieved by using redundant links. However, not all connectivity providers support this feature. Verify that your connectivity provider enables this feature before depending on it.

Availability

ExpressRoute does not support router redundancy protocols such as hot standby routing protocol (HSRP) and virtual router redundancy protocol (VRRP) to implement high availability. Instead, it uses a redundant pair of BGP sessions per peering. To facilitate highly-available connections to your network, Azure provisions you with two redundant ports on two routers (part of the Microsoft edge) in an active-active configuration.

By default, BGP sessions use an idle timeout value of 60 seconds. If a session times out three times (180 seconds total), the router is marked as unavailable, and all traffic is redirected to the remaining router. This 180-second timeout might be too long for critical applications. If so, you can change your BGP time-out settings on the on-premises router to a smaller value. ExpressRoute also supports?Bidirectional Forwarding Detection (BFD)?over private peering. By enabling BFD over ExpressRoute, you can expedite link failure detection between Microsoft Enterprise edge (MSEE) devices and the routers on which you terminate the ExpressRoute circuit (PE). You can terminate ExpressRoute over Customer Edge routing devices or Partner Edge routing devices (if you went with managed Layer 3 connection service).

You can configure high availability for your Azure connection in different ways, depending on the type of provider you use, and the number of ExpressRoute circuits and virtual network gateway connections you're willing to configure. The following summarizes your availability options:

• If you're using a layer 2 connection, deploy redundant routers in your on-premises network in an active-active configuration. Connect the primary circuit to one router, and the secondary circuit to the other. This will give you a highly available connection at both ends of the connection. This is necessary if you require the ExpressRoute service level agreement (SLA). See?SLA for Azure ExpressRoute?for details.
• The following diagram shows a configuration with redundant on-premises routers connected to the primary and secondary circuits. Each circuit handles the traffic for private peering (each peering is designated a pair of /30 address spaces, as described in the previous section).

• If you're using a layer 3 connection, verify that it provides redundant BGP sessions that handle availability for you.
• Connect the VNet to multiple ExpressRoute circuits, supplied by different service providers. This strategy provides additional high-availability and disaster recovery capabilities.
• Configure a site-to-site VPN as a failover path for ExpressRoute. For more about this option, see?Connect an on-premises network to Azure using ExpressRoute with VPN failover. This option only applies to private peering. For Azure and Microsoft 365 services, the Internet is the only failover path.

Security

Security provides assurances against deliberate attacks and the abuse of your valuable data and systems. For more information, see?Overview of the security pillar.

You can configure security options for your Azure connection in different ways, depending on your security concerns and compliance needs.

ExpressRoute operates in layer 3. Threats in the application layer can be prevented by using a network security appliance that restricts traffic to legitimate resources.

To maximize security, add network security appliances between the on-premises network and the provider edge routers. This will help to restrict the inflow of unauthorized traffic from the VNet:

For auditing or compliance purposes, it may be necessary to prohibit direct access from components running in the VNet to the Internet and implement forced tunneling. In this situation, Internet traffic should be redirected back through a proxy running on-premises where it can be audited. The proxy can be configured to block unauthorized traffic flowing out, and filter potentially malicious inbound traffic.

To maximize security, do not enable a public IP address for your VMs, and use NSGs to ensure that these VMs aren't publicly accessible. VMs should only be available using the internal IP address. These addresses can be made accessible through the ExpressRoute network, enabling on-premises DevOps staff to perform configuration or maintenance.

If you must expose management endpoints for VMs to an external network, use NSGs or access control lists to restrict the visibility of these ports to an allowlist of IP addresses or networks.

Microsoft Side:

• The?C-Tag?is different per peer type (Private/Microsoft). There are two C-Tags, one for Microsoft peering and one for Private peering. The C-Tag is used to identify the peer/routing domain. This tag is defined by customer and input in Azure Portal.
• The?S-Tag?has a one to one relationship with the ExpressRoute circuit, this applies to each circuit, an S-Tag for each circuit. This is defined by Microsoft and customer is not aware of it.
• For?Microsoft peering, as there’s no vNet gateway, it doesn’t need isolation, as all IP addresses are public & unique, there is no need for VRF. The difference being, with private peering, there’s a chance of an overlap of IP address ranges hence the need for VRF.
• VRF, works on Layer 2.5 at the MSEE level. VRF does isolation within routing, like a namespace in Linux. The Exchange Provider is not aware of VRF, because The Exchange Provider works on Layer 2, this is the maximum layer it can go to.
• Traffic to Azure?| The Exchange Provider adds the?S-Tag, then the MSEE removes both the?S-Tag?&?C-Tag?from the packet.
• Traffic from Azure?| The MSEE adds both the?S-Tag?&?C-Tag?to the packet.
• The Exchange Provider equipment is a highly available layer 2 switch stack.
• Cross Connects?are layer 1 physical connections between the Exchange Provider & the MSEE.

Customer Side focus :

• The BGP peers are setup independently as a first step prior to any NAT’ing & advertising of address ranges.
• Unlike with Public Peering, Microsoft Peering requires an additional public subnet to be available so it can be used for NAT’ing and advertising. This subnet cannot be the same as the subnet used for peering.
• Public IP addresses advertised to Microsoft over ExpressRoute must not be advertised to the Internet. This may break connectivity to other Microsoft services. However, Public IP addresses used by servers in your network that communicate with Office 365 endpoints within Microsoft may be advertised over ExpressRoute.
• With Microsoft peering, traffic destined to Microsoft cloud services must be SNATed to valid public IPv4 addresses before they enter the Microsoft network.

Sharing the same ExpressRoute circuit

“can I share the same ExpressRoute circuit across multiple subscriptions and multiple subscriptions across tenants?” The short answer is?Yes.

It’s all to do with the Azure entities in the Azure portal – the?ExpressRoute Circuit, the?Connection?and the?Virtual Network Gateway. In fact, it’s the?Connection?entity that glues the?ExpressRoute Circuit?and the?Virtual Network Gateway?together. The glue, requires a?Circuit?authorisation key in order for other?Connections?to utilise it.

Below is what it looks like:

Reference:

https://learn.microsoft.com/en-us/azure/expressroute/