Azure ExpressRoute Routing Optimization
Overview
When you have multiple ExpressRoute circuits, you have more than one path to connect to Microsoft network, your traffic may take a longer path to reach to Microsoft, and Microsoft to your network, the longer the network path, the higher the latency.
In this blog post, we summarize the solutions to influence the routing path for following scenarios:
- Customer to MS network via ExpressRoute public peering
- Customer to Azure VNet via ExpressRoute private peering
- MS network to customer via ExpressRoute Microsoft peering
- Azure VNet to customer via ExpressRoute private peering
- Azure to Azure
Before we dig into each scenario, let’s first understand Azure ExpressRoute peering.
ExpressRoute peering
An ExpressRoute circuit has multiple routing domains/peerings associated with it: Azure public, Azure private, and Microsoft. Each peering is configured identically on a pair of routers (in active-active or load sharing configuration) for high availability. Azure services are categorized as Azure public and Azure private to represent the IP addressing schemes.
Public peering is deprecated for new circuits, we still use public peering as one example to help understand fundamental concepts.
Now let’s look at the scenarios one by one.
Customer to MS network via ExpressRoute public peering
You access Azure public services in a region which is closer to your office, for resilience purpose, you also has path to access to Azure public services in remote region. Meaning Azure US west and US East prefixes are all advertised to you.
When traffic takes the longer path, for example, LA office goes to US East ExpressRoute, then reaches to Azure US West services, you will experience longer latency.
The solution is for customer to update from their corporate network and assign higher BGP local preference based on BGP community values to influence routing path.
MS assigned a unique BGP community value to each Azure region. e.g. “12076:51004” for US East, “12076:51006” for US West.
In below solution diagram, corporate network LA region has higher local preference 400 for Azure US West prefix 13.100.0.0/16. your users in Los Angeles will take the ExpressRoute circuit in US West to connect to Azure US West.
Customer to Azure VNet via ExpressRoute private peering
Similar Local Preference solution can be applied to routing from customer corporate network to your Azure Virtual Network. MS does not tag BGP community value to the prefixes advertised from Azure vnet to your corporate network.
So you make sure designate unique CIDRs to your Azure vnets per region.
Then you know which of your Azure IP prefixes is close to which of your office, and configure your routers accordingly to prefer one ExpressRoute circuit to another. For example, when destination is your designated Azure US West IP, the path is through US West ExpressRoute circuit.
MS network to customer via ExpressRoute Microsoft peering
Here is another example where connections from Microsoft take a longer path to reach your network.
For failover purpose, you advertise advertise both of the prefixes on both ExpressRoute circuits. Without any hint, the Microsoft network can’t tell which customer prefix is close to US East and which one is close to US West. It happens to pick the longer path.
The solution is to use AS PATH prepending.
In this example, you can lengthen the AS PATH for 172.2.0.0/31 in US East so that we will prefer the ExpressRoute circuit in US West for traffic destined for this prefix (as our network will think the path to this prefix is shorter in the west).
You need to peer with a public AS and append public AS numbers in the AS PATH to influence routing for Microsoft Peering.
Azure VNet to customer via ExpressRoute private peering
MS supports the same capabilities for private peering, and applicable to one single ExpressRoute circuit as well.
Azure to Azure
BGP utilizes a best path selection algorithm based on a number of factors including longest prefix match (LPM).
In case same prefix with multiple path, they do Equal-Cost-Multi-Path (ECMP) routing to load-balance inter-VNet traffic, some traffic flows will take the longer path and get routed at the remote ExpressRoute circuit.
The solution is to assign a higher weight to the local connection.
$connection = Get-AzVirtualNetworkGatewayConnection -Name “MyVirtualNetworkConnection” -ResourceGroupName “MyRG”
$connection.RoutingWeight = 100
Set-AzVirtualNetworkGatewayConnection -VirtualNetworkGatewayConnection $connectionConclusion
In Azure or hybrid network you could use longest prefix match(LPM), ExpressRoute virtual network gateway connection routing weight, BPG local preference and AS path prepending to influence routing path.
