AWS Network Latency Comparison
Overview
Are you wondering about the network latency in AWS between zones and between vpcs? In this blog post, we will test the network latency between different zones within a vpc, as well as network latency between zones from different vpcs in the same region.
For details on creating and configuring a transit gateway, please refer to my previous blogs:
TGW provisioning and route configuration: https://cloudjourney.medium.com/aws-transit-gateway-and-service-linked-role-cb37c8d166d3
Route analyzer for connectivity troubleshooting:
https://cloudjourney.medium.com/aws-transit-gateway-network-manager-4fbb7b5110b0
Attach vpc from another account to the TGW:
https://cloudjourney.medium.com/aws-ram-and-transit-gateway-8ac230f298e8
For networking benchmark CLI tools, please refer to my blog:
https://cloudjourney.medium.com/networking-benchmark-command-line-tools-dcdf7d564c60
Configuration
If you don’t want to assign public IP to the EC2 instance and want to use SSM to login, you need an outbound path to the internet.
- SSM agent setup requires outbound to couple of AWS public sites (unless setup private link)
- For the EC2 in a private subnet, in order to be able to reach out to internet, you need NAT gateway. NAT maps the private IP addresses to a public address for requests.
Also please use the zone ID (not the zone name) to match the zones.
Please ensure no IP address overlap between connected VPCs.
TGW can be shared with organization, ou or account, and add TGW vpc attachments from multiple accounts.
VPC peering connection can be added to local vpc, vpc in same account or vpc from another account.
Accept the peering connection from the target account.
Similar as TGW route configuration, you need to add a route which points to the peering connection. This configuration needs to be done from both accounts.
Same VPC
VPC1 in account 1 zone use1-az4 o use1-az4
I have one EC2 in a subnet which is designated to zone use1-az4, and another EC2 in another subnet which is also designated to zone use1-az4.
Average time for a round trip is 0.614 millisecond.
[ec2-user@ip-10-0-0-250 ~]$ ping -c 10 10.0.1.44
PING 10.0.1.44 (10.0.1.44) 56(84) bytes of data.
64 bytes from 10.0.1.44: icmp_seq=1 ttl=255 time=0.580 ms
64 bytes from 10.0.1.44: icmp_seq=2 ttl=255 time=0.617 ms
64 bytes from 10.0.1.44: icmp_seq=3 ttl=255 time=0.611 ms
64 bytes from 10.0.1.44: icmp_seq=4 ttl=255 time=0.630 ms
64 bytes from 10.0.1.44: icmp_seq=5 ttl=255 time=0.675 ms
64 bytes from 10.0.1.44: icmp_seq=6 ttl=255 time=0.572 ms
64 bytes from 10.0.1.44: icmp_seq=7 ttl=255 time=0.626 ms
64 bytes from 10.0.1.44: icmp_seq=8 ttl=255 time=0.598 ms
64 bytes from 10.0.1.44: icmp_seq=9 ttl=255 time=0.611 ms
64 bytes from 10.0.1.44: icmp_seq=10 ttl=255 time=0.622 ms
--- 10.0.1.44 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9211ms
rtt min/avg/max/mdev = 0.572/0.614/0.675/0.031 msWhen I ping the vm or localhost itself, I got following result.
[ec2-user@ip-10-0-0-250 ~]$ ping -c 10 10.0.0.250
PING 10.0.0.250 (10.0.0.250) 56(84) bytes of data.
64 bytes from 10.0.0.250: icmp_seq=1 ttl=255 time=0.023 ms
64 bytes from 10.0.0.250: icmp_seq=2 ttl=255 time=0.041 ms
64 bytes from 10.0.0.250: icmp_seq=3 ttl=255 time=0.031 ms
64 bytes from 10.0.0.250: icmp_seq=4 ttl=255 time=0.033 ms
64 bytes from 10.0.0.250: icmp_seq=5 ttl=255 time=0.032 ms
64 bytes from 10.0.0.250: icmp_seq=6 ttl=255 time=0.031 ms
64 bytes from 10.0.0.250: icmp_seq=7 ttl=255 time=0.032 ms
64 bytes from 10.0.0.250: icmp_seq=8 ttl=255 time=0.029 ms
64 bytes from 10.0.0.250: icmp_seq=9 ttl=255 time=0.032 ms
64 bytes from 10.0.0.250: icmp_seq=10 ttl=255 time=0.029 ms
--- 10.0.0.250 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9214ms
rtt min/avg/max/mdev = 0.023/0.031/0.041/0.006 ms
[ec2-user@ip-10-0-0-250 ~]$ ping -c 10 localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=255 time=0.020 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=255 time=0.032 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=255 time=0.028 ms
64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=255 time=0.029 ms
64 bytes from localhost (127.0.0.1): icmp_seq=5 ttl=255 time=0.029 ms
64 bytes from localhost (127.0.0.1): icmp_seq=6 ttl=255 time=0.030 ms
64 bytes from localhost (127.0.0.1): icmp_seq=7 ttl=255 time=0.030 ms
64 bytes from localhost (127.0.0.1): icmp_seq=8 ttl=255 time=0.029 ms
64 bytes from localhost (127.0.0.1): icmp_seq=9 ttl=255 time=0.028 ms
64 bytes from localhost (127.0.0.1): icmp_seq=10 ttl=255 time=0.030 ms
--- localhost ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9204ms
rtt min/avg/max/mdev = 0.020/0.028/0.032/0.006 msVPC1 in Account 1 zone use1-az4 to use1-az6
I have one EC2 in a subnet which is designated to zone use1-az4, and another EC2 in another subnet which is designated to zone use1-az6.
Average time for a round trip is 1.175 millisecond.
[ec2-user@ip-10-0-0-250 ~]$ ping -c 10 10.0.2.215
PING 10.0.2.215 (10.0.2.215) 56(84) bytes of data.
64 bytes from 10.0.2.215: icmp_seq=1 ttl=255 time=1.59 ms
64 bytes from 10.0.2.215: icmp_seq=2 ttl=255 time=1.14 ms
64 bytes from 10.0.2.215: icmp_seq=3 ttl=255 time=1.16 ms
64 bytes from 10.0.2.215: icmp_seq=4 ttl=255 time=1.09 ms
64 bytes from 10.0.2.215: icmp_seq=5 ttl=255 time=1.17 ms
64 bytes from 10.0.2.215: icmp_seq=6 ttl=255 time=1.15 ms
64 bytes from 10.0.2.215: icmp_seq=7 ttl=255 time=1.09 ms
64 bytes from 10.0.2.215: icmp_seq=8 ttl=255 time=1.10 ms
64 bytes from 10.0.2.215: icmp_seq=9 ttl=255 time=1.12 ms
64 bytes from 10.0.2.215: icmp_seq=10 ttl=255 time=1.10 ms
--- 10.0.2.215 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9012ms
rtt min/avg/max/mdev = 1.092/1.175/1.593/0.144 msVPC2 in Account 2 use1-az6 to VPC2 in Account 2 use1-az4
The latency is about the same as the result from account 1.
[ec2-user@ip-172-31-46-43 ~]$ ping -c 10 172.31.20.39
PING 172.31.20.39 (172.31.20.39) 56(84) bytes of data.
64 bytes from 172.31.20.39: icmp_seq=1 ttl=255 time=1.12 ms
64 bytes from 172.31.20.39: icmp_seq=2 ttl=255 time=1.20 ms
64 bytes from 172.31.20.39: icmp_seq=3 ttl=255 time=1.11 ms
64 bytes from 172.31.20.39: icmp_seq=4 ttl=255 time=1.19 ms
64 bytes from 172.31.20.39: icmp_seq=5 ttl=255 time=1.19 ms
64 bytes from 172.31.20.39: icmp_seq=6 ttl=255 time=1.16 ms
64 bytes from 172.31.20.39: icmp_seq=7 ttl=255 time=1.72 ms
64 bytes from 172.31.20.39: icmp_seq=8 ttl=255 time=1.17 ms
64 bytes from 172.31.20.39: icmp_seq=9 ttl=255 time=1.18 ms
64 bytes from 172.31.20.39: icmp_seq=10 ttl=255 time=1.14 ms
--- 172.31.20.39 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9013ms
rtt min/avg/max/mdev = 1.115/1.222/1.723/0.174 msVPCs Through TGW
VPC1 in Account 1 use1-az4 to VPC2 in Account 2 use1-az4
The latency is still sub-millisecond, but is higher than the latency between same zones in one single vpc. I guess TGW adds a slight overhead to the overall latency.
[ec2-user@ip-10-0-0-21 ~]$ ping -c 10 172.31.20.39
PING 172.31.20.39 (172.31.20.39) 56(84) bytes of data.
64 bytes from 172.31.20.39: icmp_seq=1 ttl=254 time=1.40 ms
64 bytes from 172.31.20.39: icmp_seq=2 ttl=254 time=1.02 ms
64 bytes from 172.31.20.39: icmp_seq=3 ttl=254 time=0.971 ms
64 bytes from 172.31.20.39: icmp_seq=4 ttl=254 time=0.933 ms
64 bytes from 172.31.20.39: icmp_seq=5 ttl=254 time=0.914 ms
64 bytes from 172.31.20.39: icmp_seq=6 ttl=254 time=0.891 ms
64 bytes from 172.31.20.39: icmp_seq=7 ttl=254 time=0.877 ms
64 bytes from 172.31.20.39: icmp_seq=8 ttl=254 time=0.910 ms
64 bytes from 172.31.20.39: icmp_seq=9 ttl=254 time=1.01 ms
64 bytes from 172.31.20.39: icmp_seq=10 ttl=254 time=1.00 ms
--- 172.31.20.39 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9010ms
rtt min/avg/max/mdev = 0.877/0.994/1.400/0.150 msVPC1 in Account 1 use1-az4 to VPC2 in Account 2 use1-az6
This latency is also higher than the cross zones in one single VPC.
[ec2-user@ip-10-0-0-21 ~]$ ping -c 10 172.31.46.43
PING 172.31.46.43 (172.31.46.43) 56(84) bytes of data.
64 bytes from 172.31.46.43: icmp_seq=1 ttl=254 time=1.83 ms
64 bytes from 172.31.46.43: icmp_seq=2 ttl=254 time=1.57 ms
64 bytes from 172.31.46.43: icmp_seq=3 ttl=254 time=1.55 ms
64 bytes from 172.31.46.43: icmp_seq=4 ttl=254 time=1.52 ms
64 bytes from 172.31.46.43: icmp_seq=5 ttl=254 time=1.60 ms
64 bytes from 172.31.46.43: icmp_seq=6 ttl=254 time=1.57 ms
64 bytes from 172.31.46.43: icmp_seq=7 ttl=254 time=1.53 ms
64 bytes from 172.31.46.43: icmp_seq=8 ttl=254 time=4.06 ms
64 bytes from 172.31.46.43: icmp_seq=9 ttl=254 time=1.45 ms
64 bytes from 172.31.46.43: icmp_seq=10 ttl=254 time=1.59 ms
--- 172.31.46.43 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9015ms
rtt min/avg/max/mdev = 1.457/1.831/4.060/0.749 msVPCs Through Peering
VPC 1 in Account 1 use1-az4 to VPC 2 in Account 2 use1-az4
The latency is about the same as the result from one single VPC.
[ec2-user@ip-10-0-0-21 ~]$ ping -c 30 172.31.20.39
PING 172.31.20.39 (172.31.20.39) 56(84) bytes of data.
64 bytes from 172.31.20.39: icmp_seq=1 ttl=255 time=1.10 ms
64 bytes from 172.31.20.39: icmp_seq=2 ttl=255 time=0.580 ms
64 bytes from 172.31.20.39: icmp_seq=3 ttl=255 time=0.756 ms
64 bytes from 172.31.20.39: icmp_seq=4 ttl=255 time=0.633 ms
64 bytes from 172.31.20.39: icmp_seq=5 ttl=255 time=0.637 ms
64 bytes from 172.31.20.39: icmp_seq=6 ttl=255 time=0.840 ms
64 bytes from 172.31.20.39: icmp_seq=7 ttl=255 time=0.677 ms
64 bytes from 172.31.20.39: icmp_seq=8 ttl=255 time=0.598 ms
64 bytes from 172.31.20.39: icmp_seq=9 ttl=255 time=0.548 ms
64 bytes from 172.31.20.39: icmp_seq=10 ttl=255 time=0.907 ms
64 bytes from 172.31.20.39: icmp_seq=11 ttl=255 time=0.583 ms
64 bytes from 172.31.20.39: icmp_seq=12 ttl=255 time=0.590 ms
64 bytes from 172.31.20.39: icmp_seq=13 ttl=255 time=0.547 ms
64 bytes from 172.31.20.39: icmp_seq=14 ttl=255 time=0.577 ms
64 bytes from 172.31.20.39: icmp_seq=15 ttl=255 time=0.596 ms
64 bytes from 172.31.20.39: icmp_seq=16 ttl=255 time=0.619 ms
64 bytes from 172.31.20.39: icmp_seq=17 ttl=255 time=0.569 ms
64 bytes from 172.31.20.39: icmp_seq=18 ttl=255 time=0.611 ms
64 bytes from 172.31.20.39: icmp_seq=19 ttl=255 time=0.616 ms
64 bytes from 172.31.20.39: icmp_seq=20 ttl=255 time=0.631 ms
64 bytes from 172.31.20.39: icmp_seq=21 ttl=255 time=0.545 ms
64 bytes from 172.31.20.39: icmp_seq=22 ttl=255 time=0.550 ms
64 bytes from 172.31.20.39: icmp_seq=23 ttl=255 time=0.581 ms
64 bytes from 172.31.20.39: icmp_seq=24 ttl=255 time=0.556 ms
64 bytes from 172.31.20.39: icmp_seq=25 ttl=255 time=0.670 ms
64 bytes from 172.31.20.39: icmp_seq=26 ttl=255 time=0.557 ms
64 bytes from 172.31.20.39: icmp_seq=27 ttl=255 time=0.555 ms
64 bytes from 172.31.20.39: icmp_seq=28 ttl=255 time=0.610 ms
64 bytes from 172.31.20.39: icmp_seq=29 ttl=255 time=0.633 ms
64 bytes from 172.31.20.39: icmp_seq=30 ttl=255 time=0.591 ms
--- 172.31.20.39 ping statistics ---
30 packets transmitted, 30 received, 0% packet loss, time 29620ms
rtt min/avg/max/mdev = 0.545/0.635/1.107/0.123 msVPC 1 in Account 1 use1-az4 to VPC 2 in Account 2 use1-az6
[ec2-user@ip-10-0-0-21 ~]$ ping -c 30 172.31.46.43
PING 172.31.46.43 (172.31.46.43) 56(84) bytes of data.
64 bytes from 172.31.46.43: icmp_seq=1 ttl=255 time=1.86 ms
64 bytes from 172.31.46.43: icmp_seq=2 ttl=255 time=1.25 ms
64 bytes from 172.31.46.43: icmp_seq=3 ttl=255 time=1.32 ms
64 bytes from 172.31.46.43: icmp_seq=4 ttl=255 time=1.28 ms
64 bytes from 172.31.46.43: icmp_seq=5 ttl=255 time=1.27 ms
64 bytes from 172.31.46.43: icmp_seq=6 ttl=255 time=1.25 ms
64 bytes from 172.31.46.43: icmp_seq=7 ttl=255 time=1.25 ms
64 bytes from 172.31.46.43: icmp_seq=8 ttl=255 time=1.33 ms
64 bytes from 172.31.46.43: icmp_seq=9 ttl=255 time=1.34 ms
64 bytes from 172.31.46.43: icmp_seq=10 ttl=255 time=1.29 ms
64 bytes from 172.31.46.43: icmp_seq=11 ttl=255 time=1.30 ms
64 bytes from 172.31.46.43: icmp_seq=12 ttl=255 time=1.25 ms
64 bytes from 172.31.46.43: icmp_seq=13 ttl=255 time=1.25 ms
64 bytes from 172.31.46.43: icmp_seq=14 ttl=255 time=1.32 ms
64 bytes from 172.31.46.43: icmp_seq=15 ttl=255 time=1.26 ms
64 bytes from 172.31.46.43: icmp_seq=16 ttl=255 time=1.26 ms
64 bytes from 172.31.46.43: icmp_seq=17 ttl=255 time=1.31 ms
64 bytes from 172.31.46.43: icmp_seq=18 ttl=255 time=1.27 ms
64 bytes from 172.31.46.43: icmp_seq=19 ttl=255 time=1.28 ms
64 bytes from 172.31.46.43: icmp_seq=20 ttl=255 time=1.20 ms
64 bytes from 172.31.46.43: icmp_seq=21 ttl=255 time=1.30 ms
64 bytes from 172.31.46.43: icmp_seq=22 ttl=255 time=1.27 ms
64 bytes from 172.31.46.43: icmp_seq=23 ttl=255 time=1.30 ms
64 bytes from 172.31.46.43: icmp_seq=24 ttl=255 time=1.34 ms
64 bytes from 172.31.46.43: icmp_seq=25 ttl=255 time=1.30 ms
64 bytes from 172.31.46.43: icmp_seq=26 ttl=255 time=1.34 ms
64 bytes from 172.31.46.43: icmp_seq=27 ttl=255 time=1.27 ms
64 bytes from 172.31.46.43: icmp_seq=28 ttl=255 time=1.25 ms
64 bytes from 172.31.46.43: icmp_seq=29 ttl=255 time=1.28 ms
64 bytes from 172.31.46.43: icmp_seq=30 ttl=255 time=1.27 ms
--- 172.31.46.43 ping statistics ---
30 packets transmitted, 30 received, 0% packet loss, time 29043ms
rtt min/avg/max/mdev = 1.203/1.305/1.867/0.119 msConclusions
You definitely want to keep the traffic within same zone. Network latency is comparable within a single VPC or between VPCs that are connected through VPC peering.
It is safe to say that same-zone latency is at the sub-millisecond level, regardless of whether the traffic is from one VPC, between accounts, or between VPCs. The Transit Gateway (TGW) introduces a slight overhead, and the overall latency within the same zone through the TGW is sometimes slightly over 1 millisecond. For cross-zone traffic within the same region, the latency is between 1 and 2 milliseconds.
References
“cheat sheet: IGW is attached to a VPC. NAT Gateway is scoped to a public subnet. Route table associated to public subnet has a route to 0.0.0.0/0 via IGW. Route table associated to private subnet has a route to 0.0.0.0/0 via NAT. Route table does not require ‘route propogration’ configuration”
“https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-prerequisites.html”
“https://aws.amazon.com/premiumsupport/knowledge-center/nat-gateway-vpc-private-subnet/”
“https://www.mirantis.com/blog/multi-container-pods-and-container-communication-in-kubernetes/”
