EC2 DNS Name in Multiple VPCs
Overview
It’s quite common to call your IaaS workload via DNS name, and you could also have application from other account or other VPC to access the workload hosted in your EC2.
In this blog post, I plan to explore more about EC2 public DNS name, private DNS name, route53 private hosted zone. The goal is to access EC2 workload from other region VPC via private DNS name, we will experiment with both default private DNS name and custom DNS name.
For VPC connectivity, we will include both direct VPC peering and transit gateway scenarios.
Through prior articles, we have experiences on creating VPC peering, transit gateway and EC2 instances, so we will only walk through details on private hosted zone configuration.
Lab Without Private Hosted Zone
When EC2 instances are in same region, even in different VPC, from one instance, I could resolve default private DNS name in other VPC.
However it does not resolve default private DNS name in different region. We will add connectivity between the VPCs from different region, and test again on naming resolution.
Scenario 1A
Use EC2 default private DNS name and test if it can be resolved from other region VPC which is connected via VPC peering.
Even though I’m able to ping the instance in other region after I properly configure subnet route tables in both side and security group inbound rule in the destination side, I still get NXDOMAIN when try to resolve instance DNS name from other region.
sh-4.2$ nslookup ip-10-100-2-70.us-west-2.compute.internal
Server: 172.31.0.2
Address: 172.31.0.2#53** server can't find ip-10-100-2-70.us-west-2.compute.internal: NXDOMAINsh-4.2$
It only works after I enable DNS resolution for queries from the peer VPC in both VPC peering connections. The official document does not mention to resolve private DNS hostname, it focus on enabling a VPC to resolve public IPv4 DNS hostnames to private IPv4 addresses when queried from instances in the peer VPC, but seems this option covers resolving private DNS name as well.
Scenario 1B
This scenario is similar to 1A, but instead of connecting via direct VPC peering, we use transit gateway and inter-region transit gateway peering.
Use EC2 default private DNS name and test if it can be resolved from other region VPC which is connected via transit gateway.
It does not work for across region transit gateway peering scenario. Looks like it’s by design based on AWS official doc Transit gateway peering attachments — Amazon Virtual Private Cloud
Transit gateway cross-Region peering does not support resolving public IPv4 DNS host names to private IPv4 addresses across VPCs on either side of the transit gateway peering attachment.
Once again, the document does not call out private DNS name resolution, but seems it covers both scenarios.
From us-esat-1 EC2, I’m able to ping us-west-2 instance.
[ec2-user@ip-172-31-0-177 ~]$ ping 10.100.2.70
PING 10.100.2.70 (10.100.2.70) 56(84) bytes of data.
64 bytes from 10.100.2.70: icmp_seq=1 ttl=252 time=82.8 ms
64 bytes from 10.100.2.70: icmp_seq=2 ttl=252 time=80.8 msBut I’m not able to lookup DNS name.
[ec2-user@ip-172-31-0-177 ~]$ nslookup ip-10-100-2-70.us-west-2.compute.internal
Server: 172.31.0.2
Address: 172.31.0.2#53** server can't find ip-10-100-2-70.us-west-2.compute.internal: NXDOMAIN
Let’s remote to us-west-2 instance, and test nslookup from there.
[ec2-user@ip-172-31-0-177 .ssh]$ ssh -i uswest2.pem ec2-user@10.100.2.70__| __|_ )
_| ( / Amazon Linux 2 AMI
___|\___|___|https://aws.amazon.com/amazon-linux-2/
We could resolve local instance private DNS name only.
[ec2-user@ip-10-100-2-70 ~]$ nslookup ip-10-100-2-70.us-west-2.compute.internal
Server: 10.100.0.2
Address: 10.100.0.2#53Non-authoritative answer:
Name: ip-10-100-2-70.us-west-2.compute.internal
Address: 10.100.2.70
It does not work when resolve us-east-1 instance.
[ec2-user@ip-10-100-2-70 ~]$ nslookup ip-172-31-0-177.ec2.internal
Server: 10.100.0.2
Address: 10.100.0.2#53** server can't find ip-172-31-0-177.ec2.internal: NXDOMAIN
What about resolve us-east1 instance public DNS name? The query returns expected public IP address.
[ec2-user@ip-10-100-2-70 ~]$ nslookup ec2-3-216-*-*.compute-1.amazonaws.com
Server: 10.100.0.2
Address: 10.100.0.2#53Non-authoritative answer:
Name: ec2-3-216-*-*.compute-1.amazonaws.com
Address: 3.216.*.*
Lab With Private Hosted Zone
To accommodate the exercise, we will configure Route53 Private Hosted Zone, associate both VPCs to the zone and add DNS record for both EC2 instances.
Scenario 2A
This test case is similar to test case 1B in terms of networking connectivity configuration. Difference is that private hosted zone comes into the picture.
Set Up Private Hosted Zone
Private Hosted Zone is global object, and can be associated with VPC from multiple regions.
Configure DNS Record
Add another A record for us-west-2 instance.
Validation
From us-west-2 instance, we are able to remote to us-east-1 instance through custom DNS name.
[ec2-user@ip-10-100-2-70 .ssh]$ ssh -i useast1.pem ec2-user@myec2useast1.testec2.net
Last login: Sat Jul 17 01:45:38 2021 from 108-218-44-13.lightspeed.rlghnc.sbcglobal.net__| __|_ )
_| ( / Amazon Linux 2 AMI
___|\___|___|https://aws.amazon.com/amazon-linux-2/
16 package(s) needed for security, out of 18 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-31-0-177 ~]$
From us-east-1 instance, we are able to resolve the custom DNS name and connectivity is also in place.
[ec2-user@ip-172-31-0-177 ~]$ ping myec2uswest2.testec2.net
PING myec2uswest2.testec2.net (10.100.2.70) 56(84) bytes of data.
64 bytes from ip-10-100-2-70.ec2.internal (10.100.2.70): icmp_seq=1 ttl=252 time=66.3 ms
64 bytes from ip-10-100-2-70.ec2.internal (10.100.2.70): icmp_seq=2 ttl=252 time=64.8 ms
64 bytes from ip-10-100-2-70.ec2.internal (10.100.2.70): icmp_seq=3 ttl=252 time=64.8 msScenario 2B
What happen if break the networking connectivity between the VPCs?
After I remove the transit gateway peering, nslookup still works for both regions, even though lost the connectivity between two instances.
[ec2-user@ip-172-31-0-177 ~]$ nslookup myec2useast1.testec2.net
Server: 172.31.0.2
Address: 172.31.0.2#53Non-authoritative answer:
Name: myec2useast1.testec2.net
Address: 172.31.0.177[ec2-user@ip-172-31-0-177 ~]$ nslookup myec2uswest2.testec2.net
Server: 172.31.0.2
Address: 172.31.0.2#53Non-authoritative answer:
Name: myec2uswest2.testec2.net
Address: 10.100.2.70[ec2-user@ip-172-31-0-177 ~]$ ping myec2uswest2.testec2.net
PING myec2uswest2.testec2.net (10.100.2.70) 56(84) bytes of data.
Route53 private hosted zone is global object, as long as VPC is associated with the zone, naming resolution works. What about if I disassociate us-east-1 VPC from the zone?
Apparently after the us-east-1 VPC is removed from the zone, from us-east-1 instance, we are no longer able to resolve the custom DNS name for both regions.
[ec2-user@ip-172-31-0-177 ~]$ nslookup myec2uswest2.testec2.net
Server: 172.31.0.2
Address: 172.31.0.2#53** server can't find myec2uswest2.testec2.net: NXDOMAIN[ec2-user@ip-172-31-0-177 ~]$ nslookup myec2useast1.testec2.net
Server: 172.31.0.2
Address: 172.31.0.2#53** server can't find myec2useast1.testec2.net: NXDOMAIN
Conclusion
During EC2 instance launch, I also experimented a private EC2, meaning no public IP, as expected, I’m not able to connect via session manager, since the instance needs to talk to session manager which is public endpoint.
We know it’s going to work when adding public IP and use public subnet. The other option is to create VPC endpoint as indicated in Manage private EC2 instances without internet access using Systems Manager (amazon.com)
I also experimented to leave us-west-2 instance as private, once I’m able to remote to us-east-1 instance, use it as jump box, from there ssh to us-west-2 instance.
References
Cheat Sheet:
Windows:
Purges the DNS Resolver cache:
ipconfig /flushdnsFlush DHCP and reflect new DNS server:
Invoke-Command -ComputerName -ScriptBlock {ipconfig /release}
Invoke-Command -ComputerName -ScriptBlock {ipconfig /renew}Linux:
sudo dhclient -v -r eth0
(note: check result in the resolv.conf file)
