AWS S3 Private Link
AWS Private Link Overview
What’s AWS private link in general and what’s the benefit of using AWS private link?
AWS Private Link provides private connectivity between VPCs, AWS services, and your on-premises networks, without exposing your traffic to the public internet.
- Network traffic that uses AWS Private Link doesn’t traverse the public internet
- Reduce exposure to attacks.
- Regulatory compliance
- Hybrid cloud
S3 Private Link
AWS Private Link for S3 is GA recently, I started to compare the feature with Azure storage, Azure storage private link is GA a year ago and Azure storage service endpoint support across region access.
With AWS PrivateLink for Amazon S3, you can provision interface VPC endpoints (interface endpoints) in your virtual private cloud (VPC) instead of connecting over the internet. These endpoints are directly accessible from applications that are on premises over VPN and AWS Direct Connect, or in a different AWS Region over VPC peering
Concept is pretty much same between these two CSPs.
Gateway endpoint is added to route table, while Azure service endpoint add the route behind the scene.
Interface endpoint is represented by ENIs that are assigned private IP, while Azure private endpoint is a private IP NIC.
DNS Integration
Endpoint-specific S3 DNS names can be resolved from the S3 public DNS domain.
Amazon S3 generates two types of endpoint-specific, S3 DNS names: Regional and zonal. For example vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com.
vpce-1a2b3c4d-5e6f-us-east-1a.s3.us-east-1.vpce.amazonaws.com
Azure storage account private link relies on Azure private DNS zone, and it provides nice integration. More detail can be found from my other story https://cloudjourney.medium.com/azure-private-endpoint-and-private-dns-zone-integration-26cda64ed2f
Azure private link has private endpoint specific FQDN, but you will not access resource through it, you use the original FQDN to access resource, private endpoint specific FQDN is added by Azure behind the scene in the CNAME chain, and from Azure default DNS server, it does not resolve any of the FQDN to your private IP.
Lab
Lab Overview
Interface endpoints are directly accessible from applications that are on premises over VPN and AWS Direct Connect, or in a different AWS Region over VPC peering.
It’s hard to mimic either VPN or Direct Connect connectivity, so the idea is to create interface endpoints (VPC feature) for S3 in us-east-1 , then access from an EC2 in us-west-1 region.
Keep in mind, cross region access is an enhanced feature which is provided by private link, gateway endpoint can only access from same region.
Create Interface Endpoint
First create your own test VPC (10.0.0.0/16) in us-east-1. In this case don’t use the default VPC, since later on, we will need to do VPC peering to us-west-1 default VPC. We have to create our own VPC with different CIDR in order to avoid IP address overlap.
Then from the us-east-1 test VPC, we add interface endpoint. When create interface endpoint, we start from one AZ and use default security group, just to save some effort. After provisioning, it might take couple of minutes for the endpoint status to change to green.
Now interface endpoint shows available status.
Interface endpoint FQDN is resolved to private IP.
C:\Users\rquan>nslookup bucket.vpce-0e787478ce3b1998c-hvc8qoxf.s3.us-east-1.vpce.amazonaws.com
Server: homeportal
Address: 192.168.1.254Non-authoritative answer:
Name: bucket.vpce-0e787478ce3b1998c-hvc8qoxf.s3.us-east-1.vpce.amazonaws.com
Address: 10.0.3.231C:\Users\rquan>nslookup accesspoint.vpce-0e787478ce3b1998c-hvc8qoxf.s3.us-east-1.vpce.amazonaws.com
Server: homeportal
Address: 192.168.1.254Non-authoritative answer:
Name: accesspoint.vpce-0e787478ce3b1998c-hvc8qoxf.s3.us-east-1.vpce.amazonaws.com
Address: 10.0.3.231C:\Users\rquan>nslookup control.vpce-0e787478ce3b1998c-hvc8qoxf.s3.us-east-1.vpce.amazonaws.com
Server: homeportal
Address: 192.168.1.254Non-authoritative answer:
Name: control.vpce-0e787478ce3b1998c-hvc8qoxf.s3.us-east-1.vpce.amazonaws.com
Address: 10.0.3.231
From my home PC, I’m only able to access via public endpoint.
C:\Users\rquan>aws s3 --region us-east-1 ls s3://privatelinkrq/
2021-05-15 13:21:10 846291 developer.PNGI’m not able to access the interface endpoint, that’s expected since interface endpoint can only be accessed from private network.
C:\Users\rquan>aws s3 --region us-east-1 --endpoint-url https://bucket.vpce-0e787478ce3b1998c-hvc8qoxf.s3.us-east-1.vpce.amazonaws.com ls s3://privatelinkrq/Could not connect to the endpoint URL: "https://bucket.vpce-0e787478ce3b1998c-hvc8qoxf.s3.us-east-1.vpce.amazonaws.com/privatelinkrq?list-type=2&prefix=&delimiter=%2F&encoding-type=url"
Create EC2
I skip detail steps and assume you all know how to create an EC2.
I select free tier t2.micro and Amazon Linux, public IP is assigned, so that I will be able to ssh to the box. In corporate environment, you wouldn’t need public IP for the vm at all.
I use default VPC (172.31.0.0/16) in us-west-1 region to save some effort.
From Azure portal, when create Azure VM, you may input a local account user name.
For EC2, the default username is “ec2-user”
Create VPC Peering
We are going to access from us-west-1 VPC to us-east-1 VPC, so we create the peering from source side (us-west-1).
The peering connection from us-east-1 shows as pending, we have to accept the request, the peering connection turns green after the acceptance.
Don’t forget to configure routes in the subnet from both region.
From us-west-1, when destination is 10.0.0.0/16, next hop is peering connection.
From us-east-1, when destination is 172.31.0.0/16, next hop is the peering connection as well.
Validation
From EC2 instance, we are able to access S3 using both interface endpoint and the public endpoint.
Access via interface endpoint:
[ec2-user@ip-172-31-11-119 ~]$ aws s3 --region us-east-1 --endpoint-url https://bucket.vpce-0e787478ce3b1998c-hvc8qoxf.s3.us-east-1.vpce.amazonaws.com ls s3://privatelinkrq/
2021-05-15 17:21:10 846291 developer.PNG[ec2-user@ip-172-31-11-119 ~]$ aws s3control --region us-east-1 --endpoint-url https://control.vpce-0e787478ce3b1998c-hvc8qoxf.s3.us-east-1.vpce.amazonaws.com list-jobs --account-id <xyz>
{
"Jobs": []
}
Access without interface endpoint:
[ec2-user@ip-172-31-11-119 ~]$ aws s3 --region us-east-1 ls s3://privatelinkrq/
2021-05-15 17:21:10 846291 developer.PNG
Conclusion
It’s fairly straight forward to configure AWS S3 private link, one endpoint supports working with multiple S3 buckets, DNS configuration is also taken care of by default.
In future story, I will focus on endpoint policy and compare with Azure storage account functionality.
