Hybrid networks: Site-to-Site VPN, Direct Connect and PrivateLink
Most companies that run on AWS also have a data centre, an office, or a partner network that must reach the VPCs privately: the payments service calls a mainframe, the finance team's reports read a database on premises. Three tools cover it. Site-to-Site VPN sends encrypted IPsec tunnels over the internet, works in minutes, and is limited by the internet's quality. Direct Connect is a dedicated, private fibre connection, with predictable latency and bandwidth, that takes weeks to order. PrivateLink exposes one service, not a whole network, to other VPCs and accounts.
Need to know: a VPN connection joins a customer gateway (your router's public IP and BGP ASN) to a virtual private gateway (on one VPC) or a Transit Gateway (many VPCs), always as two tunnels to two AWS endpoints. Routes are static or learned by BGP, and reach the VPC route tables through route propagation. Direct Connect uses virtual interfaces (private, public, transit) and often a Direct Connect gateway. A VPN over the internet is the usual backup for it.
The VPN to the data centre
payments-vpc (10.41.0.0/16) has a VPN to the data centre in Frankfurt (192.168.100.0/24, router at 203.0.113.12):
$ aws ec2 describe-customer-gateways --query 'CustomerGateways[].[CustomerGatewayId,IpAddress,BgpAsn,State]' --output text
cgw-01f7b879bc3195d95 203.0.113.12 65000 available
$ aws ec2 describe-vpn-gateways --query 'VpnGateways[].[VpnGatewayId,State,VpcAttachments[0].VpcId,AmazonSideAsn]' --output text
vgw-072146817718d7ed9 available vpc-0fab6173e91281414 64512
$ VPN=$(aws ec2 describe-vpn-connections --filters Name=tag:Name,Values=dc-fra --query 'VpnConnections[0].VpnConnectionId' --output text)
$ aws ec2 describe-vpn-connections --vpn-connection-ids $VPN --query 'VpnConnections[0].{state: State, staticOnly: Options.StaticRoutesOnly, routes: Routes[].[DestinationCidrBlock,State], tunnels: VgwTelemetry[].[OutsideIpAddress,Status,StatusMessage]}'
{
"state": "available",
"staticOnly": true,
"routes": [
[
"192.168.100.0/24",
"available"
]
],
"tunnels": [
[
"3.120.89.187",
"UP",
""
],
[
"3.121.89.224",
"DOWN",
"IPSEC IS DOWN"
]
]
}
One tunnel is UP, the other DOWN. That is a common and dangerous state: traffic still flows over the remaining tunnel, so nothing breaks today, and the redundancy is gone. AWS replaces tunnel endpoints during maintenance one at a time, so a single-tunnel setup has outages that the provider announced in advance. Configure both tunnels on the customer router and alarm when either is down:
$ aws cloudwatch get-metric-statistics --namespace AWS/VPN --metric-name TunnelState --dimensions Name=VpnId,Value=$VPN --start-time $(date -u -d '-30 min' +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 300 --statistics Minimum Maximum --query 'sort_by(Datapoints,&Timestamp)[-1].[Minimum,Maximum]' --output text
0.5 0.5
TunnelState is 1 when a tunnel is up and 0 when it is down, per tunnel (TunnelIpAddress dimension). Per connection (VpnId) values between 0 and 1 mean at least one tunnel is not up: 0.5 here, one of two. 0 is a full outage. Alarm when the connection's value drops below 1, or on each tunnel, and send it to the team that owns the customer router.
Routes: propagation, and what the VPC sees
$ RT=$(aws ec2 describe-route-tables --filters Name=tag:Name,Values=payments-rt --query 'RouteTables[0].RouteTableId' --output text)
$ aws ec2 describe-route-tables --route-table-ids $RT --query 'RouteTables[0].{routes: Routes[].[DestinationCidrBlock,GatewayId || VpcPeeringConnectionId,Origin,State], propagating: PropagatingVgws[].GatewayId}'
{
"routes": [
[
"10.41.0.0/16",
"local",
"CreateRouteTable",
"active"
],
[
"192.168.100.0/24",
"vgw-072146817718d7ed9",
"EnableVgwRoutePropagation",
"active"
]
],
"propagating": [
"vgw-072146817718d7ed9"
]
}
$ I_PAY=$(aws ec2 describe-instances --filters Name=tag:Name,Values=payments-app Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ P=$(aws ec2 create-network-insights-path --source $I_PAY --destination-ip 192.168.100.20 --protocol tcp --destination-port 443 --query NetworkInsightsPath.NetworkInsightsPathId --output text)
$ A=$(aws ec2 start-network-insights-analysis --network-insights-path-id $P --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text) && sleep 6
$ aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $A --query 'NetworkInsightsAnalyses[0].NetworkPathFound' --output text
True
With route propagation on, the VGW adds the VPN's routes (static routes, or those learned over BGP) to the route table: Origin: EnableVgwRoutePropagation. Without propagation you add the routes by hand. The classic "the VPN is up but nothing goes through" is a route table without the on-premises CIDR, or an on-premises router without a route back to 10.41.0.0/16.
Direct Connect
$ aws directconnect describe-connections --query 'connections[].[connectionId,connectionName,connectionState,location,bandwidth]' --output text
dxcon-ea861018 dc-fra-dx ordering EqFA5 1Gbps
A Direct Connect connection is a physical port at a Direct Connect location (EqFA5, an Equinix data centre in Frankfurt), 1, 10, 100 or 400 Gbps dedicated, or smaller hosted connections from a partner. On it you create virtual interfaces: a private VIF to a VGW or a Direct Connect gateway (which reaches VPCs in many Regions), a transit VIF to a Transit Gateway, a public VIF for public AWS endpoints (S3, DynamoDB) over the private line. It is not encrypted by itself: run a VPN over it (or MACsec on dedicated 10/100 Gbps ports) when the data needs encryption. For resilience use two connections at two locations (the resilience toolkit models it), with a Site-to-Site VPN as the cheaper backup.
PrivateLink: one service, not a network
AWS II used interface endpoints to reach AWS services (SSM, ECR) without a NAT gateway. The same mechanism exposes your own service: put it behind a Network Load Balancer, create a VPC endpoint service on that NLB, and allow the consumer accounts. Consumers create an interface endpoint in their VPC and reach only that service, over private IPs, even with overlapping CIDRs, and with no routes between the networks. SaaS providers use it, and so do platform teams that publish an internal service to a hundred account VPCs. Compared with peering or a TGW, PrivateLink exposes one port, works only in one direction (consumer to provider), and costs per endpoint-hour and per GB.
| need | use |
|---|---|
| office or data centre, quickly, encrypted over the internet | Site-to-Site VPN (to a VGW or TGW) |
| predictable bandwidth / latency, large volumes, lower data-out price | Direct Connect (+ VPN backup) |
| many VPCs and on-premises, one hub | Transit Gateway (VPN and DX attach to it) |
| expose one service to other VPCs / accounts | PrivateLink (endpoint service + NLB) |
| users' laptops into a VPC | AWS Client VPN |
| on-premises DNS for VPC names (and back) | Route 53 Resolver inbound / outbound endpoints |
On the exam (SOA-C03 5.1 and 5.3, SAA-C03 1.2 and 3.4, CLF-C02 3.5): "connect the data centre quickly and securely over the internet" = Site-to-Site VPN; "consistent network performance, dedicated" = Direct Connect; "encrypt Direct Connect traffic" = VPN over DX (or MACsec); "DX resilience at low cost" = a VPN as backup; "many VPCs + on premises" = Transit Gateway; "share a service with other accounts without exposing the VPC, overlapping CIDRs" = PrivateLink; "VPN up but no traffic" = route propagation / route tables. VPN connections always have two tunnels.
In an interview: "Users in the office lose access to an internal app in AWS for a few minutes every few weeks. The VPN shows both tunnels configured. What would you look at?" - "Whether the customer router really uses both tunnels: many configurations bring up only one, so AWS's tunnel maintenance becomes an outage. TunnelState per tunnel in CloudWatch around the incidents, the VPN connection's tunnel status history, and BGP. With static routing, failover between tunnels depends on the router noticing a dead tunnel (dead peer detection). Then alarms on each tunnel and a maintenance runbook."
You can now: read a Site-to-Site VPN's gateways, routes and tunnel telemetry, spot a connection running on one tunnel and alarm on it, check route propagation and the path to on-premises with the Reachability Analyzer, and choose between VPN, Direct Connect, Transit Gateway, PrivateLink and Client VPN for a hybrid requirement.