OnCallReadyStart learning

AWS Lesson 38.41 · AWS V: Data, Resilience & the Edge · 14 min read

Hybrid networks: Site-to-Site VPN, Direct Connect and PrivateLink

In plain words

There are three ways to join your office to AWS. A Site-to-Site VPN is a sealed tube laid along the public road, the internet: ready in minutes, but only as smooth as the road. Direct Connect is your own private road: weeks to build, and then the traffic is predictable. PrivateLink is a hatch in a wall for one counter, not a door into the whole building.

A VPN always comes as two tubes. If your side only uses one, the day AWS services that tube is the day the office goes dark.

Hybrid networks: Site-to-Site VPN, Direct Connect and PrivateLink

Most companies that run on AWS also have a data centre, an office, or a partner network that must reach the VPCs privately: the payments service calls a mainframe, the finance team's reports read a database on premises. Three tools cover it. Site-to-Site VPN sends encrypted IPsec tunnels over the internet, works in minutes, and is limited by the internet's quality. Direct Connect is a dedicated, private fibre connection, with predictable latency and bandwidth, that takes weeks to order. PrivateLink exposes one service, not a whole network, to other VPCs and accounts.

Need to know: a VPN connection joins a customer gateway (your router's public IP and BGP ASN) to a virtual private gateway (on one VPC) or a Transit Gateway (many VPCs), always as two tunnels to two AWS endpoints. Routes are static or learned by BGP, and reach the VPC route tables through route propagation. Direct Connect uses virtual interfaces (private, public, transit) and often a Direct Connect gateway. A VPN over the internet is the usual backup for it.

The VPN to the data centre

payments-vpc (10.41.0.0/16) has a VPN to the data centre in Frankfurt (192.168.100.0/24, router at 203.0.113.12):

$ aws ec2 describe-customer-gateways --query 'CustomerGateways[].[CustomerGatewayId,IpAddress,BgpAsn,State]' --output text
cgw-01f7b879bc3195d95	203.0.113.12	65000	available
$ aws ec2 describe-vpn-gateways --query 'VpnGateways[].[VpnGatewayId,State,VpcAttachments[0].VpcId,AmazonSideAsn]' --output text
vgw-072146817718d7ed9	available	vpc-0fab6173e91281414	64512
$ VPN=$(aws ec2 describe-vpn-connections --filters Name=tag:Name,Values=dc-fra --query 'VpnConnections[0].VpnConnectionId' --output text)
$ aws ec2 describe-vpn-connections --vpn-connection-ids $VPN --query 'VpnConnections[0].{state: State, staticOnly: Options.StaticRoutesOnly, routes: Routes[].[DestinationCidrBlock,State], tunnels: VgwTelemetry[].[OutsideIpAddress,Status,StatusMessage]}'
{
    "state": "available",
    "staticOnly": true,
    "routes": [
        [
            "192.168.100.0/24",
            "available"
        ]
    ],
    "tunnels": [
        [
            "3.120.89.187",
            "UP",
            ""
        ],
        [
            "3.121.89.224",
            "DOWN",
            "IPSEC IS DOWN"
        ]
    ]
}

One tunnel is UP, the other DOWN. That is a common and dangerous state: traffic still flows over the remaining tunnel, so nothing breaks today, and the redundancy is gone. AWS replaces tunnel endpoints during maintenance one at a time, so a single-tunnel setup has outages that the provider announced in advance. Configure both tunnels on the customer router and alarm when either is down:

$ aws cloudwatch get-metric-statistics --namespace AWS/VPN --metric-name TunnelState --dimensions Name=VpnId,Value=$VPN --start-time $(date -u -d '-30 min' +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 300 --statistics Minimum Maximum --query 'sort_by(Datapoints,&Timestamp)[-1].[Minimum,Maximum]' --output text
0.5	0.5

TunnelState is 1 when a tunnel is up and 0 when it is down, per tunnel (TunnelIpAddress dimension). Per connection (VpnId) values between 0 and 1 mean at least one tunnel is not up: 0.5 here, one of two. 0 is a full outage. Alarm when the connection's value drops below 1, or on each tunnel, and send it to the team that owns the customer router.

Routes: propagation, and what the VPC sees

$ RT=$(aws ec2 describe-route-tables --filters Name=tag:Name,Values=payments-rt --query 'RouteTables[0].RouteTableId' --output text)
$ aws ec2 describe-route-tables --route-table-ids $RT --query 'RouteTables[0].{routes: Routes[].[DestinationCidrBlock,GatewayId || VpcPeeringConnectionId,Origin,State], propagating: PropagatingVgws[].GatewayId}'
{
    "routes": [
        [
            "10.41.0.0/16",
            "local",
            "CreateRouteTable",
            "active"
        ],
        [
            "192.168.100.0/24",
            "vgw-072146817718d7ed9",
            "EnableVgwRoutePropagation",
            "active"
        ]
    ],
    "propagating": [
        "vgw-072146817718d7ed9"
    ]
}
$ I_PAY=$(aws ec2 describe-instances --filters Name=tag:Name,Values=payments-app Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ P=$(aws ec2 create-network-insights-path --source $I_PAY --destination-ip 192.168.100.20 --protocol tcp --destination-port 443 --query NetworkInsightsPath.NetworkInsightsPathId --output text)
$ A=$(aws ec2 start-network-insights-analysis --network-insights-path-id $P --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text) && sleep 6
$ aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $A --query 'NetworkInsightsAnalyses[0].NetworkPathFound' --output text
True

With route propagation on, the VGW adds the VPN's routes (static routes, or those learned over BGP) to the route table: Origin: EnableVgwRoutePropagation. Without propagation you add the routes by hand. The classic "the VPN is up but nothing goes through" is a route table without the on-premises CIDR, or an on-premises router without a route back to 10.41.0.0/16.

Direct Connect

$ aws directconnect describe-connections --query 'connections[].[connectionId,connectionName,connectionState,location,bandwidth]' --output text
dxcon-ea861018	dc-fra-dx	ordering	EqFA5	1Gbps

A Direct Connect connection is a physical port at a Direct Connect location (EqFA5, an Equinix data centre in Frankfurt), 1, 10, 100 or 400 Gbps dedicated, or smaller hosted connections from a partner. On it you create virtual interfaces: a private VIF to a VGW or a Direct Connect gateway (which reaches VPCs in many Regions), a transit VIF to a Transit Gateway, a public VIF for public AWS endpoints (S3, DynamoDB) over the private line. It is not encrypted by itself: run a VPN over it (or MACsec on dedicated 10/100 Gbps ports) when the data needs encryption. For resilience use two connections at two locations (the resilience toolkit models it), with a Site-to-Site VPN as the cheaper backup.

PrivateLink: one service, not a network

AWS II used interface endpoints to reach AWS services (SSM, ECR) without a NAT gateway. The same mechanism exposes your own service: put it behind a Network Load Balancer, create a VPC endpoint service on that NLB, and allow the consumer accounts. Consumers create an interface endpoint in their VPC and reach only that service, over private IPs, even with overlapping CIDRs, and with no routes between the networks. SaaS providers use it, and so do platform teams that publish an internal service to a hundred account VPCs. Compared with peering or a TGW, PrivateLink exposes one port, works only in one direction (consumer to provider), and costs per endpoint-hour and per GB.

needuse
office or data centre, quickly, encrypted over the internetSite-to-Site VPN (to a VGW or TGW)
predictable bandwidth / latency, large volumes, lower data-out priceDirect Connect (+ VPN backup)
many VPCs and on-premises, one hubTransit Gateway (VPN and DX attach to it)
expose one service to other VPCs / accountsPrivateLink (endpoint service + NLB)
users' laptops into a VPCAWS Client VPN
on-premises DNS for VPC names (and back)Route 53 Resolver inbound / outbound endpoints

On the exam (SOA-C03 5.1 and 5.3, SAA-C03 1.2 and 3.4, CLF-C02 3.5): "connect the data centre quickly and securely over the internet" = Site-to-Site VPN; "consistent network performance, dedicated" = Direct Connect; "encrypt Direct Connect traffic" = VPN over DX (or MACsec); "DX resilience at low cost" = a VPN as backup; "many VPCs + on premises" = Transit Gateway; "share a service with other accounts without exposing the VPC, overlapping CIDRs" = PrivateLink; "VPN up but no traffic" = route propagation / route tables. VPN connections always have two tunnels.

In an interview: "Users in the office lose access to an internal app in AWS for a few minutes every few weeks. The VPN shows both tunnels configured. What would you look at?" - "Whether the customer router really uses both tunnels: many configurations bring up only one, so AWS's tunnel maintenance becomes an outage. TunnelState per tunnel in CloudWatch around the incidents, the VPN connection's tunnel status history, and BGP. With static routing, failover between tunnels depends on the router noticing a dead tunnel (dead peer detection). Then alarms on each tunnel and a maintenance runbook."

You can now: read a Site-to-Site VPN's gateways, routes and tunnel telemetry, spot a connection running on one tunnel and alarm on it, check route propagation and the path to on-premises with the Reachability Analyzer, and choose between VPN, Direct Connect, Transit Gateway, PrivateLink and Client VPN for a hybrid requirement.

Why it helps

Most companies on AWS also have a data centre, an office or a partner network that must reach the VPCs privately. Hybrid links fail in quiet ways: a tunnel that is down while traffic still flows over the other, a VPN that is up but carries nothing because a route is missing, a Direct Connect line nobody realised was unencrypted.

The lesson reads the gateways, routes and tunnel telemetry, and gives you a table for choosing between VPN, Direct Connect, Transit Gateway and PrivateLink.

Commands in this lesson

aws

FAQ

Why does a VPN connection have two tunnels?

Each tunnel ends at a different AWS endpoint, and AWS replaces tunnel endpoints during maintenance one at a time. With both tunnels configured on the customer router, maintenance is invisible; with one, it is an outage the provider announced in advance. One tunnel UP and one DOWN is a common and dangerous state: it works today with no redundancy.

What does a TunnelState of 0.5 mean?

TunnelState in the AWS/VPN namespace is 1 when a tunnel is up and 0 when it is down, per tunnel. Per connection (the VpnId dimension) the value is the share of tunnels up, so 0.5 means one of two tunnels is down and 0 is a full outage. Alarm when it drops below 1 and send it to the router's owners.

The VPN is up but nothing gets through. Why?

UP only means IPsec is established; traffic still needs routes on both sides. The VPC route table must send the on-premises CIDR to the virtual private gateway, through route propagation or a static route; with static routing the route must also exist on the VPN connection; and the data centre must route the VPC's CIDR back. Then check security groups.

Is Direct Connect traffic encrypted?

No, not by itself: it is a private line, not an encrypted one. When the data needs encryption, run a Site-to-Site VPN over the Direct Connect connection, or use MACsec on dedicated 10 or 100 Gbps ports. For resilience use two connections at two locations, or a VPN over the internet as the cheaper backup.

When is PrivateLink better than peering or a Transit Gateway?

When you want to expose one service, not a network. The provider puts the service behind a Network Load Balancer and creates a VPC endpoint service; consumers create an interface endpoint and reach only that service, over private IPs, even with overlapping CIDRs and no routes between the networks. It works in one direction and costs per endpoint-hour and per GB.

In an interview Mid

Office users lose access to an internal app in AWS for a few minutes every few weeks. The VPN shows both tunnels configured. What would you look at?

Whether the customer router really uses both tunnels: many configurations bring up only one, so AWS's tunnel maintenance becomes an outage. I would read TunnelState per tunnel in CloudWatch around the incidents, the VPN connection's tunnel status history, and the BGP sessions. With static routing, failover between tunnels depends on the router noticing a dead tunnel through dead peer detection. Then alarms on each tunnel, sent to the router's owners, and a runbook for Site-to-Site VPN maintenance.

Also asked: What is the difference between Site-to-Site VPN and Direct Connect? · How would you make a Direct Connect link resilient without doubling the cost? · When would you use PrivateLink instead of VPC peering?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.