OnCallReadyStart learning

AWS Lesson 38.37 · AWS V: Data, Resilience & the Edge · 16 min read

Many VPCs: peering, Transit Gateway and the Reachability Analyzer

In plain words

A VPC peering is a private corridor between two buildings. It only joins those two: you cannot walk from building A through B into C, so every pair that must talk needs its own corridor. With many buildings that becomes a maze.

A Transit Gateway is a central station instead. Every building connects to it once, and the station's timetable, its route tables, says who may travel where. The Reachability Analyzer is a map checker: it reads the corridors, doors and timetables and tells you where the path breaks, without anyone walking it.

Many VPCs: peering, Transit Gateway and the Reachability Analyzer

One VPC per team, per environment or per account is the normal shape of an AWS estate, and sooner or later the orders service needs to call payments in another VPC. There are two ways to join VPCs: VPC peering (a private link between exactly two VPCs) and Transit Gateway (a regional router that many VPCs and VPNs attach to). Both fail in the same quiet way when a route is missing: packets go nowhere, and the client times out. The Reachability Analyzer tells you which hop is wrong without sending a packet.

Need to know: peering needs non-overlapping CIDRs, an accept by the other side, and routes in both VPCs' route tables (plus security groups that allow the other CIDR). Peering is not transitive: A-B and B-C do not make A-C. A Transit Gateway is transitive: attachments associate with a TGW route table (which decides where their traffic goes) and propagate their CIDRs into route tables; a static blackhole route drops traffic on purpose.

Three VPCs, two peerings

$ aws ec2 describe-vpcs --filters Name=tag:Name,Values=orders-vpc,payments-vpc,reports-vpc --query 'Vpcs[].[Tags[?Key==`Name`]|[0].Value,CidrBlock,VpcId]' --output text
orders-vpc	10.40.0.0/16	vpc-0fab6173e91281414
payments-vpc	10.41.0.0/16	vpc-0ecba550584326ac5
reports-vpc	10.42.0.0/16	vpc-07c7882177669541a
$ aws ec2 describe-vpc-peering-connections --query 'VpcPeeringConnections[].[VpcPeeringConnectionId,RequesterVpcInfo.CidrBlock,AccepterVpcInfo.CidrBlock,Status.Code]' --output text
pcx-06b9f2d8b91a84275	10.40.0.0/16	10.41.0.0/16	active
pcx-09457911658456530	10.41.0.0/16	10.42.0.0/16	active
$ aws ec2 describe-route-tables --filters Name=tag:Name,Values=orders-rt --query 'RouteTables[0].Routes[].[DestinationCidrBlock,GatewayId || VpcPeeringConnectionId || TransitGatewayId,State]' --output text
10.40.0.0/16	local	active
10.41.0.0/16	pcx-06b9f2d8b91a84275	active

orders (10.40.0.0/16) is peered with payments (10.41.0.0/16), and payments with reports (10.42.0.0/16). orders' route table sends 10.41.0.0/16 into its peering connection. The local route is the VPC itself.

The Reachability Analyzer

The analyzer models the path from a source (an instance, an ENI, a gateway) to a destination: route tables, security groups, NACLs, peering, TGW route tables. It sends no traffic.

$ I_ORD=$(aws ec2 describe-instances --filters Name=tag:Name,Values=orders-app Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ I_PAY=$(aws ec2 describe-instances --filters Name=tag:Name,Values=payments-app Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ I_REP=$(aws ec2 describe-instances --filters Name=tag:Name,Values=reports-app Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ P=$(aws ec2 create-network-insights-path --source $I_ORD --destination $I_PAY --protocol tcp --destination-port 443 --query NetworkInsightsPath.NetworkInsightsPathId --output text)
$ A=$(aws ec2 start-network-insights-analysis --network-insights-path-id $P --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text) && sleep 6
$ aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $A --query 'NetworkInsightsAnalyses[0].[Status,NetworkPathFound]' --output text
succeeded	True
$ P2=$(aws ec2 create-network-insights-path --source $I_ORD --destination $I_REP --protocol tcp --destination-port 443 --query NetworkInsightsPath.NetworkInsightsPathId --output text)
$ A2=$(aws ec2 start-network-insights-analysis --network-insights-path-id $P2 --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text) && sleep 6
$ aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $A2 --query 'NetworkInsightsAnalyses[0].[NetworkPathFound,Explanations[0].Explanation]' --output text
False	(simulator) route table rtb-0a6532259866993ca has no route to 10.42.1.28

orders reaches payments. orders does not reach reports, and the explanation names the first missing piece: orders' route table has no route to 10.42.0.0/16. Adding one through the payments peering would not help: payments is in the middle, but a peering connection only carries traffic between its own two VPCs. No route through payments makes it transitive. Each pair that must talk needs its own peering, so N VPCs need up to N(N-1)/2 peerings, each with routes on both sides. That is where peering stops scaling.

Peering rules worth knowing

$ aws ec2 create-vpc-peering-connection --vpc-id $(aws ec2 describe-vpcs --filters Name=tag:Name,Values=orders-vpc --query 'Vpcs[0].VpcId' --output text) --peer-vpc-id $(aws ec2 describe-vpcs --filters Name=tag:Name,Values=legacy-vpc --query 'Vpcs[0].VpcId' --output text) --query 'VpcPeeringConnection.VpcPeeringConnectionId' --output text
pcx-098a1b117fb909562
$ sleep 5
$ aws ec2 describe-vpc-peering-connections --filters Name=status-code,Values=failed --query 'VpcPeeringConnections[].Status' --output json
[
    {
        "Code": "failed",
        "Message": "Failed due to incorrect VPC-ID, Account ID, or overlapping CIDR range"
    }
]
  • No overlapping CIDRs: legacy-vpc also uses 10.40.0.0/16, and the request fails. A company-wide IP address plan (or IPAM) prevents this. Afterwards, the only fix is renumbering.
  • Peering works across accounts and Regions; the other side must accept (pending-acceptance until then).
  • Security groups can reference a group in a peered VPC of the same Region. Otherwise allow the peer's CIDR.
  • DNS: to resolve the peer's private DNS names to private IPs, enable DNS resolution support on the peering.
  • No edge-to-edge routing: a peer cannot use your internet gateway, NAT gateway, VPN or Direct Connect.
  • Peering itself costs nothing per hour; data across AZs and Regions is billed per GB.

Transit Gateway: a router in the middle

A Transit Gateway already exists (hub-tgw), with all three VPCs attached:

$ TGW=$(aws ec2 describe-transit-gateways --filters Name=tag:Name,Values=hub-tgw --query 'TransitGateways[0].TransitGatewayId' --output text)
$ aws ec2 describe-transit-gateway-attachments --filters Name=transit-gateway-id,Values=$TGW --query 'TransitGatewayAttachments[].[ResourceId,ResourceType,State,Association.TransitGatewayRouteTableId]' --output text
vpc-0fab6173e91281414	vpc	available	tgw-rtb-01e46a826419c1f46
vpc-0ecba550584326ac5	vpc	available	tgw-rtb-01e46a826419c1f46
vpc-07c7882177669541a	vpc	available	tgw-rtb-01e46a826419c1f46
$ TRT=$(aws ec2 describe-transit-gateways --transit-gateway-ids $TGW --query 'TransitGateways[0].Options.AssociationDefaultRouteTableId' --output text)
$ aws ec2 search-transit-gateway-routes --transit-gateway-route-table-id $TRT --filters Name=state,Values=active,blackhole --query 'Routes[].[DestinationCidrBlock,Type,State,TransitGatewayAttachments[0].ResourceId]' --output text
10.40.0.0/16	propagated	active	vpc-0fab6173e91281414
10.41.0.0/16	propagated	active	vpc-0ecba550584326ac5
10.42.0.0/16	propagated	active	vpc-07c7882177669541a

Each VPC attachment associates with the default TGW route table and propagates its CIDR into it, so the table knows all three VPCs. The TGW does its part; each VPC's route table still has to send traffic for the others to the TGW:

$ RT_ORD=$(aws ec2 describe-route-tables --filters Name=tag:Name,Values=orders-rt --query 'RouteTables[0].RouteTableId' --output text)
$ RT_REP=$(aws ec2 describe-route-tables --filters Name=tag:Name,Values=reports-rt --query 'RouteTables[0].RouteTableId' --output text)
$ aws ec2 create-route --route-table-id $RT_ORD --destination-cidr-block 10.42.0.0/16 --transit-gateway-id $TGW
{
    "Return": true
}
$ aws ec2 create-route --route-table-id $RT_REP --destination-cidr-block 10.40.0.0/16 --transit-gateway-id $TGW
{
    "Return": true
}
$ A3=$(aws ec2 start-network-insights-analysis --network-insights-path-id $P2 --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text) && sleep 6
$ aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $A3 --query 'NetworkInsightsAnalyses[0].NetworkPathFound' --output text
True

orders now reaches reports through the hub. A blackhole route in a TGW route table drops traffic for a CIDR on purpose (to cut off a VPC), and a route whose attachment was deleted becomes a blackhole by itself. Separate route tables give segmentation: prod and dev attachments associate with different tables and only propagate into the ones that may reach them. A shared-services table sees everything.

On the exam (SOA-C03 5.1 and 5.3, SAA-C03 3.4 and 4.4): "connect two VPCs privately, cheapest" = peering; "connect many VPCs and on-premises networks, transitive, central" = Transit Gateway; "VPC A can reach B and B can reach C, but A cannot reach C over peering" = non-transitive; "peering fails" = overlapping CIDRs; "find which component blocks the path, without sending traffic" = Reachability Analyzer; "see what was accepted or rejected" = VPC Flow Logs. TGW costs per attachment-hour and per GB; peering only per GB.

In an interview: "Two services in peered VPCs time out talking to each other. How do you debug it?" - "Reachability Analyzer from the client's ENI to the server's on the port: it names the missing piece. By hand, in order: is the peering active, does each side's route table have the other's CIDR pointing at the peering, do the security groups allow the peer's CIDR or group on the port, do the NACLs allow it both ways (ephemeral ports back), and do the CIDRs overlap? Flow logs on the server's ENI show whether packets arrive and are rejected."

You can now: read VPC peerings and the routes they need, explain why peering is not transitive and why overlapping CIDRs block it, run a Reachability Analyzer analysis and read its explanation, and route VPCs through a Transit Gateway with associations, propagations and blackholes.

Why it helps

One VPC per team, environment or account is the normal shape of an AWS estate, and sooner or later one service must call another across VPCs. When a route is missing the packets simply go nowhere and the client times out, with no error to read.

Knowing that peering is not transitive, that overlapping CIDRs can only be fixed by renumbering, and how Transit Gateway associations and propagations work, plus a tool that names the missing hop, saves hours of guessing.

Commands in this lesson

aws sleep

FAQ

Why can A not reach C when A is peered with B and B with C?

Peering is not transitive. A peering connection only delivers packets addressed to the peer VPC's own CIDR, so B never forwards traffic from A to C, whatever the route tables say. Each pair that must talk needs its own peering, up to N(N-1)/2 of them, or all of them join a Transit Gateway.

Why did my peering request fail?

Most often the CIDRs overlap: in the lab, legacy-vpc uses 10.40.0.0/16 like orders-vpc, and the request ends in the failed state. There is no fix except renumbering one side, which is why a company-wide IP address plan (or IPAM) matters before the second VPC is created.

What else does a peering need besides being active?

The other side must accept it; until then it is pending-acceptance. Each VPC's route table needs a route for the other's CIDR pointing at the peering connection, and security groups and NACLs must allow the peer's CIDR, or its security group in the same Region. For private DNS names, enable DNS resolution on the peering.

What do association and propagation mean on a Transit Gateway?

An attachment associates with one TGW route table, which decides where its traffic goes. It propagates its CIDR into one or more route tables, so they know how to reach it. Separate route tables give segmentation: prod and dev associate with different tables and propagate only where they may be reached. Each VPC still needs routes pointing at the TGW.

What is a blackhole route?

A route in a TGW route table that drops traffic for a CIDR instead of forwarding it. You add one on purpose to cut off a VPC, and a route whose attachment was deleted becomes a blackhole by itself. search-transit-gateway-routes with state blackhole lists them, and they are a common cause of "it worked yesterday".

In an interview Mid

Two services in peered VPCs time out talking to each other. How do you debug it?

Run the Reachability Analyzer from the client's ENI to the server's on the port: it names the missing piece without sending traffic. By hand, in order: is the peering active, does each side's route table send the other's CIDR to the peering, do the security groups allow the peer's CIDR or group on the port, do the NACLs allow it both ways including ephemeral ports back, and do the CIDRs overlap? VPC Flow Logs on the server's ENI show whether packets arrive and are rejected.

Also asked: When would you choose a Transit Gateway over VPC peering? · Why can two VPCs with overlapping CIDRs not be peered? · How do you separate production and development VPCs on a Transit Gateway?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.