Many VPCs: peering, Transit Gateway and the Reachability Analyzer
One VPC per team, per environment or per account is the normal shape of an AWS estate, and sooner or later the orders service needs to call payments in another VPC. There are two ways to join VPCs: VPC peering (a private link between exactly two VPCs) and Transit Gateway (a regional router that many VPCs and VPNs attach to). Both fail in the same quiet way when a route is missing: packets go nowhere, and the client times out. The Reachability Analyzer tells you which hop is wrong without sending a packet.
Need to know: peering needs non-overlapping CIDRs, an accept by the other side, and routes in both VPCs' route tables (plus security groups that allow the other CIDR). Peering is not transitive: A-B and B-C do not make A-C. A Transit Gateway is transitive: attachments associate with a TGW route table (which decides where their traffic goes) and propagate their CIDRs into route tables; a static blackhole route drops traffic on purpose.
Three VPCs, two peerings
$ aws ec2 describe-vpcs --filters Name=tag:Name,Values=orders-vpc,payments-vpc,reports-vpc --query 'Vpcs[].[Tags[?Key==`Name`]|[0].Value,CidrBlock,VpcId]' --output text
orders-vpc 10.40.0.0/16 vpc-0fab6173e91281414
payments-vpc 10.41.0.0/16 vpc-0ecba550584326ac5
reports-vpc 10.42.0.0/16 vpc-07c7882177669541a
$ aws ec2 describe-vpc-peering-connections --query 'VpcPeeringConnections[].[VpcPeeringConnectionId,RequesterVpcInfo.CidrBlock,AccepterVpcInfo.CidrBlock,Status.Code]' --output text
pcx-06b9f2d8b91a84275 10.40.0.0/16 10.41.0.0/16 active
pcx-09457911658456530 10.41.0.0/16 10.42.0.0/16 active
$ aws ec2 describe-route-tables --filters Name=tag:Name,Values=orders-rt --query 'RouteTables[0].Routes[].[DestinationCidrBlock,GatewayId || VpcPeeringConnectionId || TransitGatewayId,State]' --output text
10.40.0.0/16 local active
10.41.0.0/16 pcx-06b9f2d8b91a84275 active
orders (10.40.0.0/16) is peered with payments (10.41.0.0/16), and payments with reports (10.42.0.0/16). orders' route table sends 10.41.0.0/16 into its peering connection. The local route is the VPC itself.
The Reachability Analyzer
The analyzer models the path from a source (an instance, an ENI, a gateway) to a destination: route tables, security groups, NACLs, peering, TGW route tables. It sends no traffic.
$ I_ORD=$(aws ec2 describe-instances --filters Name=tag:Name,Values=orders-app Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ I_PAY=$(aws ec2 describe-instances --filters Name=tag:Name,Values=payments-app Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ I_REP=$(aws ec2 describe-instances --filters Name=tag:Name,Values=reports-app Name=instance-state-name,Values=running --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ P=$(aws ec2 create-network-insights-path --source $I_ORD --destination $I_PAY --protocol tcp --destination-port 443 --query NetworkInsightsPath.NetworkInsightsPathId --output text)
$ A=$(aws ec2 start-network-insights-analysis --network-insights-path-id $P --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text) && sleep 6
$ aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $A --query 'NetworkInsightsAnalyses[0].[Status,NetworkPathFound]' --output text
succeeded True
$ P2=$(aws ec2 create-network-insights-path --source $I_ORD --destination $I_REP --protocol tcp --destination-port 443 --query NetworkInsightsPath.NetworkInsightsPathId --output text)
$ A2=$(aws ec2 start-network-insights-analysis --network-insights-path-id $P2 --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text) && sleep 6
$ aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $A2 --query 'NetworkInsightsAnalyses[0].[NetworkPathFound,Explanations[0].Explanation]' --output text
False (simulator) route table rtb-0a6532259866993ca has no route to 10.42.1.28
orders reaches payments. orders does not reach reports, and the explanation names the first missing piece: orders' route table has no route to 10.42.0.0/16. Adding one through the payments peering would not help: payments is in the middle, but a peering connection only carries traffic between its own two VPCs. No route through payments makes it transitive. Each pair that must talk needs its own peering, so N VPCs need up to N(N-1)/2 peerings, each with routes on both sides. That is where peering stops scaling.
Peering rules worth knowing
$ aws ec2 create-vpc-peering-connection --vpc-id $(aws ec2 describe-vpcs --filters Name=tag:Name,Values=orders-vpc --query 'Vpcs[0].VpcId' --output text) --peer-vpc-id $(aws ec2 describe-vpcs --filters Name=tag:Name,Values=legacy-vpc --query 'Vpcs[0].VpcId' --output text) --query 'VpcPeeringConnection.VpcPeeringConnectionId' --output text
pcx-098a1b117fb909562
$ sleep 5
$ aws ec2 describe-vpc-peering-connections --filters Name=status-code,Values=failed --query 'VpcPeeringConnections[].Status' --output json
[
{
"Code": "failed",
"Message": "Failed due to incorrect VPC-ID, Account ID, or overlapping CIDR range"
}
]
- No overlapping CIDRs:
legacy-vpcalso uses 10.40.0.0/16, and the request fails. A company-wide IP address plan (or IPAM) prevents this. Afterwards, the only fix is renumbering. - Peering works across accounts and Regions; the other side must accept (
pending-acceptanceuntil then). - Security groups can reference a group in a peered VPC of the same Region. Otherwise allow the peer's CIDR.
- DNS: to resolve the peer's private DNS names to private IPs, enable DNS resolution support on the peering.
- No edge-to-edge routing: a peer cannot use your internet gateway, NAT gateway, VPN or Direct Connect.
- Peering itself costs nothing per hour; data across AZs and Regions is billed per GB.
Transit Gateway: a router in the middle
A Transit Gateway already exists (hub-tgw), with all three VPCs attached:
$ TGW=$(aws ec2 describe-transit-gateways --filters Name=tag:Name,Values=hub-tgw --query 'TransitGateways[0].TransitGatewayId' --output text)
$ aws ec2 describe-transit-gateway-attachments --filters Name=transit-gateway-id,Values=$TGW --query 'TransitGatewayAttachments[].[ResourceId,ResourceType,State,Association.TransitGatewayRouteTableId]' --output text
vpc-0fab6173e91281414 vpc available tgw-rtb-01e46a826419c1f46
vpc-0ecba550584326ac5 vpc available tgw-rtb-01e46a826419c1f46
vpc-07c7882177669541a vpc available tgw-rtb-01e46a826419c1f46
$ TRT=$(aws ec2 describe-transit-gateways --transit-gateway-ids $TGW --query 'TransitGateways[0].Options.AssociationDefaultRouteTableId' --output text)
$ aws ec2 search-transit-gateway-routes --transit-gateway-route-table-id $TRT --filters Name=state,Values=active,blackhole --query 'Routes[].[DestinationCidrBlock,Type,State,TransitGatewayAttachments[0].ResourceId]' --output text
10.40.0.0/16 propagated active vpc-0fab6173e91281414
10.41.0.0/16 propagated active vpc-0ecba550584326ac5
10.42.0.0/16 propagated active vpc-07c7882177669541a
Each VPC attachment associates with the default TGW route table and propagates its CIDR into it, so the table knows all three VPCs. The TGW does its part; each VPC's route table still has to send traffic for the others to the TGW:
$ RT_ORD=$(aws ec2 describe-route-tables --filters Name=tag:Name,Values=orders-rt --query 'RouteTables[0].RouteTableId' --output text)
$ RT_REP=$(aws ec2 describe-route-tables --filters Name=tag:Name,Values=reports-rt --query 'RouteTables[0].RouteTableId' --output text)
$ aws ec2 create-route --route-table-id $RT_ORD --destination-cidr-block 10.42.0.0/16 --transit-gateway-id $TGW
{
"Return": true
}
$ aws ec2 create-route --route-table-id $RT_REP --destination-cidr-block 10.40.0.0/16 --transit-gateway-id $TGW
{
"Return": true
}
$ A3=$(aws ec2 start-network-insights-analysis --network-insights-path-id $P2 --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text) && sleep 6
$ aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $A3 --query 'NetworkInsightsAnalyses[0].NetworkPathFound' --output text
True
orders now reaches reports through the hub. A blackhole route in a TGW route table drops traffic for a CIDR on purpose (to cut off a VPC), and a route whose attachment was deleted becomes a blackhole by itself. Separate route tables give segmentation: prod and dev attachments associate with different tables and only propagate into the ones that may reach them. A shared-services table sees everything.
On the exam (SOA-C03 5.1 and 5.3, SAA-C03 3.4 and 4.4): "connect two VPCs privately, cheapest" = peering; "connect many VPCs and on-premises networks, transitive, central" = Transit Gateway; "VPC A can reach B and B can reach C, but A cannot reach C over peering" = non-transitive; "peering fails" = overlapping CIDRs; "find which component blocks the path, without sending traffic" = Reachability Analyzer; "see what was accepted or rejected" = VPC Flow Logs. TGW costs per attachment-hour and per GB; peering only per GB.
In an interview: "Two services in peered VPCs time out talking to each other. How do you debug it?" - "Reachability Analyzer from the client's ENI to the server's on the port: it names the missing piece. By hand, in order: is the peering active, does each side's route table have the other's CIDR pointing at the peering, do the security groups allow the peer's CIDR or group on the port, do the NACLs allow it both ways (ephemeral ports back), and do the CIDRs overlap? Flow logs on the server's ENI show whether packets arrive and are rejected."
You can now: read VPC peerings and the routes they need, explain why peering is not transitive and why overlapping CIDRs block it, run a Reachability Analyzer analysis and read its explanation, and route VPCs through a Transit Gateway with associations, propagations and blackholes.