OnCallReadyStart learning

AWSNetworkingLinux · 5 min read

EC2 instance can't reach the internet: route tables, NAT gateways and blackhole routes

apt update hangs, Session Manager says TargetNotConnected. Read the private route table, spot the blackhole, rebuild the NAT gateway in a public subnet.

The nightly workers in the private subnets failed. The newest one never finished its user data, and you cannot even get a shell on it:

terminal
$ aws ssm start-session --target $W3
aws: [ERROR]: An error occurred (TargetNotConnected) when calling the StartSession operation: i-04733909647448553 is not connected.

TargetNotConnected does not mean the instance is down. It means the SSM agent on it has not checked in with the Session Manager service, and the agent reaches that service over the network like any other client: through the internet or through VPC endpoints. So the first symptom of a private subnet that lost its way out is often that you lose your way in.

What the instance printed while it booted

No shell, no SSH. The serial console log still works, because EC2 collects it from outside the instance:

terminal
$ aws ec2 get-console-output --instance-id $W3 --latest --output text | grep -m4 -i 'cloud-init.*\(err\|fail\|could not\|unable\|ign\)'
[   10.393000] cloud-init[1044]: Ign:1 http://eu-central-1.ec2.ports.ubuntu.com/ubuntu-ports resolute InRelease
[   10.811300] cloud-init[1044]: Ign:2 http://eu-central-1.ec2.ports.ubuntu.com/ubuntu-ports resolute-updates InRelease
[   11.229600] cloud-init[1044]: Ign:3 http://eu-central-1.ec2.ports.ubuntu.com/ubuntu-ports resolute-backports InRelease
[   11.647900] cloud-init[1044]: Ign:4 http://eu-central-1.ec2.ports.ubuntu.com/ubuntu-ports resolute-security InRelease
$ aws ec2 get-console-output --instance-id $W3 --latest --output text | tail -12 | head -6
[   16.249200] cloud-init[1044]: W: Some index files failed to download. They have been ignored, or old ones used instead.
[   16.667500] cloud-init[1044]: Reading package lists...
[   17.085800] cloud-init[1044]: Building dependency tree...
[   17.504100] cloud-init[1044]: Reading state information...
[   17.922400] cloud-init[1044]: E: Unable to locate package jq
[   18.340700] cloud-init[1044]: 2026-09-22 19:51:40,000 - cc_scripts_user.py[WARNING]: Failed to run module scripts_user (scripts in /var/lib/cloud/instance/scripts)

apt-get update could not fetch a single index, so apt-get install jq found nothing, so the user-data script failed. "Unable to locate package" on a fresh instance almost always means "could not reach the mirror", not a typo.

The instance has no public address, by design

terminal
$ aws ec2 describe-instances --instance-ids $W3 --query 'Reservations[0].Instances[0].[PrivateIpAddress,PublicIpAddress,SubnetId]' --output text
10.20.10.67	None	subnet-093b5a8079a3fabfe
$ VPC=$(aws ec2 describe-instances --instance-ids $W3 --query 'Reservations[0].Instances[0].VpcId' --output text)
$ aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC --query 'Subnets[].[Tags[?Key==`Name`]|[0].Value,CidrBlock,MapPublicIpOnLaunch]' --output text
shop-public-a	10.20.0.0/24	True
shop-public-b	10.20.1.0/24	True
shop-private-a	10.20.10.0/24	False
shop-private-b	10.20.11.0/24	False

"Public" and "private" are not settings on a subnet. A subnet is public when its route table sends 0.0.0.0/0 to an internet gateway, and its instances have public IPs. A private subnet sends 0.0.0.0/0 to a NAT gateway, which sits in a public subnet and translates the private addresses to its own Elastic IP on the way out. Replies come back to the NAT; nothing from the internet can start a connection in.

Read the route table the subnet actually uses

terminal
$ RT=$(aws ec2 describe-route-tables --filters Name=tag:Name,Values=shop-private --query 'RouteTables[0].RouteTableId' --output text)
$ aws ec2 describe-route-tables --route-table-ids $RT --query 'RouteTables[0].Routes[].[DestinationCidrBlock,NatGatewayId,State]' --output text
10.20.0.0/16	None	active
0.0.0.0/0	nat-07bc87b57a6297226	blackhole
$ aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,State,SubnetId]' --output text
nat-07bc87b57a6297226	deleted	subnet-07d93f499311787f9

blackhole is the route's own word for "the target is gone": packets for the internet are dropped at the first hop. Yesterday's cost clean-up deleted the NAT gateway as an "unused resource", and the route stayed behind pointing at it. (NAT gateways do cost money per hour and per gigabyte, which is exactly why they get deleted, and why S3 traffic should not go through one.)

Compare with the public subnets' table, which goes straight to the internet gateway:

terminal
$ aws ec2 describe-route-tables --filters Name=tag:Name,Values=shop-public --query 'RouteTables[0].Routes[].[DestinationCidrBlock,GatewayId,State]' --output text
10.20.0.0/16	local	active
0.0.0.0/0	igw-0342e2f2941a7c782	active

Rebuild the way out, the way it was designed

A new NAT gateway, in a public subnet, with an Elastic IP, then point the private route at it. Putting the NAT in a private subnet is the classic second mistake: it would need a NAT itself.

terminal
$ PUB=$(aws ec2 describe-subnets --filters Name=tag:Name,Values=shop-public-a --query 'Subnets[0].SubnetId' --output text)
$ EIP=$(aws ec2 allocate-address --domain vpc --query AllocationId --output text)
$ NAT=$(aws ec2 create-nat-gateway --subnet-id $PUB --allocation-id $EIP --tag-specifications 'ResourceType=natgateway,Tags=[{Key=Name,Value=shop-nat}]' --query NatGateway.NatGatewayId --output text)
$ aws ec2 wait nat-gateway-available --nat-gateway-ids $NAT
$ aws ec2 replace-route --route-table-id $RT --destination-cidr-block 0.0.0.0/0 --nat-gateway-id $NAT
$ aws ec2 describe-route-tables --route-table-ids $RT --query 'RouteTables[0].Routes[].[DestinationCidrBlock,State]' --output text
10.20.0.0/16	active
0.0.0.0/0	active

Then prove it from the instance itself. The agent reconnects on its own once it can reach the service:

terminal
$ aws ssm start-session --target $W3

Starting session with SessionId: learner-075c7a146a86196a2
$ sudo apt-get update
Hit:1 http://ports.ubuntu.com/ubuntu-ports resolute InRelease
Get:2 http://ports.ubuntu.com/ubuntu-ports resolute-updates InRelease [126 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports resolute-backports InRelease [126 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports resolute-security InRelease [126 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports resolute-updates/main arm64 Packages [512 kB]
Fetched 890 kB in 2s (471 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
7 packages can be upgraded. Run 'apt list --upgradable' to see them.

Do not "fix" it by giving the workers public IPs in the private subnet: without a route to an internet gateway those addresses do nothing, and moving the workers into a public subnet exposes them for the sake of apt.

When the routes are fine

If the route says active and it still fails, walk the rest of the path:

  • Network ACLs on the private and the public subnet: stateless, so the outbound rule is not enough; the return traffic on ephemeral ports (1024-65535) must be allowed inbound. A NACL that drops replies makes everything time out, which from a load balancer looks like a 504.
  • Security group egress: the default allows all outbound; hardened groups often do not.
  • DNS: enableDnsSupport on the VPC, and the instance's resolver. "Could not resolve" is a different failure from "connection timed out", and the DNS hunt has its own order.
  • IPv6: a private IPv6 subnet needs an egress-only internet gateway, not a NAT gateway.
  • aws ec2 create-network-insights-path (Reachability Analyzer) checks routes, NACLs and security groups between two points without sending a packet.

And if the instance should never need the internet at all, give it VPC endpoints instead: a free gateway endpoint for S3, and interface endpoints for ssm and ssmmessages (plus ec2messages for older SSM agents) so Session Manager works with no NAT. Endpoint policies are one more place where an S3 AccessDenied can come from.

Practise it

Incident: the private instances cannot reach the internet is this outage, from the console output to apt-get update on the worker. No internet on purpose: Session Manager and S3 through VPC endpoints builds the endpoint version, Incident: after the hardening, every request times out is the NACL that eats the replies, and The NAT gateway bill explains the invoice that started the clean-up.

OnCallReady is free, with no ads and no tracking. RSS · All posts