OnCallReady

Lesson 30.17 · AWS II: VPC, EC2, ELB & EKS · 12 min read

Inside the instance: IMDSv2, instance roles, Session Manager

In plain words

Every instance has a small information desk inside it that only the instance itself can reach (the metadata service). Ask it "who am I?" and it tells you its ID and its zone; ask "what is my badge?" and it hands over short-lived keys for the instance's role. In the new version of the desk you first have to ask for a ticket and show it with every question, so a trick request from outside cannot slip through.

Session Manager is a phone line the instance itself dials out to, so you can talk to it without any door being open.

Inside the instance: IMDSv2, instance roles, Session Manager

How does code on an instance get AWS credentials without anyone putting keys on it? How do you get a shell on a server with no SSH key, no port 22 and no public address? Both answers run through two services every instance has: the instance metadata service (IMDS) and the SSM agent.

Need to know: IMDS answers at http://169.254.169.254 from inside the instance only: instance ID, Region, network details, user data, and the temporary credentials of the instance role. With IMDSv2 (set HttpTokens=required) a client first PUTs /latest/api/token to get a session token and sends it on every GET; plain GETs get 401. The SDKs and the CLI do this by themselves. Session Manager gives a shell through the SSM agent, which dials out to SSM (via NAT or VPC endpoints) using the instance role's AmazonSSMManagedInstanceCore permissions; your machine needs the Session Manager plugin.

A shell without SSH

$ I=$(aws ec2 describe-instances --filters Name=tag:Name,Values=try-app --query 'Reservations[0].Instances[0].InstanceId' --output text)
$ aws ssm describe-instance-information --filters Key=InstanceIds,Values=$I --query 'InstanceInformationList[0].[InstanceId,PingStatus,AgentVersion,PlatformName,PlatformVersion]' --output text
i-05c48c61b8679868a	Online	3.3.3270.0	Ubuntu	26.04

Online means the agent reached SSM in the last minutes. An instance is missing from that list when one of three things fails, always in this order of likelihood:

  1. the role: no instance profile, or a role without AmazonSSMManagedInstanceCore;
  2. the network: no way to ssm, ssmmessages and ec2messages on 443 (no NAT, no endpoints, an outbound security group or NACL rule missing);
  3. the agent: not installed (custom images), stopped, or far too old.
$ aws ssm start-session --target $I

Starting session with SessionId: learner-0b96ead169c13f2a6
$ whoami
ssm-user
$ hostname
ip-10-99-10-30
$ sudo -n true && echo "ssm-user can sudo"
ssm-user can sudo
$ exit


Exiting session with sessionId: learner-0b96ead169c13f2a6.

Sessions run as ssm-user (with sudo), are logged in CloudTrail (StartSession) and can be recorded to S3 or CloudWatch Logs. IAM decides who may open one, on which instances (by tag conditions), with which document - the controls SSH keys never had. aws ssm start-session --document-name AWS-StartPortForwardingSession forwards a port the same way (a database on an instance, a web UI) without opening anything inbound.

IMDSv2 from inside

$ aws ssm start-session --target $I

Starting session with SessionId: learner-066a85e61b98f40a8
$ curl -s -o /dev/null -w '%{http_code}\n' http://169.254.169.254/latest/meta-data/instance-id
401
$ TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600')
$ curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
iam/
identity-credentials/
instance-action
instance-id
instance-life-cycle
instance-type
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
reservation-id
security-groups
services/
system
$ curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/availability-zone
eu-central-1a
$ curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/
oncall-web-ec2
$ exit


Exiting session with sessionId: learner-066a85e61b98f40a8.

The 401 is IMDSv2 doing its job. The token request is a PUT with a TTL header (1 s to 6 h); the token only works on this instance. iam/security-credentials/ lists the role name; one more level returns AccessKeyId, SecretAccessKey, Token and Expiration - credentials AWS rotates every few hours. That is the whole "instance profile" mechanism: the SDK's credential chain ends at IMDS (after env vars and config files), so code on the instance finds the role without configuration.

Why v1 had to go

IMDSv1 answered any GET from the instance. In 2019 a misconfigured web application firewall at Capital One could be tricked into requesting http://169.254.169.254/latest/meta-data/iam/security- credentials/<role> on an attacker's behalf (a server-side request forgery, SSRF) - and returned the role's keys, which then read 100 million customers' records from S3. IMDSv2 breaks that class of attack twice: an SSRF usually cannot send a PUT with a custom header, and the token response carries a hop limit (IP TTL) so it does not survive being forwarded.

SettingValuesUse
HttpTokensoptional (v1 + v2), required (v2 only)always required; new accounts can default to it
HttpPutResponseHopLimit1-641 = only the instance itself; 2 = also containers on it (Docker, EKS pods via the node)
HttpEndpointenabled / disableddisable when nothing on the instance needs it
$ aws ec2 modify-instance-metadata-options --instance-id $I --http-tokens required --http-put-response-hop-limit 1 --query 'InstanceMetadataOptions.[HttpTokens,HttpPutResponseHopLimit,State]' --output text
required	1	pending

Hop limit 1 on EKS nodes is a security control in its own right: pods then cannot reach the node's role through IMDS, so a pod without its own role gets nothing instead of the node's permissions (the EKS identity lesson).

In an interview: "How does an application on EC2 get AWS credentials?" - from the instance profile: the role's temporary credentials are served by the instance metadata service, which the SDK reads at the end of its credential chain; with IMDSv2 it first PUTs for a session token. No keys on disk, rotated automatically.

Keys and SSH: when they are still used

Key pairs (--key-name) inject a public key for SSH at first boot; EC2 Instance Connect pushes a short-lived key per connection. Both need port 22 reachable (a public IP or a bastion). With Session Manager none of that is needed, which is why this chapter's instances have no key at all.

What you can do now

Why it helps

Credentials on instances are where cloud breaches happen: long-lived keys in files, or metadata credentials stolen through a server-side request forgery. Knowing that the SDK gets the role from IMDS, why IMDSv2 is required, what the hop limit does for containers, and how Session Manager replaces SSH is what lets you run servers with no keys and no open ports - and answer the security review.

Commands in this lesson

aws whoami hostname sudo exit curl

FAQ

How does code on an instance get AWS credentials?

From the instance profile: the role attached to the instance. The metadata service serves the role's temporary credentials, which AWS rotates every few hours, and the SDK's credential chain reads them automatically when there are no environment variables or config files. Nothing is stored on disk.

Why do I get 401 from 169.254.169.254?

The instance requires IMDSv2. First request a token with a PUT to /latest/api/token and a TTL header, then send it in the X-aws-ec2-metadata-token header on every GET. The SDKs and the AWS CLI do this by themselves; only hand-written curl calls need the two steps.

What does the hop limit change?

It is the IP TTL on the metadata responses. With 1, only processes on the instance itself get an answer; containers behind a bridge or pods on an EKS node add a hop and get nothing. With 2 they can reach IMDS and therefore the node's role. Set 1 on EKS nodes when pods have their own roles.

Why is my instance missing from Session Manager?

Check in this order: the instance role (no profile, or no AmazonSSMManagedInstanceCore permissions), the network (no route to the ssm, ssmmessages and ec2messages endpoints: no NAT gateway, no interface endpoints, or outbound rules blocking 443), and the agent (not installed on a custom image, stopped, or very old).

Do I still need SSH keys?

Usually not. Session Manager gives shells and port forwarding with IAM-controlled access and logged sessions, and needs no inbound port at all. Key pairs and EC2 Instance Connect still exist for tools that insist on SSH, but they need port 22 reachable through a public IP or a bastion.

In an interview Mid

How does an application on EC2 get AWS credentials without access keys?

Through the instance profile: the role's temporary credentials are served by the instance metadata service at 169.254.169.254, rotated by AWS, and the SDK reads them at the end of its credential chain. With IMDSv2 the client first PUTs /latest/api/token and sends the token header on every request, which blocks the SSRF attack that stole metadata credentials in 2019; HttpTokens=required enforces it, and the hop limit decides whether containers can reach it.

Also asked: Why should IMDSv1 be disabled? · How does Session Manager work without an open port? · What would you check if an instance does not appear in Session Manager?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.