OnCallReadyStart learning

AWSNetworkingSRE · 4 min read

ALB 502 vs 503 vs 504 on AWS: what each one says about your targets

From an AWS ALB: 502 = the target refused or broke the connection, 503 = no target, 504 = no answer in time. Real outputs for each, and the fixes.

Three 5xx codes come from an AWS Application Load Balancer itself, before your application has said anything. They look alike on a dashboard and mean three different things. The first clue is the same for all of them: who signed the answer.

terminal
$ curl -si http://$DNS/ | head -3
HTTP/1.1 502 Bad Gateway
Server: awselb/2.0
Date: Tue, 22 Sep 2026 20:02:30 GMT

Server: awselb/2.0 means the load balancer generated this response. A 502 that says Server: nginx came from a proxy behind it, and that is a different hunt in nginx's error.log. In CloudWatch the split is HTTPCode_ELB_5XX_Count (the ALB's own) versus HTTPCode_Target_5XX_Count (your app's); in the access log, an ALB-generated error has - as the target status.

codethe ALB is sayingusual causes
502 Bad GatewayI reached a target and it refused, reset or garbled the connectionwrong port, app crashed, keep-alive timeout on the target shorter than the ALB's idle timeout
503 Service Temporarily UnavailableI have nobody to send this tono registered targets in the group, a rule that forwards nowhere
504 Gateway Time-outI tried, and nobody answered in timea security group or NACL dropping packets (no connection in 10 s), an app slower than the idle timeout (60 s)

502: a new image on a new port

The shop returned 502 for every request after a rollout replaced the servers. Target health first, with the reason column:

terminal
$ aws elbv2 describe-target-health --target-group-arn $TG --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State,TargetHealth.Reason]' --output text
i-0c4cc89644441bb98	8080	unhealthy	Target.FailedHealthChecks
i-01529380cb908763d	8080	unhealthy	Target.FailedHealthChecks

Every target is unhealthy, yet the ALB still sends them traffic: with all targets unhealthy it fails open and tries them anyway, betting that a bad health check is likelier than a dead fleet. So users get a 502 (connection refused on 8080) rather than a 503. The new instances' user data says why nothing listens there:

terminal
$ aws ec2 describe-instance-attribute --instance-id $W3 --attribute userData --query UserData.Value --output text | base64 -d
#!/bin/bash
set -euo pipefail
# platform standard 2026-10: app ports 8081+
echo "SHOP_PORT=8081" > /etc/default/shop-web
systemctl restart shop-web

The app moved to 8081, the target group still says 8080. No rebuild needed: register each target with its own port, deregister the old registrations, and open 8081 from the ALB's security group (and close 8080):

terminal
$ aws elbv2 register-targets --target-group-arn $TG --targets Id=$W3,Port=8081 Id=$W4,Port=8081
$ aws elbv2 deregister-targets --target-group-arn $TG --targets Id=$W3,Port=8080 Id=$W4,Port=8080
$ aws elbv2 describe-target-health --target-group-arn $TG --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State]' --output text
i-0c4cc89644441bb98	8080	draining
i-01529380cb908763d	8080	draining
i-0c4cc89644441bb98	8081	healthy
i-01529380cb908763d	8081	healthy
$ curl -s -o /dev/null -w '%{http_code}\n' http://$DNS/
200

The other classic 502 is intermittent: a few per minute, no unhealthy targets. The ALB reuses idle connections to targets for up to 60 seconds; an app server that closes them after 5 (Node.js's default keepAliveTimeout, many Gunicorn setups) races it, and the request that loses gets a 502. Set the target's keep-alive timeout above the ALB's idle timeout.

504: the packets never arrive

Same load balancer, healthy targets. Someone tightens the application's security group and removes the rule that lets the ALB in on 8080:

terminal
$ aws ec2 revoke-security-group-ingress --group-id $APP --protocol tcp --port 8080 --source-group $SG --query Return
true
$ aws elbv2 describe-target-health --target-group-arn $TG --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason]' --output text
i-02c9cb943f3ba6a34	unhealthy	Target.Timeout
i-09b9311391163d63f	unhealthy	Target.Timeout
$ time curl -si http://$DNS/ | head -3

real	0m10.046s
user	0m0.03s
sys	0m0.005s
HTTP/1.1 504 Gateway Time-out
Server: awselb/2.0
Date: Tue, 22 Sep 2026 20:02:39 GMT

Note the reason, Target.Timeout instead of Target.FailedHealthChecks, and the 10 seconds: the ALB gives a target 10 s to accept a connection. A refused connection is fast and gives 502; a dropped one is silent and gives 504. Dropped means a security group, a network ACL (remember that NACLs are stateless and need the ephemeral return ports), or a target that is down hard. The other 504 is slow: the connection works but the app takes longer than the idle timeout (60 s by default) to answer. For that one, find the slow requests (target_processing_time in the access log) before you raise the timeout.

503: nobody to send to

Put the rule back, then take every target out of the group, as a broken deploy script or an Auto Scaling group scaled to zero would:

terminal
$ aws elbv2 deregister-targets --target-group-arn $TG --targets $(for i in $IDS; do printf 'Id=%s ' $i; done)
$ aws elbv2 describe-target-health --target-group-arn $TG --output text

$ curl -s -o /dev/null -w '%{http_code}\n' http://$DNS/
503
$ curl -s http://$DNS/
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
</body>
</html>

An empty target-health list is the whole diagnosis. Look at what deregistered them: the deploy tool, the ASG's activity history (aws autoscaling describe-scaling-activities), or a Kubernetes Service behind the AWS Load Balancer Controller whose selector matches no ready pods. Other platforms have their own spelling of the same 503: OpenShift's Application is not available page and Envoy's UH flag.

The order to check

  1. Who answered? Server: awselb/2.0 or your app's header; ELB vs target metrics.
  2. Target health with the reason - FailedHealthChecks (refused, wrong code), Timeout (dropped), or an empty list.
  3. The path from the ALB's security group to the target's port, on the port each target is registered with.
  4. Timeouts: the target's keep-alive timeout must be longer than the ALB's idle timeout, so the ALB always closes an idle connection first; slow answers beyond the idle timeout are 504s.

Practise it

Incident: 502 Bad Gateway after the new image is the port mismatch above, with a security group to fix as well. Put an Application Load Balancer in front of two servers builds the ALB, target group, health checks and a fixed-response rule from the CLI, and Drill: why is the target unhealthy? deals a different broken target every round.

OnCallReady is free, with no ads and no tracking. RSS · All posts