Three 5xx codes come from an AWS Application Load Balancer itself, before your application has said anything. They look alike on a dashboard and mean three different things. The first clue is the same for all of them: who signed the answer.
$ curl -si http://$DNS/ | head -3
HTTP/1.1 502 Bad Gateway
Server: awselb/2.0
Date: Tue, 22 Sep 2026 20:02:30 GMTServer: awselb/2.0 means the load balancer generated this response. A 502 that says Server: nginx came from a proxy behind it, and that is a different hunt in nginx's error.log. In CloudWatch the split is HTTPCode_ELB_5XX_Count (the ALB's own) versus HTTPCode_Target_5XX_Count (your app's); in the access log, an ALB-generated error has - as the target status.
| code | the ALB is saying | usual causes |
|---|---|---|
| 502 Bad Gateway | I reached a target and it refused, reset or garbled the connection | wrong port, app crashed, keep-alive timeout on the target shorter than the ALB's idle timeout |
| 503 Service Temporarily Unavailable | I have nobody to send this to | no registered targets in the group, a rule that forwards nowhere |
| 504 Gateway Time-out | I tried, and nobody answered in time | a security group or NACL dropping packets (no connection in 10 s), an app slower than the idle timeout (60 s) |
502: a new image on a new port
The shop returned 502 for every request after a rollout replaced the servers. Target health first, with the reason column:
$ aws elbv2 describe-target-health --target-group-arn $TG --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State,TargetHealth.Reason]' --output text
i-0c4cc89644441bb98 8080 unhealthy Target.FailedHealthChecks
i-01529380cb908763d 8080 unhealthy Target.FailedHealthChecksEvery target is unhealthy, yet the ALB still sends them traffic: with all targets unhealthy it fails open and tries them anyway, betting that a bad health check is likelier than a dead fleet. So users get a 502 (connection refused on 8080) rather than a 503. The new instances' user data says why nothing listens there:
$ aws ec2 describe-instance-attribute --instance-id $W3 --attribute userData --query UserData.Value --output text | base64 -d
#!/bin/bash
set -euo pipefail
# platform standard 2026-10: app ports 8081+
echo "SHOP_PORT=8081" > /etc/default/shop-web
systemctl restart shop-webThe app moved to 8081, the target group still says 8080. No rebuild needed: register each target with its own port, deregister the old registrations, and open 8081 from the ALB's security group (and close 8080):
$ aws elbv2 register-targets --target-group-arn $TG --targets Id=$W3,Port=8081 Id=$W4,Port=8081
$ aws elbv2 deregister-targets --target-group-arn $TG --targets Id=$W3,Port=8080 Id=$W4,Port=8080
$ aws elbv2 describe-target-health --target-group-arn $TG --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State]' --output text
i-0c4cc89644441bb98 8080 draining
i-01529380cb908763d 8080 draining
i-0c4cc89644441bb98 8081 healthy
i-01529380cb908763d 8081 healthy
$ curl -s -o /dev/null -w '%{http_code}\n' http://$DNS/
200The other classic 502 is intermittent: a few per minute, no unhealthy targets. The ALB reuses idle connections to targets for up to 60 seconds; an app server that closes them after 5 (Node.js's default keepAliveTimeout, many Gunicorn setups) races it, and the request that loses gets a 502. Set the target's keep-alive timeout above the ALB's idle timeout.
504: the packets never arrive
Same load balancer, healthy targets. Someone tightens the application's security group and removes the rule that lets the ALB in on 8080:
$ aws ec2 revoke-security-group-ingress --group-id $APP --protocol tcp --port 8080 --source-group $SG --query Return
true
$ aws elbv2 describe-target-health --target-group-arn $TG --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason]' --output text
i-02c9cb943f3ba6a34 unhealthy Target.Timeout
i-09b9311391163d63f unhealthy Target.Timeout
$ time curl -si http://$DNS/ | head -3
real 0m10.046s
user 0m0.03s
sys 0m0.005s
HTTP/1.1 504 Gateway Time-out
Server: awselb/2.0
Date: Tue, 22 Sep 2026 20:02:39 GMTNote the reason, Target.Timeout instead of Target.FailedHealthChecks, and the 10 seconds: the ALB gives a target 10 s to accept a connection. A refused connection is fast and gives 502; a dropped one is silent and gives 504. Dropped means a security group, a network ACL (remember that NACLs are stateless and need the ephemeral return ports), or a target that is down hard. The other 504 is slow: the connection works but the app takes longer than the idle timeout (60 s by default) to answer. For that one, find the slow requests (target_processing_time in the access log) before you raise the timeout.
503: nobody to send to
Put the rule back, then take every target out of the group, as a broken deploy script or an Auto Scaling group scaled to zero would:
$ aws elbv2 deregister-targets --target-group-arn $TG --targets $(for i in $IDS; do printf 'Id=%s ' $i; done)
$ aws elbv2 describe-target-health --target-group-arn $TG --output text
$ curl -s -o /dev/null -w '%{http_code}\n' http://$DNS/
503
$ curl -s http://$DNS/
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
</body>
</html>An empty target-health list is the whole diagnosis. Look at what deregistered them: the deploy tool, the ASG's activity history (aws autoscaling describe-scaling-activities), or a Kubernetes Service behind the AWS Load Balancer Controller whose selector matches no ready pods. Other platforms have their own spelling of the same 503: OpenShift's Application is not available page and Envoy's UH flag.
The order to check
- Who answered?
Server: awselb/2.0or your app's header; ELB vs target metrics. - Target health with the reason -
FailedHealthChecks(refused, wrong code),Timeout(dropped), or an empty list. - The path from the ALB's security group to the target's port, on the port each target is registered with.
- Timeouts: the target's keep-alive timeout must be longer than the ALB's idle timeout, so the ALB always closes an idle connection first; slow answers beyond the idle timeout are 504s.
Practise it
Incident: 502 Bad Gateway after the new image is the port mismatch above, with a security group to fix as well. Put an Application Load Balancer in front of two servers builds the ALB, target group, health checks and a fixed-response rule from the CLI, and Drill: why is the target unhealthy? deals a different broken target every round.