OnCallReady

Lesson 21.8 · Spring Boot Runtime, Resilience & Python Ops · 15 min read

Graceful shutdown, TimeoutStopSec and terminationGracePeriodSeconds

In plain words

Imagine a restaurant at closing time. A good one locks the front door so no new guests come in, lets the people already eating finish their meal, and only then turns off the ovens and sends the staff home. But the landlord said the building closes at 11 sharp. If the restaurant takes too long, the landlord switches off all the lights with guests still at their tables.

That is graceful shutdown. SIGTERM is "we're closing"; server.shutdown=graceful locks the door and waits for in-flight requests; timeout-per-shutdown-phase is how long it waits. The landlord's deadline is systemd's TimeoutStopSec or Kubernetes' terminationGracePeriodSeconds, after which SIGKILL arrives. The inner timeout must be shorter than the outer one.

What happens on SIGTERM

The problem. Every deploy and every node drain stops pods. If the app dies in the middle of a request, users see errors "only during deploys" - which on a busy day means all the time. Graceful shutdown finishes the work in hand first, as long as the outer timeout lets it.

What you need to know already: SIGTERM, SIGKILL and exit 143 (3.6), systemd TimeoutStopSec (2.24), what happens when a pod is deleted (3.18), readiness (17.20), EndpointSlices and kube-proxy (16.1, 16.3).

The JVM receives SIGTERM, runs its shutdown hooks (code the app registered to run on exit), and Spring closes the application context (the container of all the app's beans, 21.1). With graceful shutdown the web server first stops accepting new requests, waits for in-flight ones to finish, and only then do the other beans (the DataSource, schedulers) shut down.

server.shutdown=graceful                         the DEFAULT since Spring Boot 3.4 (before: immediate)
spring.lifecycle.timeout-per-shutdown-phase=30s  how long to wait for in-flight requests (default 30s)

In the log:

o.s.b.w.e.tomcat.GracefulShutdown        : Commencing graceful shutdown. Waiting for active requests to complete
o.s.b.w.e.tomcat.GracefulShutdown        : Graceful shutdown complete
com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.

And when in-flight work outlives the phase timeout:

o.s.c.support.DefaultLifecycleProcessor  : Shutdown phase 2147482623 ends with 1 bean still running after timeout of 30000ms: [webServerGracefulShutdown]
o.s.b.w.e.tomcat.GracefulShutdown        : Graceful shutdown aborted with one or more requests still active

Readiness flips to REFUSING_TRAFFIC (OUT_OF_SERVICE, 503) as soon as shutdown starts, so a load balancer that probes readiness stops routing to it.

The JVM then exits with 143 (128 + 15, SIGTERM). That is why our units say SuccessExitStatus=143 - otherwise systemd would call a normal stop a failure.

The outer timeout always wins

Whoever sent SIGTERM has its own deadline, after which it sends SIGKILL:

systemd      TimeoutStopSec=           default 90s (DefaultTimeoutStopSec in system.conf)
Kubernetes   terminationGracePeriodSeconds   default 30s
Docker       docker stop -t            default 10s

If the application's graceful phase is longer than the outer timeout, the process is killed mid-drain: in-flight requests get a closed connection (curl: (52) Empty reply from server, a 502 at the proxy), and in systemd:

orders.service: State 'stop-sigterm' timed out. Killing.
orders.service: Killing process 1210 (java) with signal SIGKILL.
orders.service: Main process exited, code=killed, status=9/KILL
orders.service: Failed with result 'timeout'.

The rule: timeout-per-shutdown-phase + everything else in shutdown < the outer timeout. In Kubernetes with a preStop sleep (a hook the kubelet runs before sending SIGTERM):

preStop sleep 5  +  phase timeout 20s  +  a few seconds for the rest   <   terminationGracePeriodSeconds 30

The preStop trap

When a pod is deleted, two things happen concurrently, not in order:

kubelet                                  control plane
  runs preStop, then sends SIGTERM          removes the pod from Endpoints / EndpointSlices
                                            kube-proxy on every node updates iptables/IPVS
                                            ingress controllers update their upstreams

The endpoint removal takes a moment to reach every node. If the app stops accepting connections the instant SIGTERM arrives, traffic that is still being routed to it fails - 502s on every deploy (502 Bad Gateway = the proxy in front could not get an answer, 9.23). The fix is to delay the SIGTERM:

lifecycle:
  preStop:
    exec: { command: ["sleep", "5"] }       # or the built-in: preStop: { sleep: { seconds: 5 } } (1.30+)

The pod keeps serving for 5 seconds while the endpoint removal propagates, then graceful shutdown drains whatever is still in flight. Two fixes for "502s on every deploy": the preStop delay, and graceful shutdown with a phase timeout that fits the grace period. Most teams have one and not the other.

On this box

systemd plays the kubelet's role: systemctl stop sends SIGTERM and waits TimeoutStopSec. The same arithmetic applies, and you can watch it:

$ curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/api/reports/export &    # a 45-second request
$ sudo systemctl restart orders
$ journalctl -u orders -n 8 --no-pager

Next mission: a phase timeout of 120 s against a TimeoutStopSec of 30 s.

What you can now do

Why it helps

"We get 502s on every deploy" is one of the most common tickets a platform team receives, and this lesson is the answer. It is almost always one of two things: the pod stops accepting connections before the endpoint removal has reached kube-proxy and the ingress, or it is SIGKILLed mid-drain because the app's shutdown takes longer than the grace period.

Knowing the concurrent sequence, preStop and SIGTERM on one side and endpoint removal on the other, lets you fix it in the shared base Deployment for every team at once: a preStop sleep, graceful shutdown, and a phase timeout that fits the grace period. It also explains SuccessExitStatus=143 on your systemd units, and the timeout lines you'll see in journalctl when a stop is too slow.

Commands in this lesson

curl systemctl journalctl

FAQ

Why does the JVM exit with 143?

143 is 128 plus 15, the number of SIGTERM. A process terminated by a signal reports 128 plus the signal number, and a JVM that exits after its shutdown hooks run on SIGTERM reports 143. That is a normal stop, but systemd treats a non-zero status as failure unless told otherwise, which is why the Java units on this box say SuccessExitStatus=143. In Kubernetes you see it as the container's exit code after a normal termination.

Why do I need a preStop sleep if the app shuts down gracefully?

Because deleting a pod triggers two things at the same time: SIGTERM to the container and removal from EndpointSlices, which then has to propagate to kube-proxy on every node and to ingress controllers. For a few seconds, traffic can still be routed to the pod. Graceful shutdown stops accepting new connections immediately, so those requests fail. A sleep 5 in preStop delays SIGTERM, so the pod keeps serving while routing catches up.

What happens if the shutdown phase timeout is longer than the grace period?

The outer timeout wins. systemd waits TimeoutStopSec, 90 s by default, and Kubernetes waits terminationGracePeriodSeconds, 30 s by default, then sends SIGKILL. In-flight requests get their connections closed, clients see Empty reply from server or a 502 at the proxy, and systemd logs State 'stop-sigterm' timed out. Killing. and Failed with result 'timeout'. Keep preStop plus the phase timeout plus other shutdown work below the outer timeout.

Is graceful shutdown on by default?

Since Spring Boot 3.4, yes: server.shutdown=graceful is the default, with spring.lifecycle.timeout-per-shutdown-phase of 30 s. Before 3.4 the default was immediate, so older services need it set explicitly. Check the version before assuming. In the logs, Commencing graceful shutdown. Waiting for active requests to complete followed by Graceful shutdown complete shows it working.

Does readiness change during shutdown?

Yes. As soon as graceful shutdown begins, Spring Boot flips readiness to REFUSING_TRAFFIC, and /actuator/health/readiness returns 503 OUT_OF_SERVICE. A load balancer or readiness probe that notices stops routing to the instance. In Kubernetes the endpoint removal triggered by pod deletion usually gets there first, but for systemd services behind a load balancer that probes readiness, this is the mechanism that drains traffic.

In an interview Mid

What happens when a pod is terminated, and how do you make a Spring Boot app shut down without failing requests?

When a pod is deleted, two things happen at the same time: the kubelet runs the preStop hook and then sends SIGTERM, while the control plane removes the pod from the EndpointSlices and kube-proxy and ingress controllers update - which takes a moment to reach every node. After terminationGracePeriodSeconds (default 30 s) comes SIGKILL.

What the app needs:

On a VM systemd plays the kubelet: TimeoutStopSec then SIGKILL (Failed with result 'timeout'), and a clean SIGTERM exit is 143 - hence SuccessExitStatus=143.

Also asked: Why do some services return 502s during every deployment, and what fixes it? · What is the difference between SIGTERM and SIGKILL in a container shutdown? · Why does a systemd unit for a Java service set SuccessExitStatus=143?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.