OnCallReadyStart learning

LinuxPerformanceKubernetes · 5 min read

CPU throttling: p99 latency doubled, but the box is 70% idle

A CPU limit below the service's peak shows up as tail latency, not as high CPU. Prove it with cpu.max and nr_throttled, then raise or drop the quota.

Yesterday someone "right-sized" the pricing service to its average CPU use. Today its average latency is fine and its p99 has doubled. The box is mostly idle:

terminal
$ mpstat 1 1
Linux 7.0.0-31-generic (oncall-lab) 	09/22/26 	_aarch64_	(2 CPU)

20:00:04     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
20:00:05     all   21.15    0.00    6.13    0.00    0.00    0.02    0.00    0.00    0.00   72.70
$ pidstat -p $(pgrep pricing) 1 1
Linux 7.0.0-31-generic (oncall-lab) 	09/22/26 	_aarch64_	(2 CPU)

20:00:06      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
20:00:07     1001     17385   36.42   10.37    0.00   11.92   46.79     0  pricing

73% idle, no iowait (so not a disk problem), and pricing uses less than half a CPU. Every dashboard says "it does not need more". In fact it is being stopped every 100 milliseconds.

How a CPU limit actually works

On cgroup v2 a CPU limit is the file cpu.max: two numbers, QUOTA PERIOD, in microseconds. The kernel lets the group use QUOTA microseconds of CPU time in each PERIOD. When the group has used it up, every thread in the group is throttled: it is not scheduled again until the next period starts, even if every CPU on the box is idle.

terminal
$ systemctl show pricing -p CPUQuotaPerSecUSec
CPUQuotaPerSecUSec=500ms
$ cat /sys/fs/cgroup/system.slice/pricing.service/cpu.max
50000 100000

CPUQuota=50% in the unit became 50000 100000: 50 ms of CPU per 100 ms period, half a CPU. Now take a service with four busy worker threads on a node with four free CPUs. Together they burn the 50 ms in about 12.5 ms of wall-clock time, and then all four are frozen for the remaining ~87 ms. A request that arrives in that window waits for the next period before any code runs for it. Averaged over a second, CPU use still looks moderate (the quota caps it), so nothing looks hot. What you see is p99.

That is why average CPU is the wrong number to size a limit by. A multi-threaded service that averages half a CPU can need two CPUs for 20 ms at a time.

The proof: cpu.stat, read twice

The counters live next to cpu.max:

terminal
$ cat /sys/fs/cgroup/system.slice/pricing.service/cpu.stat; sleep 5; cat /sys/fs/cgroup/system.slice/pricing.service/cpu.stat
usage_usec 4125000
user_usec 3217500
system_usec 907500
nice_usec 0
core_sched.force_idle_usec 0
nr_periods 83
nr_throttled 69
throttled_usec 3306250
nr_bursts 0
burst_usec 0
usage_usec 6625000
user_usec 5167500
system_usec 1457500
nice_usec 0
core_sched.force_idle_usec 0
nr_periods 133
nr_throttled 110
throttled_usec 5270833
nr_bursts 0
burst_usec 0

They count from when the cgroup was created, so subtract:

CounterDelta over 5 sMeaning
nr_periods50periods in which the group had something to run
nr_throttled41periods that hit the ceiling: 82%
throttled_usec1,964,583almost 2 seconds of frozen threads in 5 seconds

nr_throttled / nr_periods going up steadily on a latency-sensitive service is the finding. throttled_usec tells you how much it costs. (The kernel documentation notes that these five bandwidth counters are not hierarchical: they count throttling caused by this cgroup's own limit.)

The same directory has cpu.pressure, PSI for just this cgroup. some is the share of time at least one of its tasks waited for a CPU, and here that is almost half:

terminal
$ cat /sys/fs/cgroup/system.slice/pricing.service/cpu.pressure
some avg10=47.92 avg60=43.13 avg300=23.96 total=5750000
full avg10=33.54 avg60=31.15 avg300=16.77 total=3833333

Why top, mpstat and pidstat miss it

mpstat shows the box idle, because it is. pidstat and top show the service at about its quota (here ~47%), which reads as "it uses what it has". None of them show demand that was refused. Only the cgroup's own counters (cpu.stat, cpu.pressure) do.

The fix

Raise the quota above the peak, or remove it:

terminal
$ sudo systemctl set-property pricing.service CPUQuota=150%
$ cat /sys/fs/cgroup/system.slice/pricing.service/cpu.max
150000 100000
$ cat /etc/systemd/system.control/pricing.service.d/50-CPUQuota.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Service]
CPUQuota=150%

Values above 100% allow more than one CPU. set-property changes the live cgroup at once (no restart) and writes the drop-in above, so the change survives a restart. CPUQuota= with an empty value removes the limit.

Then read the counters again:

terminal
$ grep -E 'nr_periods|nr_throttled|throttled_usec' /sys/fs/cgroup/system.slice/pricing.service/cpu.stat; sleep 5; grep -E 'nr_periods|nr_throttled|throttled_usec' /sys/fs/cgroup/system.slice/pricing.service/cpu.stat
nr_periods 188
nr_throttled 135
throttled_usec 5667766
nr_periods 238
nr_throttled 154
throttled_usec 5682966

nr_throttled still moves a little (19 of 50 periods hit the new ceiling for a moment at peaks), but throttled_usec grew by 15 ms in 5 seconds instead of almost 2 s. Watch the time, not just the count. A brief throttle at the end of a period costs almost nothing.

If the goal was "pricing must not starve the other services", a ceiling is the wrong tool anyway. CPUWeight= (the cgroup cpu.weight) shares the CPU in proportion only when there is contention, and lets the service use idle CPUs freely.

The same thing in Kubernetes

A container's resources.limits.cpu: 500m becomes exactly this: cpu.max of 50000 100000 on the container's cgroup. The Kubernetes docs: "cpu limits are enforced by CPU throttling". CPU requests become cpu.weight, a share under contention, not a ceiling. Many teams run latency-sensitive services with a CPU request and no CPU limit for this reason. If you keep limits, graph throttling and not only usage. cAdvisor (built into the kubelet) exports it:

output
rate(container_cpu_cfs_throttled_periods_total[5m])
  / rate(container_cpu_cfs_periods_total[5m])

container_cpu_cfs_throttled_seconds_total is the throttled_usec equivalent. A pod at 40% of its limit on average with a high throttled ratio is the pricing service above. Measure the latency side with a histogram, not an average: p99 from a Prometheus histogram is where this shows first. Memory limits fail differently: CPU time can be made to wait, memory cannot be handed back that way, so a container that reaches its memory limit is OOM-killed instead (exit code 137).

In an interview

"A CPU limit is a CFS quota per 100 ms period. A multi-threaded service can use its whole quota early in a period and then be frozen for the rest, so p99 latency goes up while average CPU looks fine. I check nr_throttled and throttled_usec in the cgroup's cpu.stat, read twice, and size limits to the peak, or use weights instead."

Practise it

In the Linux Performance & eBPF chapter, the lesson CPU limits in cgroups: quota and throttling explains cpu.max and cpu.stat. The lab Fast on average, slow at p99 throttles the pricing service exactly like this, and the incident p99 doubled after the right-sizing asks you to prove what yesterday's change did and fix it.

OnCallReady is free, with no ads and no tracking. RSS · All posts