OnCallReadyStart learning

Linux Lesson 8.7 · Linux Performance & eBPF · 4 min read

CPU limits in cgroups: quota and throttling

In plain words

A cgroup CPU limit is like a phone plan with 50 minutes every hour. If you talk fast and use all 50 minutes in the first quarter of the hour, the phone goes silent until the next hour starts, even though the network is empty. A service with a CPU quota works the same way: it spends its allowance early in each 100 ms window, then it is frozen until the next window.

The limit that hides in plain sight

A cgroup can cap CPU two ways:

Weight (cpu.weight, CPUWeight=)Quota (cpu.max, CPUQuota=)
What it doesshares contended CPU in proportionhard ceiling per period, even if CPUs are idle
Idle boxthe group may use everythingthe group still stops at its quota
Symptom when too lowwaits only under contentionthrottling: stalls of up to the rest of each 100 ms period

How a quota works. cpu.max holds "quota period" in microseconds: 50000 100000 means 50 ms of CPU time per 100 ms period, half a CPU. A process with four threads spends those 50 ms in 12.5 ms of wall time, then all four threads are throttled - frozen - until the next period. A request that arrives at the wrong moment waits up to ~87 ms doing nothing. Average CPU looks moderate; p99 latency is terrible.

$ systemctl show pricing -p CPUQuotaPerSecUSec
CPUQuotaPerSecUSec=500ms
$ cat /sys/fs/cgroup/system.slice/pricing.service/cpu.max
50000 100000
$ cat /sys/fs/cgroup/system.slice/pricing.service/cpu.stat
usage_usec 2550000
user_usec 1989000
system_usec 561000
nice_usec 0
core_sched.force_idle_usec 0
nr_periods 51
nr_throttled 42
throttled_usec 2012500
nr_bursts 0
burst_usec 0

Read cpu.stat twice and subtract: nr_throttled / nr_periods is the share of periods that hit the ceiling (here 42 of 51), throttled_usec the total time frozen. Any steady growth of nr_throttled on a latency-sensitive service is a finding. cpu.pressure in the same directory is PSI for just this cgroup.

Why the usual tools miss it. mpstat shows the box mostly idle. pidstat shows the service near its quota (50%), which looks like "it does not need more". Only cpu.stat (or PSI) shows that it wanted more and was stopped. (runqlat-bpfcc sees the stalls as run queue latency - a throttled thread stays runnable until the next period - but not why: with idle CPUs, suspect a quota.)

Fixing it. Raise the quota or drop it (sudo systemctl set-property pricing.service CPUQuota=200%, or CPUQuota= with no value), or use CPUWeight= when the goal is fair sharing, not a ceiling. set-property writes a drop-in under /etc/systemd/system.control/, so it survives a restart.

Later (Ch 19): in Kubernetes a container's resources.limits.cpu: 500m becomes exactly this: cpu.max of 50000 100000 on the container's cgroup. CPU requests become weights. Many teams run latency-sensitive services with requests and no CPU limit for this reason; if you keep limits, watch the throttling metric (cAdvisor's container_cpu_cfs_throttled_periods_total, the Observability chapters show it on a dashboard).

In an interview: "A CPU limit is a CFS quota per 100 ms period. When a multi-threaded service uses its quota early in the period it is throttled for the rest, so p99 latency goes up while average CPU looks fine. I check nr_throttled in cpu.stat."

Why it helps

CPU throttling is one of the most common causes of bad tail latency on services with a CPU limit, and the usual tools miss it: top shows modest CPU, the box looks idle, yet requests stall. Knowing where the throttling counters live and how weights differ from quotas lets you prove the cause in two commands and fix it.

Later (Ch 13, 19): a container CPU limit, and every Kubernetes CPU limit, is exactly this quota.

On the blog

Commands in this lesson

systemctl cat

FAQ

Where is a unit's CPU limit?

In its cgroup: /sys/fs/cgroup/system.slice/NAME.service/cpu.max holds the quota and the period in microseconds, for example 50000 100000 for half a CPU. systemd writes it from CPUQuota= in the unit or a drop-in; max means no limit.

How do I prove a service is throttled?

Read its cpu.stat twice, a few seconds apart. If nr_throttled and throttled_usec grow, the cgroup hit its quota and was stopped until the next period. nr_periods shows how many periods passed, so nr_throttled divided by nr_periods is the share of periods that were throttled.

Why does throttling hurt multi-threaded services most?

The quota is CPU time across all threads. Eight threads running at once burn 50 ms of quota in about 6 ms of wall time, then every thread waits for the rest of the 100 ms period. Each request that arrives then waits up to tens of milliseconds, which shows up as p99 latency.

What is the difference between CPUWeight and CPUQuota?

CPUWeight= sets cpu.weight, a share that only matters when CPUs are contended; an idle box gives a low-weight service all the CPU it wants. CPUQuota= sets cpu.max, a hard ceiling per period even when the box is idle. Weights never cause throttling; quotas do.

Should a latency-sensitive service have a CPU quota at all?

Often not. A quota is a ceiling even on an idle box, so a service that bursts is frozen in the middle of its requests. Many teams give such services a weight (CPUWeight=) for fair sharing under contention and no quota at all, because the quota is exactly what throttles them. If a quota stays, watch nr_throttled in its cpu.stat.

In an interview Mid

A service uses 50% CPU, the box is idle, but its p99 latency is bad. What do you check?

CFS throttling from a CPU quota: CPUQuota= on a systemd unit sets cpu.max, a quota per 100 ms period. A multi-threaded service burns it early in the period and is frozen for the rest, so requests stall while average CPU looks fine. I read nr_throttled and throttled_usec in the cgroup's cpu.stat twice; if they grow, I raise the limit or switch to weights.

Also asked: What is the difference between a CPU weight and a CPU quota? · Where do you find a service's cgroup CPU counters? · Should a latency-sensitive service have a CPU quota at all?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.