"No space left on device" but df shows free space: the 3 causes, in the order to check

A write fails with No space left on device - the kernel's ENOSPC - and df -h says the disk is half empty. All three causes below produce exactly the same error text, so check them in order.

1. Inodes exhausted - df -i

Every file costs one inode, however small, and an ext4 filesystem is created with a fixed number of them. Millions of tiny files (web sessions, a cache, a mail queue) run out of inodes long before they run out of blocks:

$ df -h /srv/cache        $ df -i /srv/cache
Size Used Avail Use%      Inodes IUsed IFree IUse%
2.0G 802M  1.2G  42%       12288 12288     0  100%

42% full, and not one more file fits. To find where the inodes went:

sudo du --inodes -x /srv/cache --max-depth=2 | sort -n | tail

Fix: delete files. With millions of them rm dir/* fails with "Argument list too long", so use find:

sudo find /srv/cache/sessions -type f -mtime +7 -delete

On a Docker host, image layers and container logs are the usual suspects: Docker's "no space left on device".

You can't add inodes to an existing ext4 filesystem. Re-create it with more (mkfs.ext4 -N), or use XFS, which allocates inodes as it needs them.

2. A deleted file still held open - sudo lsof +L1

Someone ran rm on a big log that a process still has open. du can't see it any more, df still counts it. Find the holder with lsof +L1, then restart it or sudo truncate -s 0 /proc/<pid>/fd/<n>. The full story: I deleted a 40 GB log and df didn't change.

3. Reserved blocks - sudo tune2fs -l /dev/vdb1 | grep -i reserved

ext4 keeps 5% of the filesystem as reserved blocks that only root may use. df subtracts them from Avail, so a normal user sees Avail 0 and Use% 100% while root can still write. That's why "it works with sudo" is a clue, not a fix. On a large data volume, 5% can be many gigabytes. Lowering it is safe for data disks (not for /):

sudo tune2fs -m 1 /dev/vdb1

Why this order

df -i and lsof +L1 take seconds and explain most real cases. Reserved blocks only matter when the disk really is nearly full for everyone but root. And adding disk fixes none of the three: the first two would still be there on the bigger disk. If processes hang instead of failing with ENOSPC, look at D-state and the load average instead.