I deleted a 40 GB log and df didn't change. Where did the space go?

It is 3 a.m., /data is at 98%, and you find the culprit: a 40 GB debug.log. You delete it and check again:

$ sudo rm /data/app/debug.log
$ df -h /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/vdb1    50G   42G  6.2G  87% /data

Still 42 GB used. du agrees with you, though:

$ sudo du -sh /data/app
8.0K	/data/app

rm deletes a name, not a file

On Linux a file's data lives in an inode. A directory entry - the name you see in ls - is just a pointer to that inode. rm removes the pointer. The kernel frees the inode and its blocks only when two things are both true:

• no names point at it any more (its link count is 0), and
• no process still has it open.

The service that was writing the log still has it open. So the name is gone and du, which walks directories, can't see it. But the blocks are still allocated, and df, which asks the filesystem, still counts them. du and df disagreeing by the size of one file is the tell.

Find the holder: lsof +L1

+L1 means "open files with a link count below 1": deleted, but still held open.

$ sudo lsof +L1
COMMAND    PID USER    FD TYPE DEVICE    SIZE/OFF NLINK NODE NAME
logwriter 1302 appuser 3w REG  253,17 42949672960     0 1157 /data/app/debug.log (deleted)

Process 1302 holds it on file descriptor 3.

Get the space back

In order of preference:

1. Restart the service. It closes the old file and opens a fresh one, and the kernel frees the inode. On Kubernetes that restart is a rollout: make it drop no requests.
2. Empty it through the descriptor, if you can't restart right now:

$ sudo truncate -s 0 /proc/1302/fd/3
$ df -h /data
Filesystem  Size  Used Avail Use% Mounted on
/dev/vdb1    50G  1.4G   47G   3% /data

/proc/<pid>/fd/3 is the open file itself, so this empties the deleted inode. The process keeps writing, into an empty file. Don't kill -9 it just to free space: a clean restart does the same job.

Don't let it happen again

• To empty a live log, truncate it instead of deleting it: sudo truncate -s 0 file or : > file.
• Rotate logs in a way the program knows about: logrotate's copytruncate (copy, then truncate in place), or a signal that makes the program reopen its log. Or log to the journal. Containers fill disks from another side: see Docker's "no space left on device".

This is one of the three reasons a disk can say one thing while writes do another. The other two are inode exhaustion and reserved blocks; see the lesson below.