OnCallReady

Lesson 4.18 · Filesystem, Permissions, Disk · 16 min read

Disks, mounts and what df is really telling you

In plain words

Imagine a house made of several separate rooms bolted together, each built from its own box of bricks. From the hallway it looks like one house, but each room has its own size and its own number of shelves. If the garage fills up, the kitchen can still be empty. And if a room was bolted on over a doorway where you had already stacked boxes, those boxes are still there behind the wall, invisible.

Filesystems are the rooms and mount points are where they are bolted on: /, /data, /srv/cache. lsblk shows the bricks (disks, partitions, LVM), findmnt shows which room is where, and df shows each room's size, used space, what ordinary users can still use, and with df -i how many shelves (inodes) are left.

Why "which disk is this path on?" comes first

/ can be fine while /data is full, and a hard link can work in one directory and fail in the next. Every disk question starts with which filesystem a path actually lives on - so first, how one directory tree is built out of several disks.

What you need to know already: 1.20 (lsblk, vda, LVM in one sentence), 4.1 (/proc/mounts), 4.13 (inodes).

The words, once

One tree, many filesystems

There is one directory tree, but it is stitched together from several filesystems, each mounted on a directory. Which one a path lives on decides how much space it has, how many inodes, and whether a hard link across it is possible.

$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda                       253:0    0   40G  0 disk
├─vda1                    253:1    0    1G  0 part /boot/efi
├─vda2                    253:2    0    2G  0 part /boot
└─vda3                    253:3    0 36.9G  0 part
  └─ubuntu--vg-ubuntu--lv 252:0    0   18G  0 lvm  /
vdb                       253:16   0   50G  0 disk
└─vdb1                    253:17   0   50G  0 part /data
vdc                       253:32   0    2G  0 disk
└─vdc1                    253:33   0    2G  0 part /srv/cache

lsblk ("list block devices") columns: MAJ:MIN driver and instance numbers, RM removable (1 = yes), SIZE, RO read-only, TYPE disk / part(ition) / lvm, and where it is mounted.

Read it as a tree: disk vda has three partitions; the third is an LVM PV holding the 18G LV that is /. Two more disks are mounted at /data and /srv/cache. The root LV is 18G on a 36.9G partition - Ubuntu's installer leaves the rest of the VG free so you can grow it later (lvextend, 1.20).

findmnt shows what is mounted where, with the options:

$ findmnt /data
TARGET            SOURCE                              FSTYPE   OPTIONS
/data             /dev/vdb1                           ext4     rw,relatime

TARGET the mount point, SOURCE the device, FSTYPE the filesystem type, OPTIONS how it is mounted (rw read-write; relatime a timestamp optimisation). findmnt -T <path> answers "which filesystem is this path on" - the first question when a write fails.

The permanent list is /etc/fstab ("filesystem table"): one line per mount that comes back at boot. A mount made by hand with mount disappears on reboot.

df, column by column

df -h lists every mounted filesystem (-h human units):

$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              593M  1.4M  592M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   19G  7.5G  9.7G  44% /
tmpfs                              2.9G     0  2.9G   0% /dev/shm
/dev/vda2                          2.0G  186M  1.7G  11% /boot
/dev/vdb1                           50G   42G  6.2G  87% /data
/dev/vdc1                          2.0G  754M  1.2G  39% /srv/cache

tmpfs lines are RAM, not disk. /dev/shm and /run live in memory and what you write there counts as memory used.

The other resource: inodes

$ df -i /
Filesystem                           Inodes     IUsed     IFree IUse% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   1179648    118789   1060859   11% /

df -i shows inodes instead of bytes: total, used, free, percent used. One inode per file, directory or symlink, fixed when the filesystem was created. ext4's default is roughly one inode per 16 KiB of space; the small 2G /srv/cache volume has only 12288. When IFree hits 0 you get the same No space left on device as a full disk.

tune2fs: the filesystem's own numbers

tune2fs reads and changes settings of an ext4 filesystem; -l lists them:

$ sudo tune2fs -l /dev/vdc1
tune2fs 1.47.2 (1-Jan-2025)
Filesystem volume name:   <none>
Last mounted on:          /srv/cache
...
Inode count:              12288
Block count:              524288
Reserved block count:     26214
Free blocks:              331413
Free inodes:              12274
Block size:               4096
Reserved blocks uid:      0 (user root)

The numbers df turns into columns: Block count x Block size is the size, Reserved block count is the root-only reserve (26214 of 524288 = 5%), Inode count is the hard inode limit. tune2fs -m 1 changes the reserve on a live filesystem; nothing changes the inode count short of re-creating it.

du walks names; df asks the filesystem

$ du -sh /srv/cache
12K	/srv/cache

du (next lesson) adds up what it can see, by name. df asks the filesystem what is allocated. Ordinary differences are small (bookkeeping). A big difference means one of: files hidden under a mount point, files you could not read, or - the famous one - files deleted while still open. Keep this picture; the next two lessons use it.

Mounting over a non-empty directory

One more way space hides: if something wrote to /data while the disk was not mounted (a boot where the mount failed), those files are on the root filesystem, underneath the mount point. Once /dev/vdb1 is mounted over /data, they are invisible to du and still eating /. The check is a bind mount - making a directory appear at a second place - of / somewhere, then looking: sudo mount --bind / /mnt && sudo du -sh /mnt/data.

What you can now do

Why it helps

The first question in every disk incident is "which filesystem is full", because / and /data fill independently. findmnt -T /path answers where a failing write is going, and df -h plus df -i show whether it is blocks or inodes. Understanding that Avail excludes the root reserve and that Use% rounds up explains why a disk says 100% while root can still write.

It also explains quieter failures: tmpfs directories count against memory, data written while a disk failed to mount hides under the mount point and fills root, and /etc/fstab mistakes drop a server into emergency mode at boot - a classic incident you can prevent with nofail and findmnt --verify.

Commands in this lesson

lsblk findmnt df tune2fs du

FAQ

Why is Used plus Avail less than Size in df?

ext4 reserves a percentage of blocks, 5% by default, for root. df counts those blocks neither as used nor as available to ordinary users. So Size minus Used is larger than Avail by the reserve. Use% is computed as Used divided by Used plus Avail and rounded up, which is why it reaches 100% when users run out, even though root still has the reserve.

What happens if an fstab entry fails at boot?

By default a failing mount of a required filesystem makes systemd wait for the device (90 seconds) and then drop into emergency mode, a root shell on the console, which on a headless or cloud VM means no SSH. Add nofail to entries that are not essential, and consider x-systemd.device-timeout=10s. Always test fstab changes with sudo mount -a and findmnt --verify before rebooting.

How many inodes does a filesystem have?

For ext4 it is fixed at creation, by default about one inode per 16 KiB of space, so a 2 GB volume has about 131 thousand, or fewer if created with a larger ratio. df -i shows totals and usage; tune2fs -l shows the inode count. You cannot add inodes to ext4 without recreating it (or growing it, which adds inodes proportionally). XFS and btrfs allocate inodes dynamically.

Does tmpfs use disk?

No, tmpfs lives in RAM (and can be swapped out). /run, /dev/shm and sometimes /tmp are tmpfs. Their contents count as memory usage, "shared" in free, and inside a cgroup they count against the memory limit, so a service writing a big file to a tmpfs can hit its own MemoryMax.

What is the difference between mount and /etc/fstab?

mount attaches a filesystem now, and it disappears at the next reboot. /etc/fstab lists filesystems that systemd mounts at boot (it generates .mount units from it). Changes to fstab need sudo systemctl daemon-reload and sudo mount -a to take effect without a reboot. For persistent mounts, use UUIDs from blkid rather than device names like /dev/vdb1, which can change.

In an interview Junior

What does df tell you, and what do its columns mean?

df -h reports every mounted filesystem:

Each filesystem is independent: / can be fine while /data is full, so check the right one - df -h /path or findmnt -T /path. df -i shows the other resource, inodes: when IFree hits 0 you get the same "No space left" with bytes to spare.

Also asked: How do you find which filesystem a path lives on? · What is /etc/fstab for? · How can files hide underneath a mount point?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.