OnCallReady

Filesystem, Permissions, Disk: interview questions

The question you are most likely to get for each topic, a model answer, and what else comes up. From chapter 4 of the course.

df shows free space, but writes fail with "No space left on device". What are the causes, and in what order do you check them? Junior

Three causes, all giving the same ENOSPC error, checked cheapest first:

  1. Inodes exhausted - df -i. Every file costs one inode, the number is fixed when the filesystem is made, and millions of tiny files (sessions, caches) use them up while the bytes are fine. IFree 0 = this. Find where with du --inodes -x, then delete files (find ... -delete, not rm *).
  2. A deleted file still held open - sudo lsof +L1. The name is gone so du cannot see it, but the blocks are allocated, so df still counts them. Restart the holder, or truncate -s 0 /proc/PID/fd/N.
  3. Reserved blocks - sudo tune2fs -l /dev/X | grep -i reserved. ext4 keeps 5% for root, so a user sees Avail 0 while root can still write. On a data volume, tune2fs -m 1.

df -i first because it is instant; lsof +L1 second because it is the most common on a long-running box.

Also asked: Why can a user with read permission on a directory still not read a file inside it? · How do you list the 10 largest directories, without crossing into other filesystems? · What is the difference between a hard link and a symbolic link?

What are the main top-level directories on a Linux system, and what goes in each? Junior

Two distinctions matter: /etc vs /var/lib (config you restore from git vs state you restore from backup), and /usr/lib vs /usr/local (overwritten by upgrades vs yours).

Also asked: What is the difference between /proc and /sys? · What is the difference between a block device and a character device? · Where should a script you wrote yourself live, and why?

Learn it: 4.1 Where things live

Explain the fields of /etc/passwd, and why passwords are not stored there. Junior

Seven colon-separated fields, e.g. learner:x:1000:1000:Learner:/home/learner:/bin/bash: name, x (the password is elsewhere), UID, primary GID, comment, home directory, login shell. Service accounts like appuser get /usr/sbin/nologin, so nobody can log in as them.

The kernel only knows the numbers; /etc/passwd is the lookup table that turns UIDs into names, and every ls -l and ps needs it, so it must be readable by everyone (644). Storing password hashes there would let any user copy them and guess offline. So the hashes live in /etc/shadow, readable only by root and group shadow (640). $y$ marks yescrypt; ! means locked - right for a service account.

UID ranges on Ubuntu: 0 root, 1-999 system accounts, 1000+ humans, 65534 nobody. To look someone up, use getent passwd name: it also finds users from a central directory.

Also asked: You added a service account to a group, but the service still gets Permission denied. Why? · What is the difference between usermod -G and usermod -aG? · What is the difference between a primary group and a supplementary group?

Learn it: 4.3 Users, groups, and who you are right now

What does chmod 754 mean, and how does Linux decide which permissions apply to you? Junior

Each digit is one triple, adding r=4, w=2, x=1: owner 7 = rwx, group 5 = r-x, others 4 = r--. As a string, -rwxr-xr--.

The kernel picks exactly one triple and ignores the others:

  1. you own the file -> the owner triple, full stop;
  2. else the file's group is one of your groups -> the group triple;
  3. else -> others.

It does not add up your best option: a file ----rw---- that you own is unreadable to you, even if you are in its group. Root skips read and write checks, but needs some x bit to execute.

chmod 640 f sets the mode absolutely; chmod g+r f changes only the bits you name. chown appuser:ops f sets owner and group (only root may give a file away). Verify with stat -c '%a %U:%G %n' f.

Also asked: A developer ran chmod -R 777 to fix a permission error. What is wrong with that, and what would you do instead? · What is the difference between "Operation not permitted" and "Permission denied"? · What does the capital X in chmod -R g+rX do?

Learn it: 4.5 Permission bits and octal

What do read, write and execute mean on a directory? Junior

A directory is a table of name -> inode, and the bits apply to that table:

Consequences: --x (as in 711) lets people reach a file by name without browsing the directory. w on a directory lets you delete files inside that you do not own and cannot read - which is why /tmp needs the sticky bit. And for a path, you need x on every directory from / down, then the right bit on the file.

To debug: namei -l /srv/vault/report.txt shows the mode at each level, and sudo -u appuser cat ... proves it as that user.

Also asked: A web server cannot read a 644 file under a user's home directory. How do you find out why? · Why is write permission on a directory more dangerous than write on the file? · How do you test whether another user can read a file, without guessing?

Learn it: 4.7 r and x mean something else on a directory

What are setuid, setgid and the sticky bit, and where are they used? Junior

Three extra bits, written as a fourth octal digit in front:

ls -l shows them as s/t; a capital S/T means the bit is set without the x underneath - almost always a mistake.

Also asked: What is umask, and what mode does a new file get with umask 027? · How would you set up a directory a whole team can share and edit? · What does NoNewPrivileges=yes protect against?

Learn it: 4.9 setuid, setgid, sticky and umask

What is the difference between a hard link and a symbolic link? Junior

A file is really an inode (metadata plus pointers to the data); a directory entry is just a name pointing at an inode number.

ls -li tells them apart: hard links share the inode number; a symlink is type l with -> target. And the data is only freed when the link count and the number of open file descriptors both reach zero - why rm of an open log frees nothing.

Also asked: You deleted a 40 GB log but disk usage did not change. Why? · Why does mv to another disk take minutes when mv on the same disk is instant? · How are relative symlink targets resolved?

Learn it: 4.13 Links and inodes

How would you delete log files older than 30 days in /var/log/app, safely? Junior

Print first, then delete:

sudo find /var/log/app -type f -name '*.log*' -mtime +30 -print
sudo find /var/log/app -type f -name '*.log*' -mtime +30 | wc -l
sudo find /var/log/app -type f -name '*.log*' -mtime +30 -delete

find also does not hit "Argument list too long" the way rm * does, because it never builds a list of names.

Also asked: How do you find all files larger than 1 GB on one filesystem? · Why does find /etc -type d -name 's*' -o -name 'host*' also print files? · What is the difference between -exec cmd {} \; and -exec cmd {} +?

Learn it: 4.15 find: the expression language

What does df tell you, and what do its columns mean? Junior

df -h reports every mounted filesystem:

Each filesystem is independent: / can be fine while /data is full, so check the right one - df -h /path or findmnt -T /path. df -i shows the other resource, inodes: when IFree hits 0 you get the same "No space left" with bytes to spare.

Also asked: How do you find which filesystem a path lives on? · What is /etc/fstab for? · How can files hide underneath a mount point?

Learn it: 4.18 Disks, mounts and what df is really telling you

How would you list the 10 largest directories on a server, without crossing into other filesystems? Junior

sudo du -xh / --max-depth=2 2>/dev/null | sort -h | tail -10

Then descend into the top entry with the same command, or use sudo find /data -size +1G -type f for single big files. If du's total is far below what df says is used, the space is held by deleted-but-open files (lsof +L1) or hidden under a mount point.

Also asked: df says 42 GB used but du finds 8 KB. What is happening? · Why does rm /srv/cache/sessions/* fail with "Argument list too long", and what do you use instead? · What is the difference between apparent size and disk usage?

Learn it: 4.19 du, df and finding the space

You deleted a 40 GB log file, but disk usage did not change. Why, and how do you get the space back? Junior

rm removes the name, not the file. The data is freed only when the link count is 0 and no process has the file open - and a running service still has it open. So df still counts the blocks while du can no longer see the file: df and du disagreeing by the size of one file is the tell.

Find the holder: sudo lsof +L1 lists open files with no name left, e.g. logwriter 1302 appuser 3w ... /data/app/debug.log (deleted).

Get the space back, in order of preference: restart the service, so it closes the old file and opens a new one; or, without a restart, empty it through the descriptor: sudo truncate -s 0 /proc/1302/fd/3. Not kill -9. Then fix the cause: rotation the program knows about (logrotate's copytruncate, or a signal to reopen the log), or log to the journal.

Also asked: What is inode exhaustion, and how do you spot it? · Why does ext4 reserve 5% of blocks for root, and when would you lower it? · A user gets "No space left on device" but root can still write. What does that tell you?

Learn it: 4.21 df shows free space but writes fail

Practise these answers with flashcards and labs Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.