Lesson 4.1 · Filesystem, Permissions, Disk · 15 min read

Where things live

Q: What is the difference between /usr/local and /opt?

/usr/local follows the normal hierarchy (bin, lib, share) for software you install yourself, outside the package manager, and /usr/local/bin is on PATH. /opt holds self-contained packages, each in its own directory with its own bin and lib, such as /opt/app or vendor software. Both survive package upgrades. Put a single hand-written script in /usr/local/bin, a whole application in /opt.

Q: Is /tmp cleared on reboot?

On Ubuntu, systemd-tmpfiles cleans /tmp at boot and also removes old files periodically based on age. Some distributions mount /tmp as tmpfs in RAM, which empties it on every reboot by definition. /var/tmp is meant for temporary files that should survive reboots and is cleaned less aggressively. Never keep anything you care about in either.

Q: What is merged /usr?

On modern Ubuntu, /bin, /sbin and /lib are symlinks to /usr/bin, /usr/sbin and /usr/lib. There is one real location for programs and libraries, so /bin/bash and /usr/bin/bash are the same file. It simplifies packaging and makes /usr a self-contained, possibly read-only, snapshot of the OS. Old scripts using /bin/... keep working.

Q: Why is /run a tmpfs?

/run holds runtime state that only makes sense for the current boot: PID files, sockets, lock files, systemd's runtime units. Keeping it in RAM means it starts empty after every reboot, so stale PID files and sockets from before a crash cannot confuse services. It replaced /var/run, which is now a symlink to /run.

Q: Why is group disk effectively root?

Members of disk can read and often write raw block devices like /dev/vda directly. Reading the raw device bypasses every file permission: you can extract /etc/shadow, private keys or any database file with filesystem tools. Writing it can modify anything. So membership in disk is as sensitive as sudo, and it should appear in access reviews like any other privilege.

Q: What are the main top-level directories on a Linux system, and what goes in each?

- /etc - configuration, text files you edit and keep in git. - /var/log - logs (rotated by logrotate; the persistent journal too). /var/lib - program state: databases, package lists - the part that needs backups. - /usr - installed software; /usr/lib is the package's (never edit), /usr/local is yours, /opt self-contained third-party software. On Ubuntu /bin, /sbin and /lib are symlinks into /usr. - /home - people's home directories; /root is root's. - /tmp - scratch for anyone; /run - runtime state since boot (a tmpfs in RAM, empty after a reboot). - /proc and /sys - not on disk: the kernel's live view of processes, devices and cgroups. - /dev - devices as files (/dev/null, /dev/vda); /srv - data the machine serves. Two distinctions matter: /etc vs /var/lib (config you restore from git vs state you restore from backup), and /usr/lib vs /usr/local (overwritten by upgrades vs yours).

In plain words

Think of a well-run house where everyone agrees where things go: letters in the drawer by the door, tools in the garage, food in the kitchen, the fuse box in the hallway. A visitor who has never been there can still find the fuse box.

The Filesystem Hierarchy Standard is that agreement for Linux. /etc is the drawer of instructions (config), /var/log is the diary (logs), /var/lib is the pantry of things programs keep (state: databases, package lists), /usr/local/bin is your own toolbox, /tmp is the scrap paper. /proc and /sys are not cupboards at all: they are windows the kernel opens onto its own live state, like /sys/fs/cgroup/system.slice/orders.service/memory.max.

Why a map of the tree saves you

A service is misbehaving and you need its config, its logs and its data. On an unfamiliar box you do not want to search the whole disk for them. Linux puts each kind of file in an agreed place, so you know where to look before you look.

What you need to know already: 1.3 (the tree, absolute paths, ls -la), 3.14 (/proc/<pid>/), 2.26 (cgroups and memory.max).

That agreement is called the FHS (Filesystem Hierarchy Standard): a list of top-level directories and what belongs in each. Every Linux distribution follows it closely.

The tour, in the order you will need them

Quick check

$ readlink /bin

On Ubuntu, is /bin a real directory?

yes, it holds the basic commands
no, it is a symlink: usr/bin
no, it is a symlink: /sbin
it does not exist any more

Answer

no, it is a symlink: usr/bin - Ubuntu merged /bin, /sbin and /lib into /usr: the old names are symlinks, kept so old scripts still work. readlink prints the target as stored, here relative.

/etc          configuration. Text files you edit and back up. Nothing else.
/var/log      logs. Old ones are rotated (renamed, compressed, eventually
              deleted) by a tool called logrotate. The journal from Ch 2
              lives under /var/log/journal when it is persistent.
/var/lib      service STATE: databases, package lists, anything a program
              keeps between runs. This is the directory that grows and
              that you must not casually delete.
/proc         not a disk. The kernel's live view of processes, shown as
              files: /proc/<pid>/..., /proc/meminfo, /proc/mounts.
/sys          devices and kernel subsystems as files. /sys/fs/cgroup is
              where every memory and CPU limit on the box is visible.
/tmp          scratch space for anyone. Cleaned at boot.
/run          runtime state since boot: PID files, sockets. It is a tmpfs
              (a filesystem kept in RAM), so it is empty after a reboot.
/opt          self-contained third-party software. /opt/app here.
/usr/local    things YOU install: /usr/local/bin is on PATH and is where a
              hand-written script belongs.
/usr/lib      package-installed libraries and unit files. Never edit.
/srv          data this machine serves to others (web files, shares).
/dev          devices as files: /dev/null, /dev/vda (the disk).
/home         humans' home directories. /root is root's home - not under
              /home, so it is still there when /home is a separate disk
              that failed to mount.

"State" here means data a program writes for itself and needs next time it starts - as opposed to configuration, which a human writes.

Two distinctions worth having straight

/etc vs /var/lib. Config you wrote versus state the program manages. You restore /etc from git; you restore /var/lib from a backup. Mixing them up is how a "clean reinstall" loses data.

/usr/lib vs /usr/local. Package-managed versus yours. Anything you put in /usr/lib will be overwritten by the next upgrade, and anything you put in /usr/local will survive it.

On Ubuntu /bin, /sbin and /lib are symlinks into /usr - a symlink is a tiny file that just points at another path (4.13 covers them properly). This arrangement is called "merged /usr":

$ ls -ld /bin /sbin /lib
lrwxrwxrwx 1 root root 7 Sep 14 17:43 /bin -> usr/bin
lrwxrwxrwx 1 root root 7 Sep 14 17:43 /lib -> usr/lib
lrwxrwxrwx 1 root root 8 Sep 14 17:43 /sbin -> usr/sbin

ls -ld: -l long listing, -d show the directory's own line instead of its contents. The leading l in lrwxrwxrwx means "symlink", and -> usr/bin is where it points.

/proc and /sys, the answers without the tools

You met /proc/<pid>/ in 3.14. The point here is where it sits: everything ps, top and free print is read out of /proc, so on a stripped-down system with no ps installed, cat still gets you the answers.

$ P=$(systemctl show -p MainPID --value orders); echo $P
1210
$ cat /proc/$P/cmdline | tr '\0' ' '; echo
/usr/bin/java -Xmx512m -jar /opt/app/orders.jar
$ grep -E 'VmRSS|VmSize|Threads|Uid' /proc/$P/status
Uid:	1001	1001	1001	1001
VmSize:	 4200000 kB
VmRSS:	  612000 kB
Threads:	64

cmdline separates arguments with NUL bytes (byte value 0), not spaces. tr '\0' ' ' ("translate") swaps each NUL for a space so the words do not run together; the echo adds the final newline.
grep -E 'A|B' keeps lines matching A or B (-E lets you use | for "or").
Uid: has four columns (real, effective, saved, filesystem UID); 1001 is appuser. VmRSS is the memory really in RAM, in kB - the same number ps -o rss shows. (orders is a Java program; java is the program that runs it.)

$ head -3 /proc/mounts
/dev/mapper/ubuntu--vg-ubuntu--lv / ext4 rw,relatime 0 0
/dev/vda2 /boot ext4 rw,relatime 0 0
/dev/vda1 /boot/efi vfat rw,relatime,fmask=0077,dmask=0077 0 0

/proc/mounts is the kernel's own list of mounts: which storage device is attached at which directory. Columns: device, mount point (the directory it appears under), filesystem type (ext4 is Ubuntu's standard disk format), options (rw = read-write). 4.18 explains mounts in full.

Under /sys are the cgroup files from 2.26:

$ cat /sys/fs/cgroup/system.slice/orders.service/memory.max
max
$ cat /sys/fs/cgroup/system.slice/orders.service/memory.current
626688000

max means no limit on that unit; memory.current is bytes in use right now.

/dev: devices as files

Quick check

$ ls -l /dev/null

What is the first letter of /dev/null's mode?

-, a regular file
d, a directory
c, a character device
l, a symlink

Answer

c, a character device - c = character device: a file that is really a door into a kernel driver. Where the size usually is, ls shows the driver's two numbers (1, 3).

$ ls -l /dev/null /dev/vda
crw-rw-rw- 1 root root 1, 3 Sep 14 17:43 /dev/null
brw-rw---- 1 root disk 1, 3 Sep 14 17:43 /dev/vda

The first letter is the type: c character device (a stream of bytes, like a terminal or /dev/null), b block device (storage read in fixed-size blocks, like a disk). Where a file shows its size, a device shows major, minor numbers: which driver, and which instance of it.

/dev/vda belongs to group disk. Anyone in that group can read the raw disk, bypassing every file permission on it - which is why disk membership is effectively root.

What you can now do

Say where a service's config, logs and state live before searching.
Read a process's command line, memory and the mount table with cat.
Tell a block device from a character device in ls -l.

Why it helps

On any unfamiliar server or node, knowing the layout tells you where to look before you even search: config in /etc, the service's data in /var/lib/name, logs in /var/log or the journal, the cgroup limits under /sys/fs/cgroup. It is also what makes backups and rebuilds correct: /etc goes in git or config management, /var/lib needs real backups, and a "clean reinstall" that wipes /var/lib loses the database.

It also turns incidents into lookups: reading /proc/PID/cmdline or /sys/fs/cgroup/.../memory.max answers questions on a box where ps or free are missing, and knowing that group disk can read raw devices is a real security review point.

Commands in this lesson

readlink ls cat grep head

FAQ

What is the difference between /usr/local and /opt?

/usr/local follows the normal hierarchy (bin, lib, share) for software you install yourself, outside the package manager, and /usr/local/bin is on PATH. /opt holds self-contained packages, each in its own directory with its own bin and lib, such as /opt/app or vendor software. Both survive package upgrades. Put a single hand-written script in /usr/local/bin, a whole application in /opt.

Is /tmp cleared on reboot?

On Ubuntu, systemd-tmpfiles cleans /tmp at boot and also removes old files periodically based on age. Some distributions mount /tmp as tmpfs in RAM, which empties it on every reboot by definition. /var/tmp is meant for temporary files that should survive reboots and is cleaned less aggressively. Never keep anything you care about in either.

What is merged /usr?

On modern Ubuntu, /bin, /sbin and /lib are symlinks to /usr/bin, /usr/sbin and /usr/lib. There is one real location for programs and libraries, so /bin/bash and /usr/bin/bash are the same file. It simplifies packaging and makes /usr a self-contained, possibly read-only, snapshot of the OS. Old scripts using /bin/... keep working.

Why is /run a tmpfs?

/run holds runtime state that only makes sense for the current boot: PID files, sockets, lock files, systemd's runtime units. Keeping it in RAM means it starts empty after every reboot, so stale PID files and sockets from before a crash cannot confuse services. It replaced /var/run, which is now a symlink to /run.

Why is group disk effectively root?

Members of disk can read and often write raw block devices like /dev/vda directly. Reading the raw device bypasses every file permission: you can extract /etc/shadow, private keys or any database file with filesystem tools. Writing it can modify anything. So membership in disk is as sensitive as sudo, and it should appear in access reviews like any other privilege.

In an interview Junior

What are the main top-level directories on a Linux system, and what goes in each?

/etc - configuration, text files you edit and keep in git.
/var/log - logs (rotated by logrotate; the persistent journal too). /var/lib - program state: databases, package lists - the part that needs backups.
/usr - installed software; /usr/lib is the package's (never edit), /usr/local is yours, /opt self-contained third-party software. On Ubuntu /bin, /sbin and /lib are symlinks into /usr.
/home - people's home directories; /root is root's.
/tmp - scratch for anyone; /run - runtime state since boot (a tmpfs in RAM, empty after a reboot).
/proc and /sys - not on disk: the kernel's live view of processes, devices and cgroups.
/dev - devices as files (/dev/null, /dev/vda); /srv - data the machine serves.

Two distinctions matter: /etc vs /var/lib (config you restore from git vs state you restore from backup), and /usr/lib vs /usr/local (overwritten by upgrades vs yours).

Also asked: What is the difference between /proc and /sys? · What is the difference between a block device and a character device? · Where should a script you wrote yourself live, and why?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.