First Login: interview questions

What happens, step by step, when you type ls -l /etc and press Enter? Junior

1. The terminal sends the keystrokes to the shell (over SSH if the box is remote). bash splits the line into words: ls, -l, /etc.
2. It works out what ls is, in order: alias, function, builtin, then a file found by searching the directories in $PATH left to right. At an Ubuntu prompt ls is an alias for ls --color=auto, and the file behind it is /usr/bin/ls (type -a ls shows both).
3. bash starts /usr/bin/ls as a child process with its arguments and waits for it.
4. ls reads the directory and writes one line per entry to stdout, the screen; errors would go to stderr.
5. It exits, bash stores the exit status in $? (0 = success) and prints the next prompt.

Also asked: What is the difference between an absolute and a relative path? · Why does sudo echo x > /etc/file fail with Permission denied? · What does exit code 127 mean, and what does 137 mean?

What is the difference between a terminal, a shell and SSH? Junior

• The terminal is the window: Terminal.app or iTerm. It draws characters and sends your keystrokes somewhere; it does not understand a single command.
• The shell is the program that reads those keystrokes and decides what they mean - on Linux servers usually bash. It knows what | means and what happens when a command fails.
• SSH is the encrypted connection between the two when the shell runs on another machine: you type on the Mac, the keystrokes travel to 10.64.0.2, bash on the VM runs them, and the output travels back. OpenSSH server is a remote command line, not a remote desktop.

The prompt learner@oncall-lab:~$ tells you which user, which machine, which directory, and $ (normal user) or # (root) - read it before pressing Enter.

Also asked: What is the difference between virtualization and emulation? · What does uname -m print on an Apple Silicon VM, and why does the architecture matter? · Why practise on a Linux VM instead of the Mac's own terminal?

Learn it: 1.1 Where you are

Explain every field of an ls -l line. Junior

Take drwxr-xr-x 2 learner learner 4096 Sep 14 17:43 labs:

• d - the type: - regular file, d directory, l symlink.
• rwxr-xr-x - permissions in three groups of three: owner, group, everyone else. r read, w write, x execute (for a directory: may go into it). In octal, r=4, w=2, x=1 added per group: 755.
• 2 - the link count, how many names point at it.
• learner learner - the owning user and group.
• 4096 - size in bytes; for a directory, the size of its list of names, not its contents.
• the last-modified date, then the name.

Gotcha: ls -l /etc lists what is inside /etc; ls -ld /etc gives the line for the directory itself. stat prints every detail.

Also asked: What is the difference between an absolute and a relative path, and when does it bite you? · What does "Permission denied" tell you, compared with "No such file or directory"? · What are hidden files in Linux, and how do you see them?

Learn it: 1.3 The tree

What does mkdir -p do, and why is it preferred in scripts? Junior

It creates every missing parent directory, and it does not complain if the directory is already there. Plain mkdir a/b/c fails with "No such file or directory" when a is missing. With -p the command is safe to run twice: the second run changes nothing and still succeeds, which is what you want in anything that gets re-run.

Combined with brace expansion it builds a whole layout in one go: mkdir -p ~/lab/{systemd,processes,bash} - the shell expands the braces into three paths before mkdir starts (check with echo first).

The gotcha: -p will happily build the whole chain in the wrong place. A relative path is taken from wherever you are standing, which is exactly how a second labs folder ends up nested inside the first. So check pwd, or anchor the path with ~/ or /.

Also asked: What is the difference between > and >>? · How do you make an rm with a wildcard safer? · What is the difference between rm, rm -r and rmdir?

Learn it: 1.5 Making things

What do exit codes 0, 1, 126, 127, 130 and 137 mean? Junior

Every command returns an exit status from 0 to 255; echo $? right after it shows it.

• 0 - success. Anything else is a failure.
• 1 - generic failure, "it did not work".
• 126 - the file was found but cannot be executed (no x permission).
• 127 - command not found: a typo, not installed, or not on $PATH.
• 130 - killed by Ctrl+C: 128 + 2, SIGINT.
• 137 - killed by SIGKILL: 128 + 9. Often the kernel, when memory runs out.

The rule behind the last two: above 128 means "died from signal N", N = status - 128 (143 = SIGTERM, 15). And read $? immediately - the next command, even the echo, overwrites it.

Also asked: What is the difference between cd /some/dir && rm -rf * and cd /some/dir; rm -rf *? · What is the difference between cmd > file 2>&1 and cmd 2>&1 > file? · What are stdin, stdout and stderr?

Learn it: 1.7 Driving the shell: exit codes, chaining, redirection

How does the shell decide what to run when you type a command name? Junior

It checks four kinds of place, in order:

1. an alias - a shortcut from ~/.bashrc, like ll for ls -alF;
2. a function defined in the shell;
3. a builtin - part of bash itself, like cd and echo;
4. a file - a program found by searching the directories in $PATH, left to right.

type tells you which one wins, and type -a lists every match: ls at your prompt is an alias for ls --color=auto, and the file behind it is /usr/bin/ls. If the name contains a slash, like ./prog, there is no search: that file runs. The current directory is not on the PATH on purpose. Two consequences: cd must be a builtin (a separate program could only change its own directory), and aliases exist only in interactive shells, so they vanish in scripts and under sudo.

Also asked: You need a flag you do not remember on a server with no internet. How do you find it? · What is the difference between which and command -v? · Why does a program in the current directory need ./ in front of it?

Learn it: 1.9 Asking the box: type, which, man, --help

Why does sudo echo "x" > /etc/motd fail with Permission denied, and how do you fix it? Junior

Because the redirection is done by your own shell, as you, before sudo even starts. The shell sees > /etc/motd and tries to open that file for writing as learner; only root may write in /etc, so it fails. sudo made echo root; nobody made the > root.

The fixes make a root process do the writing:

• echo "x" | sudo tee /etc/motd - tee runs as root and writes the file (tee -a appends; add > /dev/null to hide the echo of it);
• sudo sh -c 'echo "x" > /etc/motd' - the whole command line runs in a root shell, so its > is root too.

tee is the usual choice in written instructions because the quoting is simpler.

Also asked: What is the difference between apt update and apt upgrade, and why run update first? · Why use sudo instead of logging in as root? · What does apt autoremove clean up?

Learn it: 1.11 Packages and sudo

How does a Linux machine get its IP address, and how do you check it? Junior

Usually by DHCP: at boot the machine asks the local network "anyone got an address for me?", and a DHCP server answers with an address and network size (10.64.0.2/24), a gateway (10.64.0.1), a DNS server, and a lease - how long the address is the machine's before it must ask again. On Ubuntu the "use DHCP on enp0s1" decision is a file in /etc/netplan/.

To check: ip -br a shows each interface, its state (UP) and its addresses; ip r shows the routing table, where default via 10.64.0.1 ... proto dhcp is the gateway and where it came from; hostname -I prints just the addresses. Gotcha: a lease can change, so a machine others must reach needs a fixed address or a reservation.

Also asked: What does the /24 in 10.64.0.2/24 mean? · What does an interface name like enp0s1 tell you? · Why can you reach the VM by its IP address but not by its hostname from the Mac?

Learn it: 1.13 Network identity

What is PID 1, and what happens to a process whose parent dies? Junior

PID 1 is the first process the kernel starts at boot (/sbin/init); on Ubuntu it is systemd. It starts every service in order, it is the ancestor of every process, and it cannot be killed - the kernel refuses to deliver deadly signals to it, because if it died the machine would stop.

When a parent dies before its child, the child becomes an orphan and is handed to PID 1 as its new parent. That matters because a finished process stays listed as a zombie until its parent reads its exit status; PID 1 reaps every orphan it adopted, so zombies do not pile up.

How I'd check: ps -p 1 -o comm= prints systemd, readlink -f /sbin/init confirms it from the files on disk, and pstree -p or ps -o ppid= -p PID shows who a process's parent is.

Also asked: Walk me through what happens when a Linux server boots. · How do you check which init system a machine runs? · What is the difference between a program and a process?

Learn it: 1.15 Who's in charge

How does SSH key authentication work, and which key goes where? Junior

You make a key pair with ssh-keygen -t ed25519: a private key (~/.ssh/id_ed25519, mode 600, never leaves the machine) and a public key (id_ed25519.pub, safe to give anyone). The public key goes to the server - for GitHub, into your account's settings.

When you connect, the server sends a random challenge; your machine signs it with the private key, and the server checks the signature with the public key. No password travels, and the public key cannot be used to work out the private one.

It works the other way too: the server has a host key, so you can check you reached the real server. The first connection shows its fingerprint; once you say yes it is saved in ~/.ssh/known_hosts, and a different key later gives a loud warning. ssh -T [email protected] tests the whole chain.

Also asked: What is a host key fingerprint, and why compare it before typing yes? · How do you see which remote URL a git repository pushes to? · What does git config --global user.email set, and where is it stored?

Learn it: 1.17 Git identity and an SSH key

What is the difference between a disk, a partition, a filesystem and a mount point, and how do you see them? Junior

• Disk: the raw storage device, shown as a file in /dev, like /dev/vda.
• Partition: a fixed section of a disk (vda1, vda2, vda3).
• Filesystem: the structure written onto it that turns raw space into files and directories, usually ext4. Formatting = creating one.
• Mount point: the directory where a filesystem is attached to the tree (/, /boot, /data).

Ubuntu puts LVM in between: a partition becomes a PV, PVs form a pool (VG), and slices of it (LVs) carry the filesystems and can be grown later.

lsblk draws the whole stack as a tree with sizes and mount points; df -h shows how full each mounted filesystem is. That is why the VM's / is 18G on a 40G disk: the installer gave the root LV only half the pool.

Also asked: What is the difference between thin and thick provisioning? · The root filesystem is full but the volume group has free space. How do you grow it? · What is LUKS, and when would you not use it?

Learn it: 1.20 The disk you were given