OnCallReady

Lesson 6.17 · Bash Scripting · 21 min read

Redirection and file descriptors

In plain words

Imagine a machine with three tubes: one where things go in (0, stdin), one where the finished products come out (1, stdout), and one where complaints come out (2, stderr). Normally both output tubes point at your face, the screen. Redirection is re-attaching the tubes: > file points the products tube into a bucket, 2> err points the complaints tube somewhere else, < file feeds the input from a bucket.

2>&1 means "point the complaints tube at wherever the products tube points right now", so the order you write things in matters. exec > log 2>&1 re-attaches the tubes for the rest of the script. And you can add your own tubes, numbered 3 to 9, for example to keep a way to talk to the terminal after everything else goes to a log.

Why this lesson

A script that runs at 3am must leave a record: what it did, and above all the error messages. Redirection decides where each stream goes - and one misplaced 2>&1 means the one error you needed never reached the log. You met the basics in 1.7; this lesson is the version you need inside scripts.

What you need to know already: stdin/stdout/stderr, >, >>, 2>, 2>&1 and /dev/null (1.7); file descriptors (3.16); sudo tee (1.11); { } groups (6.5).

Three numbers

A file descriptor (fd) is a small number a process uses to refer to an open file, pipe or terminal (3.16). Every process starts with three:

0  stdin     where input comes from     (the keyboard, a file, a pipe)
1  stdout    normal output              (the terminal, a file, a pipe)
2  stderr    errors and diagnostics     (the terminal - even when stdout is piped)

A redirection rewires one of them for one command:

cmd > file        stdout to file (truncate)       same as 1> file
cmd >> file       stdout to file (append)
cmd 2> file       stderr to file
cmd < file        stdin from file
cmd > out 2> err  both, separately
cmd &> file       both to the same file           (bash; same as > file 2>&1)
cmd > /dev/null   throw stdout away

The reason stderr exists: in cmd | grep x only stdout goes down the pipe, so errors still reach your terminal instead of being filtered as data.

$ ls /etc/hostname /nope
ls: cannot access '/nope': No such file or directory
/etc/hostname
$ ls /etc/hostname /nope 2>/dev/null
/etc/hostname
$ ls /etc/hostname /nope > /dev/null
ls: cannot access '/nope': No such file or directory

2>&1, and why the order matters

2>&1 means "make fd 2 point wherever fd 1 points right now". Redirections are processed left to right, so:

cmd > log 2>&1     stdout -> log, THEN stderr -> (where stdout is) = log.   Both in log.
cmd 2>&1 > log     stderr -> (where stdout is) = terminal, THEN stdout -> log.
                   Errors on the screen, output in the file.

The second is occasionally what you want - it is the idiom for "pipe only the errors":

$ ls /nope 2>&1 >/dev/null | tr a-z A-Z
LS: CANNOT ACCESS '/NOPE': NO SUCH FILE OR DIRECTORY

stderr went to the pipe (where stdout pointed at the time), then stdout was sent to /dev/null. (tr a-z A-Z translates characters: every lowercase letter to its uppercase one - here just to prove which text went down the pipe.)

cmd 2>&1 | tee log (or bash's cmd |& tee log) is how you keep a log of everything and see it.

Writing to stderr yourself

echo "error: config not found" >&2        # >&2 is short for 1>&2
die() { echo "error: $*" >&2; exit 1; }

$* is all the function's arguments as one string (6.6) - fine for a message.

Every error message in a script goes to stderr. Anything consuming your stdout - a pipe, $(...), another program reading your output - must not receive your warnings as data.

Blocks and whole scripts

Redirect a group of commands at once:

{
  echo "report for $(date +%F)"
  df -h /
  free -m
} > report.txt 2>&1

Or redirect the rest of the script with exec and no command. (exec CMD replaces the shell with CMD; with only redirections, it applies them to the shell itself, for every command after it.)

exec >> /var/log/nightly.log 2>&1       # everything from here on is logged
echo "started"                           # goes to the log

This is the standard opening of a nightly job you want a full record of. A variation keeps a copy on the terminal: exec > >(tee -a "$log") 2>&1 (>(cmd) is process substitution the other way round: a "file" that feeds cmd; tee -a appends).

Your own descriptors

fd 3-9 are yours:

$ exec 3> /tmp/fd3.txt        # open fd 3 for writing
$ echo "hi" >&3               # write to it
$ exec 3>&-                   # close it
$ cat /tmp/fd3.txt
hi

The practical use: after exec > logfile, keep a way to talk to the user:

exec 3>&1                     # fd 3 = the original stdout (the terminal)
exec > build.log 2>&1         # everything else to the log
echo "building..." >&3        # this one line still reaches the terminal

Truncation, and the safety catch

> truncates the file before the command runs. So this destroys the file:

sort data.txt > data.txt      # data.txt is emptied first; sort reads nothing

Write to a temporary file and mv it into place, or use a tool with its own in-place option (sort -o data.txt data.txt: -o = write the output to this file, safely).

set -C (noclobber, "do not clobber" = do not overwrite) makes > refuse to overwrite an existing file; >| overrides it:

$ echo a > o
$ set -C
$ echo b > o
bash: o: cannot overwrite existing file
$ echo c >| o
$ set +C                      # back to normal

sudo and redirection, once more

sudo echo x > /etc/x          # FAILS: your shell opens /etc/x, not sudo
echo x | sudo tee /etc/x      # tee (as root) opens it
echo x | sudo tee -a /etc/x   # append
sudo sh -c 'echo x > /etc/x'  # or give root the whole command line

The redirection belongs to the shell that parses the line. It is the same rule as 1.11, and the cause of half the "Permission denied" errors on a file you ran sudo for. (sh -c '...' starts a new shell that runs the quoted text - so under sudo, that shell is root and does the >.)

What you can now do

Why it helps

Logging in cron jobs, timer jobs and systemd services depends on getting redirection right. The classic lost-errors bug, cmd 2>&1 >> log, means the one message you needed during an incident never reached the log. exec >> /var/log/job.log 2>&1 at the top of a script, or exec > >(tee -a "$log") 2>&1, gives you a complete record in one line.

Writing errors to stderr (>&2) matters whenever your script's output is consumed by another tool: $(...), a pipe into another command, or a monitoring check that reads the number your script prints. And sort file > file truncating the file first, or sudo cmd > /etc/x failing, are mistakes you will see in reviews and runbooks.

Commands in this lesson

ls exec echo cat set

FAQ

Why does cmd 2>&1 > file still show errors on screen?

Redirections are applied left to right, and 2>&1 copies wherever fd 1 points at that moment. In cmd 2>&1 > file, fd 2 is copied from fd 1 while it still points at the terminal, then fd 1 alone moves to the file. So errors go to the terminal and output to the file. Write cmd > file 2>&1 to capture both.

What does exec do without a command?

exec followed only by redirections applies them to the current shell permanently, instead of to one command. exec >> log 2>&1 sends all later output of the script to the log. exec 3> file opens fd 3; exec 3>&- closes it; exec 3>&1 saves the current stdout in fd 3. With a command, exec prog replaces the shell with that program, which is why wrapper scripts often end with exec.

Why did sort data.txt > data.txt empty my file?

The shell sets up redirections before starting the command. > data.txt opens the file with truncation, so it is empty by the time sort reads it, and sort writes nothing back. The same applies to sed ... file > file and grep ... file > file. Write to a temporary file and mv it over the original, or use tools with in-place options: sort -o data.txt data.txt, sed -i.

How do I send only the errors of a command through a pipe?

Swap the streams: cmd 2>&1 >/dev/null | grep pattern. Left to right, fd 2 first copies fd 1, which points at the pipe, then fd 1 is sent to /dev/null. So only stderr reaches grep. To keep stdout somewhere useful instead of discarding it, redirect it to a file or use a spare descriptor, as in 3>&1 1>out 2>&3.

Why should my script's errors go to stderr?

Because anything consuming the script's stdout treats it as data. If a script called in $(...), piped into another command, or read by a program expecting one number prints warnings to stdout, the consumer receives garbage mixed into its data. Errors on stderr still reach the terminal or the log, separate from the data. Use echo "error: ..." >&2, or a die function that does it.

In an interview Junior

What are stdin, stdout and stderr, and how do you redirect each?

Every process starts with three file descriptors: 0 stdin (input), 1 stdout (normal output), 2 stderr (errors and diagnostics).

Why two output streams: only stdout goes down a pipe, so errors still reach you instead of being treated as data. In scripts, send your own messages to stderr: echo "error: ..." >&2.

Two gotchas: order matters - 2>&1 copies wherever fd 1 points at that moment, so cmd 2>&1 > log leaves errors on the screen. And exec >> /var/log/nightly.log 2>&1 at the top of a script logs everything after it.

Also asked: What is the difference between cmd > log 2>&1 and cmd 2>&1 > log? · Why does sort data.txt > data.txt destroy the file? · How would you log everything a nightly script does, errors included?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.