OnCallReady

Lesson 6.12 · Bash Scripting · 13 min read

Parameter expansion and arrays

In plain words

Imagine a label maker with a few clever buttons. One button says "if the label is blank, print this instead" (${var:-default}). Another says "if the label is blank, stop everything and complain" (${var:?message}). Others trim the start or end of a label: cut everything up to the last slash to get just the file name (${p##*/}), or cut the last dot and what follows (${f%.gz}).

That is parameter expansion: bash doing small text jobs on variables without calling other programs. Arrays are a row of labelled boxes: hosts=(web1 web2 "db 1"), and "${hosts[@]}" hands out every box separately, even the one with a space in its name. Associative arrays (declare -A) are boxes labelled by name instead of number.

Why this lesson

Scripts spend a lot of their time on small text jobs: "use this setting, or a default if it is missing", "the file name without the folder", "the name without .tar.gz". You could call a program for each (basename, dirname), but bash can do them itself, instantly, inside the ${ } of a variable. And when one variable must hold a list - three host names, say - you need an array.

What you need to know already: variables and expansion (6.1); quoting and word splitting (6.6).

The long form of $var

$var and ${var} mean the same thing. The braces let you add an operator after the name - that is parameter expansion ("parameter" is bash's word for a variable). Everything in this lesson is one of those operators.

Defaults and requirements

${var:-default}    use "default" if var is unset OR empty. var is not changed.
${var:=default}    ...and also store "default" in var.
${var:?message}    STOP the script with "message" if var is unset or empty.
${var:+alt}        use "alt" only if var IS set and not empty. (The reverse.)

${var:?} is the one to reach for at the top of a script:

: "${DEPLOY_ENV:?DEPLOY_ENV must be set}"

: is a command that does nothing (a no-op). It is there only so bash evaluates the expansion. If DEPLOY_ENV is missing, the script stops right here with your message, instead of failing confusingly ten lines later.

DEPLOY_ENV in capitals is an environment variable - a variable handed to a program by whoever started it (you met them in 2.21). DEPLOY_ENV=prod ./x.sh sets one for that single command only.

Drop the colon (${var-default}) and the test becomes "unset" only, so an empty value is accepted. Usually you want the colon.

Trimming

These cut a pattern (a glob, 6.6) off the front or the end of the value:

p=/var/log/nginx/access.log

${p##*/}     access.log            ## = longest match from the FRONT
${p#*/}      var/log/nginx/access.log   # = shortest
${p%/*}      /var/log/nginx        %  = shortest match from the END
${p%%.*}     /var/log/nginx/access %% = longest

f=archive.tar.gz
${f%.gz}     archive.tar
${f%%.*}     archive

Memory aid: # is left of $ on the keyboard (so it cuts from the left), % is right. Doubled means greedy: cut the longest possible match.

basename PATH and dirname PATH are programs that print the file part and the folder part of a path. The expansions do the same without forking (starting a new process) - which matters inside a loop over ten thousand files.

Substitution and length

${p/nginx/apache}     replace the FIRST match
${p//o/0}             replace ALL
${#p}                 length in characters
${var^^} ${var,,}     upper / lower case

Arrays

An array is one variable holding a numbered list of values. Each value is an element, and its number is its index, counting from 0:

hosts=(web1 web2 "db 1")
echo "${hosts[0]}"       web1
echo "${#hosts[@]}"      3          <- count
echo "${hosts[@]}"       all elements
for h in "${hosts[@]}"; do ...       <- ALWAYS quoted, ALWAYS [@]

hosts+=(web3)            append
unset 'hosts[1]'         remove (leaves a gap in the indices)

"${arr[@]}" with the quotes and the @. ${arr[*]} joins everything into one string; unquoted ${arr[@]} word-splits each element again. Only the quoted @ form gives you exactly the elements you put in.

An associative array uses names instead of numbers as its indices - a lookup table. The names are called keys. It must be declared first with declare -A (declare is the builtin that sets a variable's type):

declare -A ports=([http]=80 [https]=443)
echo "${ports[https]}"                         # 443
for k in "${!ports[@]}"; do echo "$k -> ${ports[$k]}"; done

${!arr[@]} gives the keys (or the indices of a normal array).

declare -i n=0 makes n an integer variable, so n+=1 adds 1 instead of gluing the text "1" on the end. That is what makes the demo counter script from 2.5 print iteration 2 rather than iteration 11.

What you can now do

Why it helps

Defaults and required values are the first lines of most deploy and backup scripts: : "${DEPLOY_ENV:?must be set}" stops the script cleanly when a variable is missing, instead of running with an empty value against the wrong target. ${dir:?} in rm -rf "${dir:?}/" is what shellcheck recommends to prevent disasters.

Trimming replaces basename, dirname and sed calls inside loops, which matters for speed over thousands of files. Arrays are the correct way to build command lines with optional flags (args+=(-name "$pattern") for find), instead of string concatenation that breaks on spaces. These are everyday tools in platform scripts and common in interviews.

FAQ

What is the difference between ${var:-x} and ${var-x}?

With the colon, the default is used when the variable is unset or empty. Without it, only when unset; an empty value is kept. The same applies to :=, :? and :+. Usually you want the colon, because an empty environment variable is almost always as useless as a missing one. Without the colon, VAR= ./script deliberately passes an empty value through.

How do I remember # versus %?

On a US keyboard # is to the left of $ and % to the right, so # trims from the start and % from the end. A single character removes the shortest match, doubled removes the longest: ${p##*/} strips everything up to the last slash (basename), ${p%/*} strips from the last slash (dirname), ${f%%.*} removes everything from the first dot.

Why must I use "${arr[@]}" with quotes and @?

Quoted "${arr[@]}" expands to each element as a separate word, exactly as stored, including spaces. "${arr[*]}" joins all elements into one string. Unquoted forms split every element again on whitespace and glob them. Only the quoted @ form preserves the array. Also $arr without an index is just the first element, a frequent bug.

Why use an array instead of a string for command options?

A string like opts="-name my file.log" must be left unquoted to split into separate arguments, and then it splits in the wrong places and globs. An array keeps each argument intact: opts=(-name "my file.log") and then find /var/log "${opts[@]}". You can also build it conditionally: [[ $old == 1 ]] && opts+=(-mtime +7).

Are associative arrays available everywhere?

They need bash 4.0 or later and an explicit declare -A name; without the declaration, bash treats the name as an indexed array and string keys evaluate to index 0. Ubuntu has bash 5. macOS still ships bash 3.2 as /bin/bash, where declare -A fails, which matters for scripts developers run on Macs. POSIX sh has no arrays at all.

In an interview Junior

Explain ${var:-default}, ${var:=default}, ${var:?message} and ${var:+alt}.

Without the colon (${var-default}) only "unset" counts, so an empty value is accepted. Why they matter: a script fails fast with a clear message instead of running with an empty value, and they work under set -u.

Also asked: How do you get a file's name, directory and extension without calling basename or dirname? · How do you store a list in an array and loop over it safely? · What is an associative array, and how do you declare one?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.