Text Processing & jq: glossary
50 terms, in plain words, with where each one is taught.
- accumulator (accumulators, accumulation)
- A variable you add to on every line (
sum += $10) and print at the end. Lesson 7.8 - anchor (anchors, anchored)
- A regex part that matches a position rather than a character:
^start of line,$end of line. Lesson 7.3 - array slice (.[0:3])
- A range of elements from an array:
.[0:3]is elements 0, 1 and 2. Lesson 7.11 - awk program (awk programs)
- The text in single quotes after awk: a list of
PATTERN { ACTION }rules applied to every line. Lesson 7.8 - back-reference (back-references, backreference, \1)
\1,\2...: the text a capture group matched, reused later in the pattern or in a sed replacement. Lesson 7.3- BEGIN (BEGIN block)
- In awk, a block that runs once before the first line is read (set a separator, print a header). Its partner END runs once after the last line. Lesson 7.8
- BEGIN block (END block, BEGIN/END)
- In awk,
BEGIN { }runs once before the first line andEND { }once after the last - where totals get printed. Lesson 7.8 - BRE (basic regular expressions)
- The basic regex dialect grep and sed use by default:
+ ? | ( ) { }are plain characters unless you put a backslash in front. Lesson 7.3 - character class (character classes, bracket expression)
- A regex part in square brackets that matches exactly one character from a set:
[0-9]a digit,[^"]anything except a quote. Lesson 7.1 - composite key (composite keys)
- A lookup key made by gluing two fields together, like
$1" "$7(client and path), to count combinations. Lesson 7.8 - context lines (lines of context)
- The lines printed around each match with grep
-B N(before),-A N(after) or-C N(both), which usually show why something happened. Lesson 7.1 - endpoint (endpoints)
- One path of an API, such as
/api/payments/refund. Lesson 7.1 - ERE (extended regular expressions, extended regex)
- The extended regex dialect (
grep -E,sed -E, always in awk), where+ ? | ( ) { }work without backslashes. Lesson 7.3 - fixed string (grep -F)
- A search pattern taken literally, with no regex meaning at all (
grep -F), so a.in it is just a dot. Lesson 7.1 - fold (reduce)
- Combining a list into one value step by step: start with a value and update it once per item (jq
reduce). Lesson 7.13 - FS (OFS)
- awk's input field separator (default: any run of spaces or tabs), set with
-F:. OFS is whatprint a, bputs between a and b. Lesson 7.8 - in-place edit (edit in place, sed -i, -i.bak)
- Writing the changed text back into the file itself instead of printing it (
sed -i).-i.bakkeeps the original as file.bak. Lesson 7.6 - injection (injection bug)
- A bug where data is pasted into program text, so a crafted value (with a quote in it) changes what the program does.
--argavoids it in jq. - input record (records (awk))
- awk's word for one line of input (one unit of data). awk splits each record into fields. Lesson 7.8
- JavaScript (JS)
- The programming language of web pages. JSON was borrowed from how JavaScript writes objects and lists. Lesson 7.11
- jq filter (jq filters, jq program)
- The small program you give jq, like
.items[] | .name. It describes what to take out of the JSON;|chains filters. Lesson 7.11 - jq path (path expression)
- The chain of keys and indexes that leads to a value, like
.items[0].metadata.name;|=updates the value at a path. Lesson 7.13 - JSON array (JSON arrays)
- In JSON, an ordered list of values inside
[ ]. Lesson 7.11 - JSON key (JSON keys)
- The name on the left of a
:in a JSON object (or the name of a slot in an associative array); the value is what it holds. Lesson 7.11 - JSON object (JSON objects)
- In JSON, a set of
"key": valuepairs inside{ }. Lesson 7.11 - label (labels)
- A free-form
key: valuetag attached to something (team: payments) so it can be grouped and searched. Lesson 7.13 - lazy match (non-greedy)
- Matching as little as possible (
.*?in PCRE) - the opposite of greedy. Lesson 7.3 - lexical sort (lexical, sorted as text)
- Sorting as text, one character at a time, so "100" comes before "99".
sort -nsorts as numbers instead. Lesson 7.4 - literal (literal character, literal characters)
- Matched or kept exactly as written, with no special meaning: a literal dot matches only ".". Lesson 7.3
- locale (locales)
- The language and region settings a program runs with; they can change how letters are sorted and what
[a-z]covers. Lesson 7.3 - lookaround (lookarounds, lookahead, lookbehind)
- A PCRE check on what comes just before or after a match, without including it in the match:
(?<=user=)\w+is the word after "user=". Lesson 7.3 - mawk (gawk)
- Two versions of awk. Ubuntu ships mawk; gawk (GNU awk) has a few extras and installs with
sudo apt install gawk. Lesson 7.8 - metacharacter (metacharacters, regex operator)
- A character with a special meaning in a regex, like
.*+^$. Put a backslash in front to make it literal. Lesson 7.3 - newline-delimited JSON (NDJSON, JSON lines)
- A file with one complete JSON object per line - a common format for application logs. Lesson 7.13
- NF ($NF)
- awk's number of fields on the current line;
$NFis therefore the last field. Lesson 7.8 - NR
- awk's count of lines read so far; in an END block, the total number of lines. Lesson 7.8
- null
- JSON's value for "nothing here". jq gives null when you ask for a key that does not exist. Lesson 7.11
- PCRE (Perl regex, grep -P)
- The Perl-style regex dialect (
grep -P), which adds shortcuts like\d(digit) and lookarounds. Lesson 7.3 - POSIX character class ([[:digit:]], [[:space:]], [[:alpha:]])
- A named set of characters used inside brackets, like
[[:digit:]]or[[:space:]]. Works the same in every regex dialect. Lesson 7.3 - quantifier (quantifiers)
- A regex part saying how many times the thing before it may repeat:
*any number,+one or more,?zero or one,{2,5}two to five. Lesson 7.3 - raw output (jq -r)
- jq's
-roption: print strings as plain text, without the JSON quotes, so other commands can use them. Lesson 7.11 - recursive descent (..)
- jq's
..: visit every value at every depth of the document... | .image? // emptyfinds animagekey anywhere. Lesson 7.11 - sed address (line range)
- In sed, what goes before a command to pick lines: a number (
3), a range (3,8), a regex (/^#/) or$for the last line. Lesson 7.6 - slurp (jq -s)
- jq's
-s: read every JSON document in the input into one array, so you can aggregate over all of them. Lesson 7.13 - stream editor
- sed: a tool that reads text line by line, changes it by rules you give it (like
s/old/new/), and prints the result. Lesson 7.6 - string interpolation (\(expr))
- Putting a value inside a string. In jq:
"\(.name) is \(.phase)". Lesson 7.11 - substitution (s///, s/old/new/)
- sed's
s/PATTERN/REPLACEMENT/command: replace the text a regex matched. A trailinggreplaces every match on the line. Lesson 7.6 - text stream (stream of lines)
- Text flowing from one command to the next through a pipe, line by line, without being saved in a file first. Lesson 7.4
- top-N pipeline (sort | uniq -c | sort -rn, counting pipeline)
... | sort | uniq -c | sort -rn | head: sort so equal lines touch, count each run, sort by the count, keep the top lines. Lesson 7.4- web server (web servers)
- A program that answers requests from browsers and other programs over the network and sends back pages or data. nginx is one. Lesson 7.1