OnCallReadyStart learning

Text Processing & jq: glossary

50 terms, in plain words, with where each one is taught.

accumulator (accumulators, accumulation)
A variable you add to on every line (sum += $10) and print at the end. Lesson 7.8
anchor (anchors, anchored)
A regex part that matches a position rather than a character: ^ start of line, $ end of line. Lesson 7.3
array slice (.[0:3])
A range of elements from an array: .[0:3] is elements 0, 1 and 2. Lesson 7.11
awk program (awk programs)
The text in single quotes after awk: a list of PATTERN { ACTION } rules applied to every line. Lesson 7.8
back-reference (back-references, backreference, \1)
\1, \2...: the text a capture group matched, reused later in the pattern or in a sed replacement. Lesson 7.3
BEGIN (BEGIN block)
In awk, a block that runs once before the first line is read (set a separator, print a header). Its partner END runs once after the last line. Lesson 7.8
BEGIN block (END block, BEGIN/END)
In awk, BEGIN { } runs once before the first line and END { } once after the last - where totals get printed. Lesson 7.8
BRE (basic regular expressions)
The basic regex dialect grep and sed use by default: + ? | ( ) { } are plain characters unless you put a backslash in front. Lesson 7.3
character class (character classes, bracket expression)
A regex part in square brackets that matches exactly one character from a set: [0-9] a digit, [^"] anything except a quote. Lesson 7.1
composite key (composite keys)
A lookup key made by gluing two fields together, like $1" "$7 (client and path), to count combinations. Lesson 7.8
context lines (lines of context)
The lines printed around each match with grep -B N (before), -A N (after) or -C N (both), which usually show why something happened. Lesson 7.1
endpoint (endpoints)
One path of an API, such as /api/payments/refund. Lesson 7.1
ERE (extended regular expressions, extended regex)
The extended regex dialect (grep -E, sed -E, always in awk), where + ? | ( ) { } work without backslashes. Lesson 7.3
fixed string (grep -F)
A search pattern taken literally, with no regex meaning at all (grep -F), so a . in it is just a dot. Lesson 7.1
fold (reduce)
Combining a list into one value step by step: start with a value and update it once per item (jq reduce). Lesson 7.13
FS (OFS)
awk's input field separator (default: any run of spaces or tabs), set with -F:. OFS is what print a, b puts between a and b. Lesson 7.8
in-place edit (edit in place, sed -i, -i.bak)
Writing the changed text back into the file itself instead of printing it (sed -i). -i.bak keeps the original as file.bak. Lesson 7.6
injection (injection bug)
A bug where data is pasted into program text, so a crafted value (with a quote in it) changes what the program does. --arg avoids it in jq.
input record (records (awk))
awk's word for one line of input (one unit of data). awk splits each record into fields. Lesson 7.8
JavaScript (JS)
The programming language of web pages. JSON was borrowed from how JavaScript writes objects and lists. Lesson 7.11
jq filter (jq filters, jq program)
The small program you give jq, like .items[] | .name. It describes what to take out of the JSON; | chains filters. Lesson 7.11
jq path (path expression)
The chain of keys and indexes that leads to a value, like .items[0].metadata.name; |= updates the value at a path. Lesson 7.13
JSON array (JSON arrays)
In JSON, an ordered list of values inside [ ]. Lesson 7.11
JSON key (JSON keys)
The name on the left of a : in a JSON object (or the name of a slot in an associative array); the value is what it holds. Lesson 7.11
JSON object (JSON objects)
In JSON, a set of "key": value pairs inside { }. Lesson 7.11
label (labels)
A free-form key: value tag attached to something (team: payments) so it can be grouped and searched. Lesson 7.13
lazy match (non-greedy)
Matching as little as possible (.*? in PCRE) - the opposite of greedy. Lesson 7.3
lexical sort (lexical, sorted as text)
Sorting as text, one character at a time, so "100" comes before "99". sort -n sorts as numbers instead. Lesson 7.4
literal (literal character, literal characters)
Matched or kept exactly as written, with no special meaning: a literal dot matches only ".". Lesson 7.3
locale (locales)
The language and region settings a program runs with; they can change how letters are sorted and what [a-z] covers. Lesson 7.3
lookaround (lookarounds, lookahead, lookbehind)
A PCRE check on what comes just before or after a match, without including it in the match: (?<=user=)\w+ is the word after "user=". Lesson 7.3
mawk (gawk)
Two versions of awk. Ubuntu ships mawk; gawk (GNU awk) has a few extras and installs with sudo apt install gawk. Lesson 7.8
metacharacter (metacharacters, regex operator)
A character with a special meaning in a regex, like . * + ^ $. Put a backslash in front to make it literal. Lesson 7.3
newline-delimited JSON (NDJSON, JSON lines)
A file with one complete JSON object per line - a common format for application logs. Lesson 7.13
NF ($NF)
awk's number of fields on the current line; $NF is therefore the last field. Lesson 7.8
NR
awk's count of lines read so far; in an END block, the total number of lines. Lesson 7.8
null
JSON's value for "nothing here". jq gives null when you ask for a key that does not exist. Lesson 7.11
PCRE (Perl regex, grep -P)
The Perl-style regex dialect (grep -P), which adds shortcuts like \d (digit) and lookarounds. Lesson 7.3
POSIX character class ([[:digit:]], [[:space:]], [[:alpha:]])
A named set of characters used inside brackets, like [[:digit:]] or [[:space:]]. Works the same in every regex dialect. Lesson 7.3
quantifier (quantifiers)
A regex part saying how many times the thing before it may repeat: * any number, + one or more, ? zero or one, {2,5} two to five. Lesson 7.3
raw output (jq -r)
jq's -r option: print strings as plain text, without the JSON quotes, so other commands can use them. Lesson 7.11
recursive descent (..)
jq's ..: visit every value at every depth of the document. .. | .image? // empty finds an image key anywhere. Lesson 7.11
sed address (line range)
In sed, what goes before a command to pick lines: a number (3), a range (3,8), a regex (/^#/) or $ for the last line. Lesson 7.6
slurp (jq -s)
jq's -s: read every JSON document in the input into one array, so you can aggregate over all of them. Lesson 7.13
stream editor
sed: a tool that reads text line by line, changes it by rules you give it (like s/old/new/), and prints the result. Lesson 7.6
string interpolation (\(expr))
Putting a value inside a string. In jq: "\(.name) is \(.phase)". Lesson 7.11
substitution (s///, s/old/new/)
sed's s/PATTERN/REPLACEMENT/ command: replace the text a regex matched. A trailing g replaces every match on the line. Lesson 7.6
text stream (stream of lines)
Text flowing from one command to the next through a pipe, line by line, without being saved in a file first. Lesson 7.4
top-N pipeline (sort | uniq -c | sort -rn, counting pipeline)
... | sort | uniq -c | sort -rn | head: sort so equal lines touch, count each run, sort by the count, keep the top lines. Lesson 7.4
web server (web servers)
A program that answers requests from browsers and other programs over the network and sends back pages or data. nginx is one. Lesson 7.1