Observability I: Prometheus & PromQL: glossary
101 terms, in plain words, with where each one is taught.
- __address__
- The hidden label holding host:port to scrape. If nothing sets instance, instance is copied from it. Lesson 43.4
- __name__
- The metric name, stored as a label. Lets you select or relabel by name: {__name__=~"node_memory_.*"}. Lesson 43.4
- _over_time functions (max_over_time, avg_over_time, min_over_time)
- Functions that summarise a gauge's samples over a window: max_over_time(x[1h]) is the highest value in the last hour. Lesson 43.8
- --data-urlencode
- A curl option that sends a form field with special characters safely encoded. Use it for PromQL: raw brackets and braces in a URL break. Lesson 43.2
- /api/v1/rules (rules API)
- The Prometheus endpoint listing every loaded rule group and whether each rule's last evaluation was ok. Lesson 43.27
- /api/v1/status/config
- The Prometheus endpoint returning the config it is RUNNING, which differs from the file on disk after a rejected reload.
- /api/v1/status/tsdb (TSDB status)
- The Prometheus endpoint listing the metrics with most series and the labels with most values. Where cardinality problems are found.
- /api/v1/targets (targets API)
- The Prometheus HTTP endpoint listing every target with its labels, health (up/down) and lastError, as JSON. Lesson 43.2
- absent() (absent)
- Returns 1 when its selector matches nothing at all. The only way to notice a job vanishing entirely, which up == 0 cannot see. Lesson 43.15
- aggregation operator (aggregation operators)
- sum, avg, min, max, count, topk and friends: combine many series into fewer, optionally keeping some labels with by or dropping some with without. Lesson 43.8
- backfill
- Filling in data for the past. Recording rules do not: a recorded series starts at its first evaluation after loading. Lesson 43.27
- bad_data
- The Prometheus error for a query that does not parse (HTTP 400). It gives line:column of where the parser gave up. Lesson 43.8
- binary operator (binary operators)
- An operator between two things: + - * / and comparisons like > or ==. Between two vectors, series are matched by their labels. Lesson 43.15
- bool modifier (== bool)
- Without bool a comparison filters series out; with bool (up == bool 0) every series stays and the value becomes 1 or 0. Lesson 43.15
- CMDB
- Configuration management database: a company's inventory of servers and services, often the source a script turns into a file_sd target list.
- context deadline exceeded
- Go's wording for a timeout: the scrape took longer than scrape_timeout, usually because packets are dropped or the target is overloaded. Lesson 43.2
- context propagation (propagates context)
- Passing the trace id (and the calling span's id) from one service to the next in an HTTP header, so both services' spans join the same trace. If one hop drops the header, the trace splits into two unrelated halves. Lesson 43.1
- counter reset (counter resets)
- A counter going back to 0 because the process restarted. rate() and increase() detect the drop and add the old value back. Lesson 43.2
- dashboard panel (a panel)
- One graph or number on a dashboard, drawn from one query over the chosen time range.
- deriv() (deriv)
- The slope of a gauge per second over a window. Free space divided by minus the slope is time until full. Lesson 43.8
- discovered labels (discoveredLabels, __meta_)
- The labels a target starts with, before relabelling: __address__, __scheme__, __metrics_path__ and __meta_* facts from the discovery source. Lesson 43.4
- distributed trace (traces, a trace)
- The record of one request's path through every service it touched, as a tree of timed steps (spans). Lesson 43.1
- droppedTargets
- The part of /api/v1/targets listing targets a keep or drop relabel rule removed. The first place to look when a target "does not show up". Lesson 43.4
- evaluation_interval (evaluation interval)
- How often Prometheus runs its rules: 15 seconds on this box. Lesson 43.2
- exposition format (exposition text)
- The plain-text format a target serves: # HELP and # TYPE comment lines, then one line per series: name, labels in braces, value. Lesson 43.2
- extrapolation (extrapolate)
- Stretching a result to the edges of the window because the first and last samples are never exactly there. Why increase() returns fractions. Lesson 43.8
- file_sd_configs (file_sd)
- Service discovery from a JSON or YAML file that another tool writes. Prometheus watches the file: edits apply within seconds, no reload. Lesson 43.4
- free-text log (free text)
- A log line written as a sentence for humans. Searching it needs a regex per question, and breaks when the wording changes. Lesson 43.1
- group_left (group_right)
- Allows many series on the left to match one on the right, and copies named labels from the right onto the result. How you join an info metric. Lesson 43.15
- hashmod
- A relabel action that hashes a label into a number, used to split targets between several Prometheus servers. Lesson 43.4
- head block
- The newest two hours of samples, kept in Prometheus's memory before being written to disk as a block. Its series count drives RAM use. Lesson 43.2
- histogram_quantile() (histogram_quantile)
- Estimates a percentile from histogram buckets: histogram_quantile(0.99, sum by (le) (rate(x_bucket[5m]))). Keep le in the aggregation. Lesson 43.15
- honor_labels (exported_job)
- If true, a target's own job/instance labels win over Prometheus's. By default Prometheus wins and renames the target's copy to exported_job. Lesson 43.4
- increase()
- How much a counter grew over a window: increase(x[1h]) = rate(x[1h]) * 3600. An estimate, rarely a whole number. Lesson 43.8
- info metric (info metrics)
- A series whose value is always 1 and whose labels carry facts (version, hostname, owner), like node_uname_info. Multiply by it to add those labels. Lesson 43.15
- instant vector
- A query result with one sample per series, the value "now". What you graph and do arithmetic on. Lesson 43.8
- irate()
- Per-second rate from only the last two samples in the window. Reacts fast and jumps around: fine for live graphs, bad for alerts. Lesson 43.8
- label matcher (label matchers)
- One condition on a label inside braces: = exact, != not equal, =~ regex, !~ negative regex. Several in one {} must all be true. Lesson 43.8
- label_replace() (label_replace)
- Adds or rewrites a label inside a query from another label with an anchored regex: label_replace(v, "host", "$1", "instance", "(.+):\\d+"). Lesson 43.15
- labelmap (labeldrop, labelkeep)
- Relabel actions that work on label NAMES: labelmap renames matching labels, labeldrop deletes them, labelkeep deletes all others. Lesson 43.4
- lastError
- The field in /api/v1/targets that says why a target's last scrape failed: connection refused, 404, deadline exceeded, no such host. Lesson 43.2
- le
- "Less than or equal": the label naming a histogram bucket's upper bound. le="+Inf" counts every observation. Lesson 43.2
- level:metric:operations
- The naming convention for recorded series: aggregation level, source metric, what was done, e.g. job_uri:http_server_requests_errors:ratio_rate5m. Lesson 43.27
- lifecycle API (/-/reload, --web.enable-lifecycle)
- Prometheus's HTTP endpoints for reload and shutdown. Off unless started with --web.enable-lifecycle, as on Ubuntu; reload with systemctl instead. Lesson 43.2
- lookback delta
- How old a series' latest sample may be and still count as "now" in a query: 5 minutes by default. Lesson 43.2
- metric_relabel_configs
- Relabel rules applied to every scraped series before it is stored. A guard to drop series an app should not expose. Lesson 43.4
- metrics_path
- The URL path Prometheus scrapes on a target, /metrics unless set. Spring Boot needs /actuator/prometheus. Lesson 43.2
- NaN
- "Not a number": the result of 0/0, e.g. a ratio or percentile over a series that had no requests in the window. Lesson 43.15
- node exporter (prometheus-node-exporter, node_exporter)
- The standard exporter for a Linux machine: it reads /proc and /sys and publishes CPU, memory, disk and network metrics on port 9100. Lesson 43.1
- observability
- Being able to answer a new question about a running system from the outside, using the data it already produces, without shipping new code to ask it. Lesson 43.1
- offset modifier (offset 1h, offset 1d)
- A modifier after a selector that reads it as it was some time ago: rate(x[5m] offset 1h) is the rate an hour ago. Lesson 43.8
- on / ignoring
- Say which labels to match two vectors on: on (instance) matches only by instance; ignoring (x) matches on everything except x. Lesson 43.15
- predict_linear() (predict_linear)
- Draws a straight line through a gauge's recent samples and says where it will be later: predict_linear(free[1h], 4*3600) < 0 means full within 4 hours. Lesson 43.15
- Prometheus Operator
- A program running in a Kubernetes cluster that installs Prometheus and generates its scrape config from ServiceMonitor and PodMonitor objects. Lesson 43.4
- prometheus_config_last_reload_successful (last_reload_successful)
- 1 if the last config reload was accepted, 0 if Prometheus rejected it and kept the old config. Worth an alert of its own. Lesson 43.2
- prometheus_tsdb_head_series (head series, active series)
- The number of series in the head block, reported by Prometheus about itself. The number to watch when Prometheus memory grows. Lesson 43.2
- prometheus.yml (scrape config, scrape_configs)
- Prometheus's main config file, /etc/prometheus/prometheus.yml: global intervals, where alerts go, rule files, and the scrape_configs list of jobs. Lesson 43.2
- PromQL selector (vector selector)
- A metric name plus label matchers in braces: http_requests_total{job="orders"}. It picks which series a query reads. Lesson 43.8
- promtool
- The command-line tool that ships with Prometheus: check config and check rules validate files, query instant runs a PromQL query. Lesson 43.2
- promtool check config
- Parses prometheus.yml and every rule file it references and prints SUCCESS or the error. Run it before every reload. Lesson 43.2
- promtool check rules
- Parses a rule file and every query in it, printing SUCCESS or the file position and parser error. Lesson 43.27
- promtool query instant
- Runs one PromQL query against a server "now" and prints one line per series: promtool query instant http://localhost:9090 'up'. Lesson 43.2
- Pushgateway
- A small Prometheus helper that short-lived batch jobs push their metrics to, so Prometheus can scrape them after the job has ended. The only place push belongs. Lesson 43.2
- query_range (/api/v1/query_range)
- The Prometheus API that evaluates a query at every step between a start and end time, returning the points a graph draws.
- range vector (range selector)
- Every sample per series inside a window ending now, written selector[5m]. Only functions like rate() accept it. Lesson 43.8
- rate window (4x scrape interval)
- The [5m] in rate(x[5m]). It must hold at least two samples; make it at least four times the scrape interval. Lesson 43.8
- rate()
- The per-second average increase of a counter over a window: rate(x[5m]). Handles restarts. Needs at least two samples in the window. Lesson 43.8
- RE2
- The regex flavour Prometheus uses (Go's), close to ERE. In PromQL and relabelling it is fully anchored: "5" matches only "5". Lesson 43.8
- recording rule (recording rules, recorded series)
- A rule that stores its result as a new series with a name you choose, so dashboards and alerts read one cheap series instead of recomputing. Lesson 43.15
- relabel_configs
- Relabel rules on a job, applied to each target's labels once, before scraping: which targets to keep and what their labels are. Lesson 43.4
- relabelling (relabeling, relabel, relabel rule)
- Prometheus rewriting labels with rules you give it: keep or drop targets or series, rename, copy or rewrite labels with a regex. Lesson 43.4
- resets()
- Counts how many times a counter dropped (the process restarted) inside a window. Lesson 43.8
- rule file (rule files, rule_files)
- A PromQL query Prometheus runs by itself every evaluation_interval, from a rule file listed under rule_files. Lesson 43.2
- rule group (groups:)
- A named list of rules in a rule file. Rules in a group run in order; groups run in parallel. Lesson 43.27
- sample (samples)
- One (timestamp, number) point of a time series, stored at each scrape. Lesson 43.1
- sample_limit
- A per-job cap on series per scrape: a target that suddenly exposes too many has its scrape failed (up = 0) instead of flooding the TSDB.
- scalar
- In PromQL, a bare number with no labels, like 3600. Lesson 43.8
- scrape (scraped, scrapes, scraping)
- One HTTP GET Prometheus makes to a target's metrics URL, storing every line that comes back as a sample. Lesson 43.2
- scrape interval (scrape_interval)
- How often Prometheus scrapes each target. 15 seconds on this box; set globally and overridable per job. Lesson 43.2
- scrape job (job_name, job label)
- A named group of targets that do the same thing (both copies of orders). Its job_name becomes the job label on every series. Lesson 43.2
- scrape target (scrape targets)
- A URL Prometheus scrapes, written host:port in the config. Its address becomes the instance label. Lesson 43.2
- scrape_timeout
- How long one scrape may take before it counts as failed ("context deadline exceeded"). Lesson 43.2
- service discovery (SD)
- Prometheus getting its list of targets from somewhere else (the Kubernetes API, Azure, a file) instead of a fixed list, so new services are scraped automatically. Lesson 43.4
- ServiceMonitor (PodMonitor)
- A Kubernetes object (from the Prometheus Operator) saying "scrape the pods behind Services with these labels, on this port and path". PodMonitor selects pods directly. Lesson 43.4
- source_labels (target_label)
- In a relabel rule: the labels whose values are joined with ; and matched by the regex; target_label is where replacement (with $1 etc.) is written. Lesson 43.4
- span (spans)
- One timed step inside a trace: a name, start time, duration, status and a few attributes. A span can have child spans. Lesson 43.1
- span id (spanId, span ID)
- The id of one span. A child span records its parent's span id, which is how the tree is built. Lesson 43.1
- staleness marker (staleness markers)
- A special sample Prometheus writes when a series ends (failed scrape, removed target), so the series disappears from queries at once. Lesson 43.2
- static_configs
- A fixed list of targets written in prometheus.yml, with optional labels to add to all of them. Lesson 43.2
- subquery (subqueries)
- (expr)[3h:1m]: evaluate expr once a minute over the last 3 hours, giving a range vector for max_over_time and friends. Expensive. Lesson 43.8
- sum by (avg by, count by)
- Aggregate keeping only the listed labels: sum by (uri) (...) gives one series per uri.
- sum without
- Aggregate removing only the listed labels and keeping every other one: sum without (instance) (...). Lesson 43.8
- summary metric (Summaries (quantiles)
- A metric type whose percentiles are computed inside the app ({quantile="0.99"}). Cheap to read, but percentiles from several copies cannot be combined. Lesson 43.2
- telemetry
- The data a program reports about itself while it runs: metrics, logs and traces. Lesson 43.1
- three pillars (three signals, pillars of observability)
- Metrics, logs and traces: the three kinds of telemetry. Metrics say THAT something is wrong, logs WHAT happened, traces WHERE the time went. Lesson 43.1
- time()
- PromQL function returning the evaluation time as a Unix timestamp. time() - last_success_timestamp is "how long ago". Lesson 43.15
- topk (bottomk)
- topk(3, x) keeps the 3 series with the biggest values, labels intact; bottomk the smallest.
- TSDB (time-series database)
- Time-series database: storage built for series of timestamped numbers. Prometheus has its own, on local disk under /var/lib/prometheus/metrics2. Lesson 43.1
- Unix timestamp (epoch seconds, Unix time)
- A moment written as seconds since 1 January 1970 UTC, like 1790107223. date -u -d @1790107223 turns it into a date. Lesson 43.8
- up metric (up == 0, up{job)
- The series Prometheus writes itself for every target: 1 if the last scrape worked, 0 if it failed. Lesson 43.2
- vector matching
- How Prometheus pairs series on the two sides of an operator: by default only series with exactly the same labels. Lesson 43.15