OnCallReadyStart learning

Observability II: Alerting, Alertmanager & SLO alerts: glossary

68 terms, in plain words, with where each one is taught.

/api/v1/alerts
Prometheus HTTP API endpoints: /api/v1/alerts lists every pending and firing alert, /api/v1/rules every loaded rule with its state and health. Both return JSON for jq. Lesson 44.1
$labels ({{ $labels }})
In an alert template, the alert's labels: {{ $labels.job }} is its job label. Lesson 44.1
$value ({{ $value }})
In an alert template, the value of the alert's result series. For a filter like x > 0.05 it is the value of x. Never put it in a label: the label would change every evaluation and make a new alert each time. Lesson 44.1
absent_over_time
A PromQL function that returns 1 if a series had no samples at all in the given window, e.g. absent_over_time(x[10m]). Lesson 44.1
activeAt
The moment an alert's condition first became true: the start of its for: clock. Shown in /api/v1/alerts. Lesson 44.1
alert instance (alert instances)
One series in the result of an alerting rule: one "thing that is wrong right now", told apart from the others by its labels. up == 0 with two dead targets gives two instances. Lesson 44.1
alert template (templated)
Text with placeholders in {{ }} that Prometheus fills in when the alert is evaluated, such as {{ $labels.instance }}. The same Go template syntax as Helm. Lesson 44.1
alert_rule_test (exp_alerts, exp_labels, exp_annotations)
The part of a promtool test that lists which alerts must be firing at an eval_time, with every label (except alertname) and every annotation. exp_alerts: [] means none may fire. Lesson 44.4
alerting rule (alerting rules, alert rules)
A PromQL expression with a name, kept in a rule file. Prometheus runs it again and again; every series it returns becomes an alert. It sits in the same groups: / rules: structure as recording rules, under alert: instead of record:. Lesson 44.1
Alertmanager
The program that receives firing alerts from Prometheus and turns them into notifications: it routes, groups, throttles, inhibits and silences them. On this box: prometheus-alertmanager.service on port 9093. Lesson 44.1
AlertmanagerConfig
A namespaced custom resource with routes and receivers that the operator merges into Alertmanager's routing tree, so each team can own its own routing. Lesson 44.24
alertname
The label Prometheus puts on every alert, holding the rule's name. alertname="TargetDown" selects that alert everywhere: in ALERTS, in amtool, in silences. Lesson 44.1
ALERTS series (ALERTS_FOR_STATE)
The time series Prometheus writes for every pending or firing alert, with the alert's labels plus alertstate. Query it afterwards to find out whether an alert fired. max_over_time(ALERTS{alertname="X"}[3h]) answers "was X pending or firing at any time in the last three hours?". ALERTS_FOR_STATE keeps the for: clocks across restarts. Lesson 44.1
alertstate
The label on the ALERTS series saying whether the alert was pending or firing. Lesson 44.1
amtool
Alertmanager's command-line tool: check a config, test routes, and query alerts and silences on a running Alertmanager. amtool check-config, amtool config routes show|test --config.file=..., amtool --alertmanager.url=... alert query | silence add | silence query | silence expire.
amtool check-config
Validates an alertmanager.yml without loading it; catches a route pointing at a receiver that does not exist. Exit code 1 on failure, so it works as a CI gate. Lesson 44.7
amtool config routes test (routes test)
Prints which receiver an alert with the given labels would reach, from a config file, without touching the running Alertmanager. The safe way to check a routing change. Lesson 44.7
anchored regex
A regex that must match the whole value, not just part of it. PromQL =~ matchers are anchored: status=~"5xx" matches only the text 5xx, never 500.
annotation (annotations)
Free text attached to an alert (a summary, a description, a runbook link). Unlike a label it does not identify the alert; it is the message a human reads. Templated: {{ $labels.job }} and {{ $value }} are filled in at evaluation time. Lesson 44.1
cAdvisor
"Container advisor", built into the kubelet: it reads each container's cgroup and exports what the container uses (CPU, memory, disk, network) as container_* metrics. Lesson 44.24
chat tool (Slack, Microsoft Teams)
Team messaging such as Slack or Teams. Warnings and tickets usually go to a team channel there instead of paging. Lesson 44.7
container_memory_working_set_bytes (container_memory_usage_bytes)
cAdvisor's memory metric that leaves out cache the kernel can drop. Compare it with the memory limit to predict OOM kills; the plain usage metric includes cache and looks alarming for nothing. Lesson 44.24
continue: true (continue)
On an Alertmanager route: after this route matches, keep trying its siblings, so the alert can reach several receivers. Lesson 44.7
deduplication (deduplicate, dedup)
Sending one notification when several copies of the same alert arrive, e.g. from two Alertmanagers in a cluster. Lesson 44.7
error budget remaining (budget remaining)
The fraction of the period's error budget not yet spent: 1 is untouched, 0 is spent, negative means the SLO is broken. A dashboard number, not an alert. Lesson 44.14
eval_time
In a promtool test, the moment on the simulated clock at which an expectation is checked. Lesson 44.4
firing (fires, fired)
The state of an alert whose condition has been true for at least its for:. Only firing alerts are sent to Alertmanager. Lesson 44.1
for: (for clause, for: clause)
How long an alerting rule's condition must stay true, at every evaluation, before the alert fires. Until then it is pending. One false evaluation starts the clock again. Lesson 44.1
gossip (gossip protocol)
How clustered Alertmanagers talk to each other (port 9094): they share silences and which notifications were already sent. Lesson 44.7
group_by
Which labels define a group in Alertmanager: alerts with the same values of these labels go out in one notification. ['...'] turns grouping off. Lesson 44.7
group_interval
The minimum time between notifications about new alerts joining a group that was already notified (5 m here). Lesson 44.7
group_wait
How long Alertmanager waits after the first alert of a new group before notifying, so alerts arriving together share one message (30 s here).
high availability (HA)
Running several copies of a component so that losing one does not stop the service. For Alertmanager: two or three instances, every Prometheus sending to all of them, no load balancer in between. Lesson 44.7
humanizePercentage (humanize, humanizeDuration)
Template functions that format a number for people: humanizePercentage turns 0.0642 into 6.42%, humanize writes 1234567 as 1.235M, humanizeDuration writes seconds as "2m 3s". Lesson 44.1
inactive
The state of an alert whose expression returns nothing: nothing is wrong - or the expression can never match. Prometheus cannot tell the two apart. Lesson 44.1
inhibit rule (inhibit_rules, source_matchers, target_matchers)
An Alertmanager rule that mutes target alerts while a source alert fires, when the labels listed in equal have the same values on both. Without equal, one source alert anywhere mutes every target alert everywhere. Lesson 44.7
keep_firing_for
A rule option (Prometheus 2.42+) that keeps an alert firing for a while after its condition clears, so a noisy condition does not send "resolved, firing, resolved" storms. Lesson 44.1
kube-prometheus-stack
A Helm chart that installs the Prometheus Operator, Prometheus, Alertmanager, the node exporter, kube-state-metrics, a dashboard tool and a few hundred default rules into a cluster. Lesson 44.24
kube-state-metrics
An exporter that watches the Kubernetes API and exports the state of objects - replicas wanted vs available, restarts, phases, requests and limits - as kube_* metrics. Lesson 44.24
load test (load-test)
Sending a lot of requests on purpose to see how a service behaves under pressure - here, to push orders into failure and watch the page arrive.
long window (short window)
The two windows of one burn-rate condition: the long one (1h) proves enough budget was burnt to matter, the short one (5m, 1/12 of it) proves it is still happening, so the alert resets soon after recovery. Lesson 44.14
minimum-traffic clause (minimum traffic)
An extra and sum(rate(...)) > N on a burn-rate alert so that, on a quiet service, one failed request cannot page. Lesson 44.14
notification (notifications, notify)
One message Alertmanager sends to a receiver: a page, a chat message, an email. One notification can carry many alerts.
off-by-one
An error of exactly one step, such as expecting an alert at minute 4 when it fires at minute 5. The commonest reason a new rule test fails. Lesson 44.4
pager-webhook
The lab's stand-in for a paging service and a team chat (simulator). It logs every notification it gets; read it with journalctl -u pager-webhook.
paging service (Opsgenie)
A company that phones the on-call person, keeps the rota and escalates when nobody answers. PagerDuty and Opsgenie are the common ones; Alertmanager sends to them as a receiver. Lesson 44.7
positive case (negative case, negative test)
A positive case checks that an alert fires when it should; a negative case checks it does not fire when it should not, e.g. one minute earlier. A test with only positive cases passes for an alert that always fires. Lesson 44.4
PrometheusRule (PrometheusRules)
A Kubernetes custom resource that holds alerting and recording rules; its spec is exactly the rule-file format. The Prometheus Operator loads it into Prometheus. Lesson 44.4
promql_expr_test (exp_samples)
The part of a promtool test that checks what an expression - often a recording rule - returns at an eval_time. Lesson 44.4
promtool test rules (rule test, rule tests, unit-test the rules)
Runs your real rule files against synthetic series on a simulated clock and checks which alerts fire when. Needs no running Prometheus, so it fits in CI. Lesson 44.4
receiver (receivers)
A named destination in alertmanager.yml, such as the on-call pager or a team chat, and how to reach it (webhook, email, a paging service). Lesson 44.7
repeat_interval
How often Alertmanager re-sends a group that is still firing and unchanged (4 h here). Lesson 44.7
resolve_timeout
How long Alertmanager keeps an alert it has stopped hearing about before treating it as resolved. Lesson 44.7
routing tree (route tree, routes:, child route)
The route: block of alertmanager.yml: a root route with child routes, each with matchers and a receiver. Children are tried in order and the first match wins. Settings a child does not set are inherited from its parent. An alert matching no child stays at the parent's receiver. Lesson 44.7
rule evaluation (evaluation, evaluations, evaluated)
One run of a rule by Prometheus. It happens every evaluation_interval (15 s on this box). Lesson 44.1
rule health (health ok)
In /api/v1/rules, whether a rule's last evaluation worked (ok) or failed (err). "ok" says nothing about whether the rule can ever match. Lesson 44.1
ruleSelector (ruleNamespaceSelector)
The label selector on the Prometheus custom resource that decides which PrometheusRule objects the operator loads. A rule without the matching label (often release: kube-prometheus-stack) is silently ignored. Lesson 44.24
runbook_url
The conventional annotation holding the link to the runbook for an alert. Every page should have one. Lesson 44.1
silence (silences, silenced)
A mute that a person creates at run time, for planned work: every alert matching its matchers is suppressed until it expires. Always narrow, with a duration and a comment. Lesson 44.7
simulated clock
In a rule test, time jumps from one interval to the next instead of passing for real, so an hour of alert behaviour is checked in a fraction of a second. Lesson 44.4
SRE Workbook (the Workbook, Google SRE Workbook)
Google's free book of SRE practice. Its alerting chapter is the source of the 14.4 / 6 / 3 / 1 burn-rate thresholds and the long/short window pairs. Lesson 44.14
suppressed
Alertmanager's state for an alert it received but is not notifying about, because a silence or an inhibit rule covers it. amtool alert query -s shows them. Lesson 44.7
symptom-based alert (symptom alert, symptom alerts)
An alert on what users experience (errors, latency, unavailability). It fires for every cause, including ones nobody predicted, so it is what should page. Lesson 44.1
synthetic series (input_series, synthetic data)
Fake time series written by hand in a promtool test, in place of real scrapes. Lesson 44.4
throttled periods (container_cpu_cfs_throttled_periods_total, container_cpu_cfs_periods_total)
cAdvisor counters of the CPU time slices (100 ms each) in which a container hit its CPU limit and was paused, and of all slices. Their ratio is how throttled the container is. Lesson 44.24
unit test (unit tests)
A small automatic check that feeds known input to one piece of a system and compares the result with what is expected. Lesson 44.4
values notation (series notation, expanding notation)
promtool's shorthand for sample values: 0+10x5 is 0 10 20 30 40 50, 1x4 repeats 1, _ is a missing sample, stale a staleness marker. Lesson 44.4
webhook (webhook_configs, webhooks)
An HTTP URL that a program POSTs events to, as JSON. Alertmanager's most generic receiver type; the lab's pager-webhook is one.