Observability II: Alerting, Alertmanager & SLO alerts: glossary
68 terms, in plain words, with where each one is taught.
- /api/v1/alerts
- Prometheus HTTP API endpoints: /api/v1/alerts lists every pending and firing alert, /api/v1/rules every loaded rule with its state and health. Both return JSON for jq. Lesson 44.1
- $labels ({{ $labels }})
- In an alert template, the alert's labels:
{{ $labels.job }}is its job label. Lesson 44.1 - $value ({{ $value }})
- In an alert template, the value of the alert's result series. For a filter like
x > 0.05it is the value ofx. Never put it in a label: the label would change every evaluation and make a new alert each time. Lesson 44.1 - absent_over_time
- A PromQL function that returns 1 if a series had no samples at all in the given window, e.g.
absent_over_time(x[10m]). Lesson 44.1 - activeAt
- The moment an alert's condition first became true: the start of its
for:clock. Shown in/api/v1/alerts. Lesson 44.1 - alert instance (alert instances)
- One series in the result of an alerting rule: one "thing that is wrong right now", told apart from the others by its labels.
up == 0with two dead targets gives two instances. Lesson 44.1 - alert template (templated)
- Text with placeholders in
{{ }}that Prometheus fills in when the alert is evaluated, such as{{ $labels.instance }}. The same Go template syntax as Helm. Lesson 44.1 - alert_rule_test (exp_alerts, exp_labels, exp_annotations)
- The part of a promtool test that lists which alerts must be firing at an eval_time, with every label (except alertname) and every annotation.
exp_alerts: []means none may fire. Lesson 44.4 - alerting rule (alerting rules, alert rules)
- A PromQL expression with a name, kept in a rule file. Prometheus runs it again and again; every series it returns becomes an alert. It sits in the same
groups:/rules:structure as recording rules, underalert:instead ofrecord:. Lesson 44.1 - Alertmanager
- The program that receives firing alerts from Prometheus and turns them into notifications: it routes, groups, throttles, inhibits and silences them. On this box: prometheus-alertmanager.service on port 9093. Lesson 44.1
- AlertmanagerConfig
- A namespaced custom resource with routes and receivers that the operator merges into Alertmanager's routing tree, so each team can own its own routing. Lesson 44.24
- alertname
- The label Prometheus puts on every alert, holding the rule's name.
alertname="TargetDown"selects that alert everywhere: in ALERTS, in amtool, in silences. Lesson 44.1 - ALERTS series (ALERTS_FOR_STATE)
- The time series Prometheus writes for every pending or firing alert, with the alert's labels plus
alertstate. Query it afterwards to find out whether an alert fired.max_over_time(ALERTS{alertname="X"}[3h])answers "was X pending or firing at any time in the last three hours?". ALERTS_FOR_STATE keeps thefor:clocks across restarts. Lesson 44.1 - alertstate
- The label on the ALERTS series saying whether the alert was
pendingorfiring. Lesson 44.1 - amtool
- Alertmanager's command-line tool: check a config, test routes, and query alerts and silences on a running Alertmanager.
amtool check-config,amtool config routes show|test --config.file=...,amtool --alertmanager.url=... alert query | silence add | silence query | silence expire. - amtool check-config
- Validates an alertmanager.yml without loading it; catches a route pointing at a receiver that does not exist. Exit code 1 on failure, so it works as a CI gate. Lesson 44.7
- amtool config routes test (routes test)
- Prints which receiver an alert with the given labels would reach, from a config file, without touching the running Alertmanager. The safe way to check a routing change. Lesson 44.7
- anchored regex
- A regex that must match the whole value, not just part of it. PromQL
=~matchers are anchored:status=~"5xx"matches only the text 5xx, never 500. - annotation (annotations)
- Free text attached to an alert (a summary, a description, a runbook link). Unlike a label it does not identify the alert; it is the message a human reads. Templated:
{{ $labels.job }}and{{ $value }}are filled in at evaluation time. Lesson 44.1 - cAdvisor
- "Container advisor", built into the kubelet: it reads each container's cgroup and exports what the container uses (CPU, memory, disk, network) as
container_*metrics. Lesson 44.24 - chat tool (Slack, Microsoft Teams)
- Team messaging such as Slack or Teams. Warnings and tickets usually go to a team channel there instead of paging. Lesson 44.7
- container_memory_working_set_bytes (container_memory_usage_bytes)
- cAdvisor's memory metric that leaves out cache the kernel can drop. Compare it with the memory limit to predict OOM kills; the plain usage metric includes cache and looks alarming for nothing. Lesson 44.24
- continue: true (continue)
- On an Alertmanager route: after this route matches, keep trying its siblings, so the alert can reach several receivers. Lesson 44.7
- deduplication (deduplicate, dedup)
- Sending one notification when several copies of the same alert arrive, e.g. from two Alertmanagers in a cluster. Lesson 44.7
- error budget remaining (budget remaining)
- The fraction of the period's error budget not yet spent: 1 is untouched, 0 is spent, negative means the SLO is broken. A dashboard number, not an alert. Lesson 44.14
- eval_time
- In a promtool test, the moment on the simulated clock at which an expectation is checked. Lesson 44.4
- firing (fires, fired)
- The state of an alert whose condition has been true for at least its
for:. Only firing alerts are sent to Alertmanager. Lesson 44.1 - for: (for clause, for: clause)
- How long an alerting rule's condition must stay true, at every evaluation, before the alert fires. Until then it is pending. One false evaluation starts the clock again. Lesson 44.1
- gossip (gossip protocol)
- How clustered Alertmanagers talk to each other (port 9094): they share silences and which notifications were already sent. Lesson 44.7
- group_by
- Which labels define a group in Alertmanager: alerts with the same values of these labels go out in one notification.
['...']turns grouping off. Lesson 44.7 - group_interval
- The minimum time between notifications about new alerts joining a group that was already notified (5 m here). Lesson 44.7
- group_wait
- How long Alertmanager waits after the first alert of a new group before notifying, so alerts arriving together share one message (30 s here).
- high availability (HA)
- Running several copies of a component so that losing one does not stop the service. For Alertmanager: two or three instances, every Prometheus sending to all of them, no load balancer in between. Lesson 44.7
- humanizePercentage (humanize, humanizeDuration)
- Template functions that format a number for people: humanizePercentage turns 0.0642 into 6.42%, humanize writes 1234567 as 1.235M, humanizeDuration writes seconds as "2m 3s". Lesson 44.1
- inactive
- The state of an alert whose expression returns nothing: nothing is wrong - or the expression can never match. Prometheus cannot tell the two apart. Lesson 44.1
- inhibit rule (inhibit_rules, source_matchers, target_matchers)
- An Alertmanager rule that mutes target alerts while a source alert fires, when the labels listed in
equalhave the same values on both. Withoutequal, one source alert anywhere mutes every target alert everywhere. Lesson 44.7 - keep_firing_for
- A rule option (Prometheus 2.42+) that keeps an alert firing for a while after its condition clears, so a noisy condition does not send "resolved, firing, resolved" storms. Lesson 44.1
- kube-prometheus-stack
- A Helm chart that installs the Prometheus Operator, Prometheus, Alertmanager, the node exporter, kube-state-metrics, a dashboard tool and a few hundred default rules into a cluster. Lesson 44.24
- kube-state-metrics
- An exporter that watches the Kubernetes API and exports the state of objects - replicas wanted vs available, restarts, phases, requests and limits - as
kube_*metrics. Lesson 44.24 - load test (load-test)
- Sending a lot of requests on purpose to see how a service behaves under pressure - here, to push orders into failure and watch the page arrive.
- long window (short window)
- The two windows of one burn-rate condition: the long one (1h) proves enough budget was burnt to matter, the short one (5m, 1/12 of it) proves it is still happening, so the alert resets soon after recovery. Lesson 44.14
- minimum-traffic clause (minimum traffic)
- An extra
and sum(rate(...)) > Non a burn-rate alert so that, on a quiet service, one failed request cannot page. Lesson 44.14 - notification (notifications, notify)
- One message Alertmanager sends to a receiver: a page, a chat message, an email. One notification can carry many alerts.
- off-by-one
- An error of exactly one step, such as expecting an alert at minute 4 when it fires at minute 5. The commonest reason a new rule test fails. Lesson 44.4
- pager-webhook
- The lab's stand-in for a paging service and a team chat (simulator). It logs every notification it gets; read it with
journalctl -u pager-webhook. - paging service (Opsgenie)
- A company that phones the on-call person, keeps the rota and escalates when nobody answers. PagerDuty and Opsgenie are the common ones; Alertmanager sends to them as a receiver. Lesson 44.7
- positive case (negative case, negative test)
- A positive case checks that an alert fires when it should; a negative case checks it does not fire when it should not, e.g. one minute earlier. A test with only positive cases passes for an alert that always fires. Lesson 44.4
- PrometheusRule (PrometheusRules)
- A Kubernetes custom resource that holds alerting and recording rules; its
specis exactly the rule-file format. The Prometheus Operator loads it into Prometheus. Lesson 44.4 - promql_expr_test (exp_samples)
- The part of a promtool test that checks what an expression - often a recording rule - returns at an eval_time. Lesson 44.4
- promtool test rules (rule test, rule tests, unit-test the rules)
- Runs your real rule files against synthetic series on a simulated clock and checks which alerts fire when. Needs no running Prometheus, so it fits in CI. Lesson 44.4
- receiver (receivers)
- A named destination in alertmanager.yml, such as the on-call pager or a team chat, and how to reach it (webhook, email, a paging service). Lesson 44.7
- repeat_interval
- How often Alertmanager re-sends a group that is still firing and unchanged (4 h here). Lesson 44.7
- resolve_timeout
- How long Alertmanager keeps an alert it has stopped hearing about before treating it as resolved. Lesson 44.7
- routing tree (route tree, routes:, child route)
- The
route:block of alertmanager.yml: a root route with child routes, each with matchers and a receiver. Children are tried in order and the first match wins. Settings a child does not set are inherited from its parent. An alert matching no child stays at the parent's receiver. Lesson 44.7 - rule evaluation (evaluation, evaluations, evaluated)
- One run of a rule by Prometheus. It happens every
evaluation_interval(15 s on this box). Lesson 44.1 - rule health (health ok)
- In /api/v1/rules, whether a rule's last evaluation worked (
ok) or failed (err). "ok" says nothing about whether the rule can ever match. Lesson 44.1 - ruleSelector (ruleNamespaceSelector)
- The label selector on the Prometheus custom resource that decides which PrometheusRule objects the operator loads. A rule without the matching label (often
release: kube-prometheus-stack) is silently ignored. Lesson 44.24 - runbook_url
- The conventional annotation holding the link to the runbook for an alert. Every page should have one. Lesson 44.1
- silence (silences, silenced)
- A mute that a person creates at run time, for planned work: every alert matching its matchers is suppressed until it expires. Always narrow, with a duration and a comment. Lesson 44.7
- simulated clock
- In a rule test, time jumps from one interval to the next instead of passing for real, so an hour of alert behaviour is checked in a fraction of a second. Lesson 44.4
- SRE Workbook (the Workbook, Google SRE Workbook)
- Google's free book of SRE practice. Its alerting chapter is the source of the 14.4 / 6 / 3 / 1 burn-rate thresholds and the long/short window pairs. Lesson 44.14
- suppressed
- Alertmanager's state for an alert it received but is not notifying about, because a silence or an inhibit rule covers it.
amtool alert query -sshows them. Lesson 44.7 - symptom-based alert (symptom alert, symptom alerts)
- An alert on what users experience (errors, latency, unavailability). It fires for every cause, including ones nobody predicted, so it is what should page. Lesson 44.1
- synthetic series (input_series, synthetic data)
- Fake time series written by hand in a promtool test, in place of real scrapes. Lesson 44.4
- throttled periods (container_cpu_cfs_throttled_periods_total, container_cpu_cfs_periods_total)
- cAdvisor counters of the CPU time slices (100 ms each) in which a container hit its CPU limit and was paused, and of all slices. Their ratio is how throttled the container is. Lesson 44.24
- unit test (unit tests)
- A small automatic check that feeds known input to one piece of a system and compares the result with what is expected. Lesson 44.4
- values notation (series notation, expanding notation)
- promtool's shorthand for sample values:
0+10x5is 0 10 20 30 40 50,1x4repeats 1,_is a missing sample,stalea staleness marker. Lesson 44.4 - webhook (webhook_configs, webhooks)
- An HTTP URL that a program POSTs events to, as JSON. Alertmanager's most generic receiver type; the lab's pager-webhook is one.