OnCallReadyStart learning

Observability III: Dashboards, Logs, Traces & Incidents: glossary

65 terms, in plain words, with where each one is taught.

__error__
A label LogQL sets on lines a parser could not read. | __error__="" keeps only the lines that parsed. Lesson 45.6
$__rate_interval (__rate_interval, ${__rate_interval})
A built-in Grafana variable that picks a safe rate() window: the larger of (panel step + scrape interval) and 4 scrape intervals. Use it in every rate on a dashboard; use fixed windows in rules. On a 3-hour graph with a 15 s scrape it becomes 3m15s, which grafana-preview prints in its header. Lesson 45.1
auto-instrumentation (auto-instrumented)
Getting spans without changing the application's code: an agent hooks into common libraries (web framework, JDBC, HTTP clients) and creates spans for them. Lesson 45.11
chunk (chunks)
A compressed block of one stream's log lines, the way Loki stores them. A query decides which chunks to open from the labels, then reads their lines. Lesson 45.6
Collector (OpenTelemetry Collector, OTel Collector)
The OpenTelemetry program that receives telemetry from apps, processes it (batching, sampling, removing attributes) and forwards it to a backend. Apps send to it, never straight to the backend. Lesson 45.11
Collector pipeline (receivers, processors and exporters)
How a Collector is configured: receivers (what it accepts, e.g. OTLP), processors (what it does in between, e.g. batch, tail_sampling) and exporters (where it sends, e.g. Tempo). Lesson 45.11
common labels
The labels every line in a logcli result shares, printed once at the top (to stderr) so each line shows only the labels that differ, often {}. Lesson 45.6
count_over_time (count_over_time())
A LogQL range function: how many lines each stream logged in the window. count_over_time({unit="orders.service"} |= "ERROR" [5m]). Lesson 45.6
critical path
The chain of spans that decides a request's total time. Speeding up a span off the critical path does not make the request faster. Lesson 45.11
dashboard JSON (dashboard model, JSON model)
The JSON document a Grafana dashboard really is: a title, a time range and a list of panels with their types, queries and units. It can be written by hand, kept in git and imported. Lesson 45.1
dashboard provisioning (dashboards as code, dashboard-as-code, provisioned)
Grafana loading dashboards (and data sources) from files at startup instead of from clicks in the UI, so the copy in git is the one that counts. Lesson 45.1
derived field (derived fields)
A Grafana setting on a Loki data source that turns a value in a log line, such as the trace id, into a clickable link to the trace. Lesson 45.6
Elasticsearch (ELK)
A search engine often used for logs. It indexes every word of every line, so any text search is fast, but it needs far more disk and memory than Loki. Lesson 45.6
exemplar (exemplars, exemplar-storage)
One trace id stored next to a histogram bucket: an example request that landed there. Grafana draws exemplars as dots on a latency graph; a click opens that trace. Lesson 45.11
Grafana
The web app that draws graphs and tables from monitoring databases such as Prometheus and Loki. It stores no data itself: every graph is a query it sends to a data source. The Dashboards console in this app stands in for it; grafana-preview is its terminal stand-in. Lesson 45.1
Grafana Alloy (Alloy, config.alloy)
Grafana's log shipper and telemetry collector, the replacement for Promtail. On this box it reads the systemd journal and pushes it to Loki; its config is /etc/alloy/config.alloy. Lesson 45.6
Grafana variable (dashboard variable)
A $name placeholder in a Grafana query that Grafana replaces with a value before it sends the query to the data source. Lesson 45.1
grafana-preview
(simulator) A stand-in for Grafana on this box: it reads a dashboard JSON file, runs every panel query against Prometheus and draws the results as sparklines. --range 6h changes the range. Lesson 45.1
head sampling (parentbased_traceidratio, head-based sampling)
Deciding at the first span whether to keep a trace (e.g. 10% of trace ids); later services follow that decision. Cheap and consistent, but blind to which requests turn out slow. Lesson 45.11
Jaeger
An older, widely used open-source trace database with its own web UI. Like Tempo it accepts OpenTelemetry data. Lesson 45.11
Java agent (-javaagent, opentelemetry-javaagent.jar)
A jar the JVM loads at startup with -javaagent:<jar>, which can change classes as they load. OpenTelemetry's Java agent uses this to auto-instrument Spring, JDBC and HTTP clients. Lesson 45.11
JDBC
Java Database Connectivity: the standard Java way to talk to a database. Every Java database driver (and connection pool) sits on it. Lesson 45.11
label filter (label filters, numeric label filter)
A LogQL stage after a parser that keeps lines by a parsed field: | level="ERROR", or numerically | duration_ms > 1000, | status >= 500. It compares the field, not the text. Lesson 45.6
label_values (label_values())
A Grafana function (not PromQL) that lists every value a label has, used to fill a template variable: label_values(up{job="orders"}, instance). Lesson 45.1
legendFormat
The field of a panel query that names each line in the legend. {{uri}} is replaced by the value of the uri label, so each endpoint gets a readable name. Lesson 45.1
line filter (line filters, |=, |~)
A LogQL stage that keeps or drops lines by their text: |= "ERROR" contains, != "health" does not contain, |~ and !~ match or do not match a regex. Filters run left to right; put the one that drops the most lines first. Lesson 45.6
line_format
A LogQL stage that rewrites each line from a template: | line_format "{{.traceId}} {{.message}}" prints only those two fields. Lesson 45.6
log metric query (metric query, metric queries)
A LogQL query that returns numbers instead of lines: a log query plus a window, wrapped in a range function such as count_over_time or rate, then aggregated with sum by like PromQL. Lesson 45.6
log parser (| json, | logfmt, parser)
A LogQL stage that reads each line and turns its fields into labels for the rest of the query only: | json for JSON lines, | logfmt for key=value lines. Nothing is indexed. Field names are made label-safe: @timestamp becomes _timestamp, http.uri becomes http_uri. Lesson 45.6
log shipper (log agent, log shippers)
A program on each machine that reads local logs (the journal, files, container logs) and sends them to the log database. Lesson 45.6
log stream (stream, streams, log streams)
In Loki, all the log lines that share exactly the same set of labels, such as {job="systemd-journal", unit="orders.service"}. Each new label combination is a new stream. Lesson 45.6
logcli (logcli query, logcli instant-query)
Loki's command-line client. logcli query '<LogQL>' prints matching lines, newest first (--limit, --since, -o raw); logcli instant-query runs a metric query and prints numbers as JSON. Lesson 45.6
logfmt
A log line format of key=value pairs separated by spaces, e.g. method=GET path=/api/cart status=502. Prometheus itself logs this way. Lesson 45.6
LogQL
Loki's query language, built to look like PromQL: a stream selector in {}, then a pipeline of filters and parsers, optionally wrapped in a range function to turn lines into numbers. Lesson 45.6
Loki
A log database from the makers of Grafana. It indexes only each stream's labels and keeps the text of the lines compressed, reading it when you query. Cheap to run; fast when you narrow by labels first. Here it runs as loki.service on port 3100. Lesson 45.6
Loki ruler (ruler)
The part of Loki that evaluates LogQL rules on a schedule, like Prometheus's rule engine, and sends alerts to the same Alertmanager. Lesson 45.6
N+1 query (N+1 queries)
One database query for a list, then one more query per item in it: 200 items, 201 queries. In a trace it looks like a comb of tiny database spans. Lesson 45.11
No data
What a Grafana panel shows when its query ran fine but matched nothing. It looks exactly like an idle service, which is why a wrong query can hide an outage. A query error is loud (the panel shows the error); an empty result is silent. count(<expr>) by hand proves a query returns something. Lesson 45.1
OpenMetrics
The standardised successor of Prometheus's text exposition format. Needed to expose exemplars. Lesson 45.11
OpenTelemetry (OTel)
The vendor-neutral standard, libraries and tools for producing traces, metrics and logs: an API and SDK per language, auto-instrumentation agents and the Collector. Instrument once, pick the backend later. Lesson 45.11
OTLP
The OpenTelemetry Protocol: the format apps use to send traces, metrics and logs, over gRPC on port 4317 or HTTP on port 4318. Lesson 45.11
panel (panels)
One box on a Grafana dashboard: a graph, a big number or a table, drawn from one or more queries. Lesson 45.1
panel step ($__interval, graph step)
The time between two points on a graph, picked by Grafana from the time range and the panel width: 3 hours over ~56 points is one point every ~3 minutes. $__interval holds it. Lesson 45.1
panel target (targets[].expr, panel query)
One query inside a Grafana panel. In the dashboard JSON it is an object in the panel's targets list, with the PromQL in expr. Not the same as a Prometheus scrape target (43.2): Grafana reuses the word. Lesson 45.1
panel type (timeseries, heatmap)
How a panel draws its data: timeseries is a line graph over time, stat one big number, table rows, heatmap a histogram over time as coloured cells. Lesson 45.1
panel unit (percentunit, reqps, fieldConfig.defaults.unit)
What a panel's numbers mean, set in fieldConfig.defaults.unit: s seconds, percentunit a 0-1 ratio shown as %, reqps requests per second, bytes. It decides how the axis is labelled. Lesson 45.1
parent span id (parent_span_id, root span)
In a span, the id of the span that called it. Following parent ids builds the tree of a trace; the root span, the first one, has none. Lesson 45.11
Promtail
The old log shipper for Loki. Deprecated in 2025 and out of support since early 2026; Grafana Alloy replaces it. You will still meet it in older setups. Lesson 45.6
QA (quality assurance)
Quality assurance: the people or team who test software before (and sometimes after) it ships.
resource attributes (service.name, otel.service.name)
Facts about the process that produced the telemetry, set once and attached to every span: service.name, service.version, deployment.environment. Lesson 45.11
sampling (sampled)
Keeping only some traces because keeping all of them costs too much. The kept ones are examples, not statistics. Lesson 45.11
semantic conventions
OpenTelemetry's agreed names for attributes (http.status_code, db.system, service.name), so every tool and backend understands them the same way. Lesson 45.11
span attributes (attributes)
Key/value details attached to a span: http.method, http.route, db.statement. Where the specifics of one request live. Lesson 45.11
span event (span events)
A timestamped moment inside a span, such as "exception thrown" or "retrying", without a duration of its own. Lesson 45.11
sparkline (sparklines)
A tiny graph drawn as one line of bars, with no axes: enough to see the shape of a series at a glance. Lesson 45.1
stat panel (stat panels)
A Grafana panel that shows one big number, such as the current burn rate. Right for "how bad is it right now". Lesson 45.1
stream selector (log stream selector)
The required first part of a LogQL query: label matchers in curly braces, like {unit="orders.service"}, that pick which streams to read. The cheap part of a query. Lesson 45.6
tail sampling (tail_sampling, tail-based sampling, decision_wait)
Deciding after a trace has finished, in the Collector: keep every error, everything slower than a threshold and a small share of the rest. Needs all spans of a trace at the same Collector. Lesson 45.11
template variable ($instance)
A drop-down at the top of a Grafana dashboard, filled by a query, whose choice is substituted into every panel. Lets one dashboard serve every instance. Match it with =~ (instance=~"$instance"): picking "All" or several values becomes a regex like a|b. Lesson 45.1
Tempo (Grafana Tempo)
Grafana's trace database. It stores traces cheaply in object storage and shows them in Grafana, with links to and from Loki and Prometheus. Lesson 45.11
traceparent (traceparent header)
The HTTP header that carries trace context between services: 00-<trace id>-<parent span id>-<flags>. Each hop keeps the trace id and puts its own span id in the middle; flags 01 means sampled. Lesson 45.11
tracestate
A second trace-context header that carries vendor-specific extras alongside traceparent. Lesson 45.11
W3C
The World Wide Web Consortium, the body that writes web standards - including Trace Context, the traceparent header format every tracing library speaks. Lesson 45.11
W3C Trace Context (W3C trace context)
The web standard that defines the traceparent and tracestate headers, so every tracing tool propagates context the same way. Lesson 45.11
waterfall (trace waterfall)
The standard drawing of a trace: one bar per span, time left to right, children indented under their parent. The widest bar with nothing under it is where the time went. Lesson 45.11