Alerts that page: alert rules, action groups and alert processing rules
An alert that never reaches a human is a dashboard nobody looks at. Azure Monitor splits "something is wrong" from "tell someone" on purpose: an alert rule watches a signal and fires; an action group says who and how to notify; alert processing rules sit in between and can silence or re-route what fired. Each part is configured in a different place, often by different teams - which is exactly why "the alert fired but nobody was paged" is one of the most common post-mortem findings. This chapter's incident is one of those; this lesson is how the pipeline works.
Need to know: an alert rule has a scope (the resources), a condition on a signal - a metric (avg Percentage CPU > 80), a log search query, or an activity log event - an evaluation frequency (how often), a window (look-back), a severity (Sev0 critical ... Sev4 verbose) and action groups. A fired alert has a monitor condition (Fired, then Resolved when the condition clears, if the rule auto-mitigates) set by Azure and an alert state (New, Acknowledged, Closed) set by people. An action group is a reusable list of receivers (email, SMS, push, voice, webhook, Logic App, Function, Automation runbook, ITSM, Event Hub) with a short name of at most 12 characters. An alert processing rule applies to fired alerts in a scope: suppress notifications (RemoveAllActionGroups) or add action groups, optionally filtered by severity, rule name or monitor service, and optionally only during a schedule.
Action groups: who gets told
$ az monitor action-group create -g rg-az5-alerts -n ag-oncall --short-name oncall --action email oncall-email [email protected] --action sms oncall-sms 40 712345678 -o table
Enabled GroupShortName Location Name ResourceGroup
------- -------------- -------- --------- -------------
True oncall Global ag-oncall rg-az5-alerts
$ az monitor action-group create -g rg-az5-alerts -n ag-platform --short-name platform-oncall-team --action email x [email protected]
ERROR: (BadRequest) The value 'platform-oncall-team' for 'groupShortName' is invalid: it must be 1 to 12 characters.
Code: BadRequest
Message: The value 'platform-oncall-team' for 'groupShortName' is invalid: it must be 1 to 12 characters.
$ mail -H
>N 1 Microsoft Azure Tue Sep 22 20:00 18/770 You've been added to an Azure Monitor ac
The short name appears in SMS and voice messages, so it is limited to 12 characters. Every email receiver gets a "You've been added to an Azure Monitor action group" mail - and can unsubscribe from it, after which the action group still exists, still looks correct, and silently skips that address. Receivers are rate-limited (roughly one SMS and one voice call per number every 5 minutes, 100 emails an hour per address); a storm of alerts does not become a storm of texts. Test the group end to end without waiting for a real incident:
$ az monitor action-group test-notifications create -g rg-az5-alerts --action-group ag-oncall -a email oncall-email [email protected] --alert-type servicehealth --query "{state:state, email:actionDetails[0].status}" -o json
{
"email": "Complete",
"state": "Complete"
}
$ mail -H | tail -1
>N 2 Microsoft Azure Tue Sep 22 20:00 14/603 Fired:Sev4 Azure Monitor Alert Test Aler
Webhooks (to PagerDuty, Opsgenie, Teams, your own endpoint) and the common alert schema - one JSON shape for every alert type, enabled per receiver - are how alerts reach an incident tool instead of an inbox.
Metric alert rules
$ VM=$(az vm show -g rg-az5-alerts -n vm-api --query id -o tsv)
$ az monitor metrics alert create -g rg-az5-alerts -n cpu-test --scopes $VM --condition "avg Percentage CPU > 1" --window-size 5m --evaluation-frequency 1m --severity 3 --action ag-oncall --description "test rule: fires on an idle VM" -o none
$ az monitor metrics alert show -g rg-az5-alerts -n cpu-test --query "{sev:severity, every:evaluationFrequency, window:windowSize, condition:criteria.allOf[0].{metric:metricName, op:operator, threshold:threshold, agg:timeAggregation}, autoMitigate:autoMitigate}" -o json
{
"autoMitigate": true,
"condition": {
"agg": "Average",
"metric": "Percentage CPU",
"op": "GreaterThan",
"threshold": 1
},
"every": "PT1M",
"sev": 3,
"window": "PT5M"
}
Read the condition as: every 1 minute, take the average of Percentage CPU over the last 5 minutes; if it is greater than 1, fire. The threshold is silly on purpose so it fires on an idle VM and you can watch the pipeline - real rules are tuned the other way: long enough windows to ignore one noisy minute, a severity that matches who must wake up, and auto-mitigation (the default) so the alert resolves itself and sends a "Resolved" notification. A metric rule can watch many resources of one type in a region (--scopes with several ids, or a resource group) and still alert per resource. Dynamic thresholds learn a metric's normal pattern and alert on deviations instead of a fixed number.
$ sleep 180
$ mail -H | tail -2
N 2 Microsoft Azure Tue Sep 22 20:00 13/602 Fired:Sev4 Azure Monitor Alert Test Aler
>N 3 Microsoft Azure Tue Sep 22 20:03 25/1183 Fired:Sev3 Azure Monitor Alert cpu-test
$ az rest --method get --url "/subscriptions/00000000-1111-2222-3333-444444444444/providers/Microsoft.AlertsManagement/alerts?api-version=2019-05-05-preview" --query "value[].properties.essentials.{rule:alertRule, severity:severity, condition:monitorCondition, state:alertState}" -o table
Rule Severity Condition State
----------------------------------------------------------------------------------------------------------------------------------- -------- --------- -----
/subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-az5-alerts/providers/Microsoft.Insights/metricAlerts/cpu-test Sev3 Fired New
The alert fired, an alert instance exists (Monitor > Alerts in the portal, or the Alerts Management API above), and the action group sent the mail. Its alert state is New until someone acknowledges or closes it - a human workflow field; it does not change what Azure does.
Log search and activity log alerts
Not everything is a metric:
$ az monitor scheduled-query create -g rg-az5-alerts -n web-5xx --scopes $(az monitor log-analytics workspace show -g rg-oncall-lab -n law-sysop --query id -o tsv) --condition "count 'Errors' > 10" --condition-query Errors="AppServiceHTTPLogs | where ScStatus >= 500" --evaluation-frequency 5m --window-size 15m --severity 1 --action-groups $(az monitor action-group show -g rg-az5-alerts -n ag-oncall --query id -o tsv) -o none
$ az monitor activity-log alert create -g rg-az5-alerts -n vm-deleted --scope /subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-az5-alerts --condition category=Administrative and operationName=Microsoft.Compute/virtualMachines/delete --action-group ag-oncall -o none
$ az monitor scheduled-query list -g rg-az5-alerts --query "[].{name:name, sev:severity, every:evaluationFrequency, window:windowSize}" -o table
Name Sev Every Window
------- --- ----- ------
web-5xx 1 PT5M PT15M
- Log search alerts run a KQL query on a schedule against a workspace (or a resource's logs) and fire on the number of rows or a measured value. They only see what a diagnostic setting sends - no logs, no alert. They cost per rule and evaluation.
- Activity log alerts fire on a control-plane event (a delete, a failed policy, a role assignment), on Service Health events (Azure's incidents and planned maintenance in your regions) and on Resource Health (this VM became unavailable). They are free.
Alert processing rules: silence and route centrally
The platform team patches these VMs every night between 22:00 and 06:00 - low-severity alerts during that window should not page anyone, but should still be recorded:
$ az monitor alert-processing-rule create -g rg-az5-alerts -n maint-tonight --rule-type RemoveAllActionGroups --scopes /subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-az5-alerts --filter-severity Equals Sev2 Sev3 Sev4 --schedule-recurrence-type Daily --schedule-recurrence-start-time 22:00:00 --schedule-recurrence-end-time 06:00:00 -o none
$ az monitor alert-processing-rule show -g rg-az5-alerts -n maint-tonight --query "properties.{actions:actions[0].actionType, filters:conditions, schedule:schedule.recurrences, enabled:enabled}" -o json
{
"actions": "RemoveAllActionGroups",
"enabled": true,
"filters": [
{
"field": "Severity",
"operator": "Equals",
"values": [
"Sev2",
"Sev3",
"Sev4"
]
}
],
"schedule": [
{
"endTime": "06:00:00",
"recurrenceType": "Daily",
"startTime": "22:00:00"
}
]
}
Alerts still fire and are recorded during the window - only the notifications are removed (unlike disabling the alert rules, which loses the history and is easy to forget to undo). The other rule type, AddActionGroups, attaches an action group to every alert in a scope ("all Sev0/Sev1 in production also go to the incident tool") so that individual rules do not need to be edited. Processing rules apply to all alert types in their scope - which makes a forgotten one with no end date the most effective way to stop an organisation's paging:
When an alert did not page
Work the pipeline from the signal to the inbox:
| question | where |
|---|---|
| is the rule enabled, on the right scope and signal? | az monitor metrics alert show (enabled, scopes, criteria) |
| did it fire? | Monitor > Alerts / az rest .../Microsoft.AlertsManagement/alerts (monitor condition, fired time) |
| was it suppressed? | az monitor alert-processing-rule list - scope, filters, schedule, enabled |
| which action group, which receivers? | az monitor action-group show - addresses, receiver status (unsubscribed = Disabled) |
| does the group deliver at all? | az monitor action-group test-notifications create |
| was it rate-limited or filtered as spam? | the receiver side; the action group's history |
On the exam (AZ-104 5.1.4): "notify by SMS and email for several alerts" = one action group reused by the rules; "stop notifications during a maintenance window without losing alerts" = an alert processing rule (RemoveAllActionGroups) with a schedule; "add the incident tool to every alert in the subscription" = an alert processing rule that adds an action group; "alert when a VM is deleted" = activity log alert; "alert on a log query result" = log search alert; "fire when CPU is unusual for this time of day" = dynamic thresholds.
In an interview: "An alert fired during an incident but nobody was paged. How do you investigate?" - "Confirm it fired and when, then check alert processing rules on its scope - a suppression left on after maintenance is common - then the action group: the right receivers, nobody unsubscribed, a test notification arrives. And I'd add a regular synthetic test of the paging path so we find out before the next incident."