OnCallReadyStart learning

Azure Lesson 33.38 · Azure V: Compute, App Hosting & the AZ-104 exam · 19 min read

Alerts that page: alert rules, action groups and alert processing rules

In plain words

An alert rule is a smoke detector: it watches one thing and goes off when it crosses a line. An action group is the list of people to phone when a detector goes off, and how - email, text message, a webhook into the on-call tool.

Alert processing rules sit in between, like a switch in the hallway: during planned maintenance you can turn off the phoning for a while, or add extra people to call for the serious ones.

Alerts that page: alert rules, action groups and alert processing rules

An alert that never reaches a human is a dashboard nobody looks at. Azure Monitor splits "something is wrong" from "tell someone" on purpose: an alert rule watches a signal and fires; an action group says who and how to notify; alert processing rules sit in between and can silence or re-route what fired. Each part is configured in a different place, often by different teams - which is exactly why "the alert fired but nobody was paged" is one of the most common post-mortem findings. This chapter's incident is one of those; this lesson is how the pipeline works.

Need to know: an alert rule has a scope (the resources), a condition on a signal - a metric (avg Percentage CPU > 80), a log search query, or an activity log event - an evaluation frequency (how often), a window (look-back), a severity (Sev0 critical ... Sev4 verbose) and action groups. A fired alert has a monitor condition (Fired, then Resolved when the condition clears, if the rule auto-mitigates) set by Azure and an alert state (New, Acknowledged, Closed) set by people. An action group is a reusable list of receivers (email, SMS, push, voice, webhook, Logic App, Function, Automation runbook, ITSM, Event Hub) with a short name of at most 12 characters. An alert processing rule applies to fired alerts in a scope: suppress notifications (RemoveAllActionGroups) or add action groups, optionally filtered by severity, rule name or monitor service, and optionally only during a schedule.

Action groups: who gets told

$ az monitor action-group create -g rg-az5-alerts -n ag-oncall --short-name oncall --action email oncall-email [email protected] --action sms oncall-sms 40 712345678 -o table
Enabled  GroupShortName  Location  Name       ResourceGroup
-------  --------------  --------  ---------  -------------
True     oncall          Global    ag-oncall  rg-az5-alerts
$ az monitor action-group create -g rg-az5-alerts -n ag-platform --short-name platform-oncall-team --action email x [email protected]
ERROR: (BadRequest) The value 'platform-oncall-team' for 'groupShortName' is invalid: it must be 1 to 12 characters.
Code: BadRequest
Message: The value 'platform-oncall-team' for 'groupShortName' is invalid: it must be 1 to 12 characters.
$ mail -H
>N   1 Microsoft Azure    Tue Sep 22 20:00  18/770   You've been added to an Azure Monitor ac

The short name appears in SMS and voice messages, so it is limited to 12 characters. Every email receiver gets a "You've been added to an Azure Monitor action group" mail - and can unsubscribe from it, after which the action group still exists, still looks correct, and silently skips that address. Receivers are rate-limited (roughly one SMS and one voice call per number every 5 minutes, 100 emails an hour per address); a storm of alerts does not become a storm of texts. Test the group end to end without waiting for a real incident:

$ az monitor action-group test-notifications create -g rg-az5-alerts --action-group ag-oncall -a email oncall-email [email protected] --alert-type servicehealth --query "{state:state, email:actionDetails[0].status}" -o json
{
  "email": "Complete",
  "state": "Complete"
}
$ mail -H | tail -1
>N   2 Microsoft Azure    Tue Sep 22 20:00  14/603   Fired:Sev4 Azure Monitor Alert Test Aler

Webhooks (to PagerDuty, Opsgenie, Teams, your own endpoint) and the common alert schema - one JSON shape for every alert type, enabled per receiver - are how alerts reach an incident tool instead of an inbox.

Metric alert rules

$ VM=$(az vm show -g rg-az5-alerts -n vm-api --query id -o tsv)
$ az monitor metrics alert create -g rg-az5-alerts -n cpu-test --scopes $VM --condition "avg Percentage CPU > 1" --window-size 5m --evaluation-frequency 1m --severity 3 --action ag-oncall --description "test rule: fires on an idle VM" -o none
$ az monitor metrics alert show -g rg-az5-alerts -n cpu-test --query "{sev:severity, every:evaluationFrequency, window:windowSize, condition:criteria.allOf[0].{metric:metricName, op:operator, threshold:threshold, agg:timeAggregation}, autoMitigate:autoMitigate}" -o json
{
  "autoMitigate": true,
  "condition": {
    "agg": "Average",
    "metric": "Percentage CPU",
    "op": "GreaterThan",
    "threshold": 1
  },
  "every": "PT1M",
  "sev": 3,
  "window": "PT5M"
}

Read the condition as: every 1 minute, take the average of Percentage CPU over the last 5 minutes; if it is greater than 1, fire. The threshold is silly on purpose so it fires on an idle VM and you can watch the pipeline - real rules are tuned the other way: long enough windows to ignore one noisy minute, a severity that matches who must wake up, and auto-mitigation (the default) so the alert resolves itself and sends a "Resolved" notification. A metric rule can watch many resources of one type in a region (--scopes with several ids, or a resource group) and still alert per resource. Dynamic thresholds learn a metric's normal pattern and alert on deviations instead of a fixed number.

$ sleep 180
$ mail -H | tail -2
 N   2 Microsoft Azure    Tue Sep 22 20:00  13/602   Fired:Sev4 Azure Monitor Alert Test Aler
>N   3 Microsoft Azure    Tue Sep 22 20:03  25/1183   Fired:Sev3 Azure Monitor Alert cpu-test
$ az rest --method get --url "/subscriptions/00000000-1111-2222-3333-444444444444/providers/Microsoft.AlertsManagement/alerts?api-version=2019-05-05-preview" --query "value[].properties.essentials.{rule:alertRule, severity:severity, condition:monitorCondition, state:alertState}" -o table
Rule                                                                                                                                 Severity  Condition  State
-----------------------------------------------------------------------------------------------------------------------------------  --------  ---------  -----
/subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-az5-alerts/providers/Microsoft.Insights/metricAlerts/cpu-test  Sev3      Fired      New

The alert fired, an alert instance exists (Monitor > Alerts in the portal, or the Alerts Management API above), and the action group sent the mail. Its alert state is New until someone acknowledges or closes it - a human workflow field; it does not change what Azure does.

Log search and activity log alerts

Not everything is a metric:

$ az monitor scheduled-query create -g rg-az5-alerts -n web-5xx --scopes $(az monitor log-analytics workspace show -g rg-oncall-lab -n law-sysop --query id -o tsv) --condition "count 'Errors' > 10" --condition-query Errors="AppServiceHTTPLogs | where ScStatus >= 500" --evaluation-frequency 5m --window-size 15m --severity 1 --action-groups $(az monitor action-group show -g rg-az5-alerts -n ag-oncall --query id -o tsv) -o none
$ az monitor activity-log alert create -g rg-az5-alerts -n vm-deleted --scope /subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-az5-alerts --condition category=Administrative and operationName=Microsoft.Compute/virtualMachines/delete --action-group ag-oncall -o none
$ az monitor scheduled-query list -g rg-az5-alerts --query "[].{name:name, sev:severity, every:evaluationFrequency, window:windowSize}" -o table
Name     Sev  Every  Window
-------  ---  -----  ------
web-5xx  1    PT5M   PT15M
  • Log search alerts run a KQL query on a schedule against a workspace (or a resource's logs) and fire on the number of rows or a measured value. They only see what a diagnostic setting sends - no logs, no alert. They cost per rule and evaluation.
  • Activity log alerts fire on a control-plane event (a delete, a failed policy, a role assignment), on Service Health events (Azure's incidents and planned maintenance in your regions) and on Resource Health (this VM became unavailable). They are free.

Alert processing rules: silence and route centrally

The platform team patches these VMs every night between 22:00 and 06:00 - low-severity alerts during that window should not page anyone, but should still be recorded:

$ az monitor alert-processing-rule create -g rg-az5-alerts -n maint-tonight --rule-type RemoveAllActionGroups --scopes /subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-az5-alerts --filter-severity Equals Sev2 Sev3 Sev4 --schedule-recurrence-type Daily --schedule-recurrence-start-time 22:00:00 --schedule-recurrence-end-time 06:00:00 -o none
$ az monitor alert-processing-rule show -g rg-az5-alerts -n maint-tonight --query "properties.{actions:actions[0].actionType, filters:conditions, schedule:schedule.recurrences, enabled:enabled}" -o json
{
  "actions": "RemoveAllActionGroups",
  "enabled": true,
  "filters": [
    {
      "field": "Severity",
      "operator": "Equals",
      "values": [
        "Sev2",
        "Sev3",
        "Sev4"
      ]
    }
  ],
  "schedule": [
    {
      "endTime": "06:00:00",
      "recurrenceType": "Daily",
      "startTime": "22:00:00"
    }
  ]
}

Alerts still fire and are recorded during the window - only the notifications are removed (unlike disabling the alert rules, which loses the history and is easy to forget to undo). The other rule type, AddActionGroups, attaches an action group to every alert in a scope ("all Sev0/Sev1 in production also go to the incident tool") so that individual rules do not need to be edited. Processing rules apply to all alert types in their scope - which makes a forgotten one with no end date the most effective way to stop an organisation's paging:

When an alert did not page

Work the pipeline from the signal to the inbox:

questionwhere
is the rule enabled, on the right scope and signal?az monitor metrics alert show (enabled, scopes, criteria)
did it fire?Monitor > Alerts / az rest .../Microsoft.AlertsManagement/alerts (monitor condition, fired time)
was it suppressed?az monitor alert-processing-rule list - scope, filters, schedule, enabled
which action group, which receivers?az monitor action-group show - addresses, receiver status (unsubscribed = Disabled)
does the group deliver at all?az monitor action-group test-notifications create
was it rate-limited or filtered as spam?the receiver side; the action group's history

On the exam (AZ-104 5.1.4): "notify by SMS and email for several alerts" = one action group reused by the rules; "stop notifications during a maintenance window without losing alerts" = an alert processing rule (RemoveAllActionGroups) with a schedule; "add the incident tool to every alert in the subscription" = an alert processing rule that adds an action group; "alert when a VM is deleted" = activity log alert; "alert on a log query result" = log search alert; "fire when CPU is unusual for this time of day" = dynamic thresholds.

In an interview: "An alert fired during an incident but nobody was paged. How do you investigate?" - "Confirm it fired and when, then check alert processing rules on its scope - a suppression left on after maintenance is common - then the action group: the right receivers, nobody unsubscribed, a test notification arrives. And I'd add a regular synthetic test of the paging path so we find out before the next incident."

Why it helps

Alerting is the part of monitoring that wakes people, so its failures are the expensive ones: the alert fired but the action group was not attached, the receiver had unsubscribed, the maintenance suppression was never removed.

AZ-104 asks you to create alert rules, action groups and processing rules; on call you will mostly debug them. This chapter's incident is an alert that never paged, and the lesson gives you the checklist.

Commands in this lesson

az mail sleep

FAQ

Monitor condition vs alert state?

The monitor condition is set by Azure: Fired when the condition is met, Resolved when it clears (if the rule auto-mitigates). The alert state is set by people: New, Acknowledged, Closed. Closing an alert does not resolve the condition.

How short must an action group's short name be?

At most 12 characters. It appears in SMS and email notifications so people know which group alerted them. A longer one is refused when you create the group, so pick something the on-call engineer recognises at a glance, like oncall or db-team.

What are the action group rate limits?

No more than one SMS per 5 minutes, one voice call per 5 minutes and 100 emails per hour per number or address; extra notifications are dropped. A receiver that unsubscribes is disabled until it is enabled again (az monitor action-group enable-receiver).

Which kinds of alert rules are there?

Metric alerts (a platform or custom metric over a window, static or dynamic thresholds), log search alerts (a KQL query run on a schedule), activity log alerts (an administrative operation, service health or resource health event) and others like Prometheus and smart detection. Each type is billed differently.

What can an alert processing rule do?

Within a scope and optional filters (severity, alert rule name, monitor service, resource type) and an optional schedule, it either removes all action groups (suppression) or adds action groups to fired alerts. The alerts still fire and are visible; only notifications change.

In an interview Mid

An alert fired during an incident but nobody was paged. How do you investigate?

Confirm it fired and when, then check alert processing rules on its scope - a suppression left on after maintenance is common - then the action group: the right receivers, nobody unsubscribed, a test notification arrives (az monitor action-group test-notifications create). And I would add a regular synthetic test of the paging path so we find out before the next incident.

Also asked: What is the difference between an alert rule and an action group? · How do you silence alerts during a planned maintenance window? · When would you use a log search alert instead of a metric alert?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.