OnCallReadyStart learning

Azure Lesson 33.35 · Azure V: Compute, App Hosting & the AZ-104 exam · 22 min read

Azure Monitor: metrics, diagnostic settings, log queries and Insights

In plain words

Azure Monitor is the building's sensor system. Some sensors are on by default - every resource reports numbers like CPU and requests (platform metrics). The detailed logbooks (resource logs) are only written down if you tell Azure where to keep them, with a diagnostic setting.

Inside a VM, Azure cannot see memory or disk space at all until you install a small agent and tell it what to collect - a data collection rule.

Azure Monitor: metrics, diagnostic settings, log queries and Insights

Azure III was about reading telemetry: KQL against a workspace someone had already filled. This lesson is the administrator's half - deciding what gets collected and where it goes. Out of the box Azure gives you platform metrics for every resource and the activity log for every change; the detailed resource logs (every HTTP request, every storage read, every Key Vault access) are collected only if you create a diagnostic setting; what happens inside a VM needs an agent. Most "we have no logs for the incident" post-mortems are a missing diagnostic setting or a VM without the agent.

Need to know: metrics are numeric time series, collected automatically, kept 93 days, read with az monitor metrics list (aggregation: Average, Total, Maximum, Minimum, Count; time grain: --interval). Resource logs are off until a diagnostic setting sends chosen categories to a Log Analytics workspace, a storage account, an event hub or a partner (up to 5 settings per resource). The activity log (subscription-level control-plane events, 90 days) reaches a workspace through a subscription diagnostic setting. VM insights needs the Azure Monitor Agent extension and a data collection rule (DCR) associated with the VM; its data lands in InsightsMetrics. Insights (VM, storage, network, container) are curated views built on these sources.

Metrics: what you get for free

$ az monitor metrics list-definitions --resource $(az webapp show -g rg-az5-mon -n shop-mon-5 --query id -o tsv) --query "[].{metric:name.value, unit:unit, aggregation:primaryAggregationType}" -o table
Metric            Unit     Aggregation
----------------  -------  -----------
Requests          Count    Total
Http2xx           Count    Total
Http4xx           Count    Total
Http5xx           Count    Total
HttpResponseTime  Seconds  Average
$ for i in $(seq 12); do curl -s -o /dev/null https://shop-mon-5.azurewebsites.net/; done
$ sleep 120
$ az monitor metrics list --resource $(az webapp show -g rg-az5-mon -n shop-mon-5 --query id -o tsv) --metric Requests --aggregation Total --interval PT1M --offset 5m -o table
Timestamp            Name      Total
-------------------  --------  -----
2026-09-22 19:57:00  Requests
2026-09-22 19:58:00  Requests
2026-09-22 19:59:00  Requests
2026-09-22 20:00:00  Requests  12
2026-09-22 20:01:00  Requests  0

Every resource type publishes its own set (list-definitions). Reading them well is three choices:

  • the aggregation - Total for counts (requests, errors, bytes), Average for utilisation, Maximum for "did it ever spike"; each metric has a primary one (primaryAggregationType);
  • the time grain (--interval PT1M, PT5M, PT1H, P1D) - one point per grain; a 1-minute spike disappears into a 1-hour average;
  • the window (--offset 1h or --start-time/--end-time) - at most 1,440 points per call.

Platform metrics are kept 93 days at full resolution; for longer, or to join them with logs, send them to a workspace with a diagnostic setting (AllMetrics -> the AzureMetrics table). VM CPU, disk and network are platform metrics (from the host); memory and disk space inside the guest are not - the host cannot see them. That needs the agent (below).

Log settings: diagnostic settings

$ APP=$(az webapp show -g rg-az5-mon -n shop-mon-5 --query id -o tsv)
$ az monitor diagnostic-settings categories list --resource $APP --query "value[].{category:name, type:categoryType}" -o table
Category
------------------------
AppServiceHTTPLogs
AppServiceConsoleLogs
AppServiceAppLogs
AppServiceAuditLogs
AppServiceIPSecAuditLogs
AppServicePlatformLogs
AllMetrics
$ az monitor diagnostic-settings list --resource $APP -o table

The web app can produce six log categories and AllMetrics; none is collected until a diagnostic setting says where. A setting picks categories (or a category group: allLogs, audit) and one destination of each kind:

destinationfornotes
Log Analytics workspacequerying, alerts, InsightsKQL; per-GB ingestion cost; resource-specific tables (AppServiceHTTPLogs) or the legacy AzureDiagnostics
Storage accountcheap archive, complianceblobs per hour; retention by lifecycle rules
Event Hubsstreaming to a SIEM or a third party
Partner solutionDatadog, Elastic...
$ LAW=$(az monitor log-analytics workspace show -g rg-oncall-lab -n law-sysop --query id -o tsv)
$ az monitor diagnostic-settings create -n to-law --resource $APP --workspace $LAW --logs '[{"category":"AppServiceHTTPLogs","enabled":true},{"category":"AppServiceAppLogs","enabled":true}]' --metrics '[{"category":"AllMetrics","enabled":true}]' -o table
Name    ResourceGroup  WorkspaceId
------  -------------  ---------------------------------------------------------------------------------------------------------------------------------------------
to-law  rg-az5-mon     /subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-oncall-lab/providers/Microsoft.OperationalInsights/workspaces/law-sysop
$ for i in $(seq 8); do curl -s -o /dev/null https://shop-mon-5.azurewebsites.net/; done
$ sleep 300

Logs arrive with a few minutes' delay, and only from the moment the setting exists - nothing is backfilled. That is why diagnostic settings belong in the template or a policy (DeployIfNotExists "Configure diagnostic settings ... to Log Analytics") rather than in a ticket after the outage. A resource can have up to five settings; two settings sending the same category to the same workspace duplicate the data (and the bill).

The activity log - who created, changed, deleted or failed to change what, at the control plane - is kept 90 days for free and can be queried with az monitor activity-log list. To keep it longer or query it with KQL, a subscription-level diagnostic setting sends it to a workspace (AzureActivity table):

$ az monitor diagnostic-settings subscription create -n activity-to-law -l westeurope --workspace $LAW --logs '[{"category":"Administrative","enabled":true},{"category":"Policy","enabled":true},{"category":"Autoscale","enabled":true}]' --query "logs[].category" -o tsv
Administrative
Policy
Autoscale

Querying the logs

The rows are in the workspace now. KQL - which Azure III covered in depth - from the CLI:

$ WS=$(az monitor log-analytics workspace show -g rg-oncall-lab -n law-sysop --query customerId -o tsv)
$ az monitor log-analytics query -w $WS --analytics-query "AppServiceHTTPLogs | where TimeGenerated > ago(15m) | summarize requests=count() by CsUriStem, ScStatus" -o table
CsUriStem  ScStatus  TableName      Requests
---------  --------  -------------  --------
/          200       PrimaryResult  8
$ az monitor log-analytics query -w $WS --analytics-query "AzureMetrics | where MetricName == 'Requests' | summarize total=sum(Total) by bin(TimeGenerated, 5m) | order by TimeGenerated desc | take 3" -o table
TableName      TimeGenerated         Total
-------------  --------------------  -----
PrimaryResult  2026-09-22T20:00:00Z  8

Two things the exam likes: az monitor log-analytics query -w takes the workspace ID (the GUID, customerId), not the resource id; and a query runs in the context of a workspace, a resource (only that resource's rows, "resource-context" access) or a whole resource group, which is how RBAC can let a team read their own logs without seeing everyone else's.

Inside the VM: the agent, data collection rules and VM insights

$ VM=$(az vm show -g rg-az5-mon -n vm-api --query id -o tsv)
$ az monitor log-analytics query -w $WS --analytics-query "InsightsMetrics | where Computer == 'vm-api' | count" -o tsv
0	PrimaryResult
$ az vm extension list -g rg-az5-mon --vm-name vm-api -o table

Nothing: no agent, so nothing from inside the guest. The modern pipeline has three parts:

  1. the Azure Monitor Agent (AMA), a VM extension (AzureMonitorLinuxAgent / AzureMonitorWindowsAgent) - it replaced the old Log Analytics (MMA/OMS) agent, retired in August 2024;
  2. a data collection rule (DCR): what to collect (performance counters, syslog, Windows events, text logs) and where to send it;
  3. a DCR association linking the rule to the VM (one VM can have several rules; one rule many VMs).

VM insights is a DCR that collects a fixed counter set (\VmInsights\DetailedMetrics) into the Microsoft-InsightsMetrics stream:

$ cd ~/oncall-lab/labs/4b-azure/az5/monitor
$ jq '.properties | {sources: .dataSources.performanceCounters[0].counterSpecifiers, flows: .dataFlows}' dcr-vminsights.json
{
  "sources": [
    "\\VmInsights\\DetailedMetrics"
  ],
  "flows": [
    {
      "streams": [
        "Microsoft-InsightsMetrics"
      ],
      "destinations": [
        "vmInsightworkspace"
      ]
    }
  ]
}
$ az vm extension set -g rg-az5-mon --vm-name vm-api -n AzureMonitorLinuxAgent --publisher Microsoft.Azure.Monitor --enable-auto-upgrade true -o none
$ az monitor data-collection rule create -g rg-az5-mon -n dcr-vminsights --rule-file dcr-vminsights.json --query "{name:name, streams:dataFlows[0].streams}" -o json
{
  "name": "dcr-vminsights",
  "streams": [
    "Microsoft-InsightsMetrics"
  ]
}
$ az monitor data-collection rule association create --name vminsights-api --rule-id $(az monitor data-collection rule show -g rg-az5-mon -n dcr-vminsights --query id -o tsv) --resource $VM -o none
$ sleep 420
$ az monitor log-analytics query -w $WS --analytics-query "InsightsMetrics | where Computer == 'vm-api' and TimeGenerated > ago(5m) | summarize avg(Val) by Namespace, Name" -o table
Name                   Namespace    TableName      Avg_Val
---------------------  -----------  -------------  ------------------
UtilizationPercentage  Processor    PrimaryResult  3.4163333333333328
AvailableMB            Memory       PrimaryResult  2911.957
FreeSpacePercentage    LogicalDisk  PrimaryResult  81.6
ReadBytesPerSecond     Network      PrimaryResult  167.211
$ az monitor log-analytics query -w $WS --analytics-query "Heartbeat | where Computer == 'vm-api' | summarize last_seen=max(TimeGenerated) by Computer, Category" -o table
Category             Computer  TableName      Last_seen
-------------------  --------  -------------  --------------------
Azure Monitor Agent  vm-api    PrimaryResult  2026-09-22T20:13:00Z

Memory available and disk free space - the two numbers platform metrics cannot give you - now arrive every minute. The VM insights blades in the portal (performance charts, and the Map of processes and connections, which also needs the Dependency agent) are views over InsightsMetrics. When VM insights shows "no data": is the agent installed and healthy (Heartbeat), is a DCR associated, and does the DCR send to the workspace you are looking at?

The other Insights

  • Storage insights - a workbook over every storage account's platform metrics: transactions by API and response type (throttling, ClientOtherError, authorization failures), end-to-end vs server latency (a gap means the client or network is slow, not storage), availability, capacity. No setup; per-request detail needs the account's diagnostic setting (StorageBlobLogs).
  • Network insights - the topology and health of VNets, gateways, load balancers, Application Gateways; it links to Network Watcher and connection monitor (the next lessons).
  • Container insights - AKS, from Azure III.

On the exam (AZ-104 5.1.1-5.1.3, 5.1.5): "collect a resource's logs" = a diagnostic setting (they are not collected by default); "keep the activity log longer than 90 days / query it" = a subscription diagnostic setting to a workspace; "memory usage of a VM" = the Azure Monitor Agent with a DCR (VM insights), not platform metrics; "which aggregation" = Total for counts, Average for utilisation, Maximum for peaks; "query the logs" = Log Analytics with KQL.

In an interview: "During an incident we found the web app's request logs were never collected. How do you stop that happening again?" - "Diagnostic settings are part of the resource's definition: put them in the template or module, and back that with an Azure Policy that deploys them to the central workspace when they are missing, plus a compliance report. Logs only exist from the moment a setting exists, so it can't be an incident-time task."

Why it helps

The worst moment to discover that logs were never collected is during an incident, and it happens often, because resource logs are off until someone adds a diagnostic setting and there is no backfill.

AZ-104 asks you to read metrics, configure diagnostic settings, query Log Analytics and set up VM insights. The lesson does each, and shows the exact failure: a VM with no Heartbeat in the workspace because its data collection rule was never associated.

Commands in this lesson

az sleep cd jq

FAQ

How long are platform metrics kept?

93 days, at up to one-minute granularity, at no cost and with no setup. For longer, or to query them with logs, send them to a Log Analytics workspace with a diagnostic setting (the AllMetrics category).

Where can a diagnostic setting send data?

To a Log Analytics workspace (to query with KQL), a storage account (cheap archive), an event hub (to stream to a SIEM or another tool) or a partner solution. One setting can send to several of them; a resource can have up to five settings.

What is the activity log?

The subscription's control-plane log: who created, changed or deleted what, plus service health events. It is kept 90 days. Send it to a workspace with a subscription diagnostic setting to query it (the AzureActivity table) and keep it longer.

What is a data collection rule?

A resource that tells the Azure Monitor Agent what to collect (performance counters, syslog, Windows events) and where to send it. A data collection rule association links it to each VM. VM insights is a ready-made rule that fills InsightsMetrics and the map.

Why does VM insights show no data?

Check, in order: the Azure Monitor Agent extension is installed and provisioned, the VM has an association to a VM insights data collection rule, and that rule sends to the workspace you are looking at. The Heartbeat table shows whether the agent reports at all.

In an interview Mid

During an incident we found the web app's request logs were never collected. How do you stop that happening again?

Diagnostic settings are part of the resource's definition: put them in the template or module, and back that with an Azure Policy that deploys them to the central Log Analytics workspace when they are missing, plus a compliance report. Logs only exist from the moment a setting exists, so it cannot be an incident-time task.

Also asked: What is the difference between platform metrics and resource logs? · How do you collect memory and disk usage from inside an Azure VM? · What does the activity log record and how long is it kept?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.