Azure Monitor: metrics, diagnostic settings, log queries and Insights
Azure III was about reading telemetry: KQL against a workspace someone had already filled. This lesson is the administrator's half - deciding what gets collected and where it goes. Out of the box Azure gives you platform metrics for every resource and the activity log for every change; the detailed resource logs (every HTTP request, every storage read, every Key Vault access) are collected only if you create a diagnostic setting; what happens inside a VM needs an agent. Most "we have no logs for the incident" post-mortems are a missing diagnostic setting or a VM without the agent.
Need to know: metrics are numeric time series, collected automatically, kept 93 days, read with az monitor metrics list (aggregation: Average, Total, Maximum, Minimum, Count; time grain: --interval). Resource logs are off until a diagnostic setting sends chosen categories to a Log Analytics workspace, a storage account, an event hub or a partner (up to 5 settings per resource). The activity log (subscription-level control-plane events, 90 days) reaches a workspace through a subscription diagnostic setting. VM insights needs the Azure Monitor Agent extension and a data collection rule (DCR) associated with the VM; its data lands in InsightsMetrics. Insights (VM, storage, network, container) are curated views built on these sources.
Metrics: what you get for free
$ az monitor metrics list-definitions --resource $(az webapp show -g rg-az5-mon -n shop-mon-5 --query id -o tsv) --query "[].{metric:name.value, unit:unit, aggregation:primaryAggregationType}" -o table
Metric Unit Aggregation
---------------- ------- -----------
Requests Count Total
Http2xx Count Total
Http4xx Count Total
Http5xx Count Total
HttpResponseTime Seconds Average
$ for i in $(seq 12); do curl -s -o /dev/null https://shop-mon-5.azurewebsites.net/; done
$ sleep 120
$ az monitor metrics list --resource $(az webapp show -g rg-az5-mon -n shop-mon-5 --query id -o tsv) --metric Requests --aggregation Total --interval PT1M --offset 5m -o table
Timestamp Name Total
------------------- -------- -----
2026-09-22 19:57:00 Requests
2026-09-22 19:58:00 Requests
2026-09-22 19:59:00 Requests
2026-09-22 20:00:00 Requests 12
2026-09-22 20:01:00 Requests 0
Every resource type publishes its own set (list-definitions). Reading them well is three choices:
- the aggregation -
Totalfor counts (requests, errors, bytes),Averagefor utilisation,Maximumfor "did it ever spike"; each metric has a primary one (primaryAggregationType); - the time grain (
--interval PT1M,PT5M,PT1H,P1D) - one point per grain; a 1-minute spike disappears into a 1-hour average; - the window (
--offset 1hor--start-time/--end-time) - at most 1,440 points per call.
Platform metrics are kept 93 days at full resolution; for longer, or to join them with logs, send them to a workspace with a diagnostic setting (AllMetrics -> the AzureMetrics table). VM CPU, disk and network are platform metrics (from the host); memory and disk space inside the guest are not - the host cannot see them. That needs the agent (below).
Log settings: diagnostic settings
$ APP=$(az webapp show -g rg-az5-mon -n shop-mon-5 --query id -o tsv)
$ az monitor diagnostic-settings categories list --resource $APP --query "value[].{category:name, type:categoryType}" -o table
Category
------------------------
AppServiceHTTPLogs
AppServiceConsoleLogs
AppServiceAppLogs
AppServiceAuditLogs
AppServiceIPSecAuditLogs
AppServicePlatformLogs
AllMetrics
$ az monitor diagnostic-settings list --resource $APP -o table
The web app can produce six log categories and AllMetrics; none is collected until a diagnostic setting says where. A setting picks categories (or a category group: allLogs, audit) and one destination of each kind:
| destination | for | notes |
|---|---|---|
| Log Analytics workspace | querying, alerts, Insights | KQL; per-GB ingestion cost; resource-specific tables (AppServiceHTTPLogs) or the legacy AzureDiagnostics |
| Storage account | cheap archive, compliance | blobs per hour; retention by lifecycle rules |
| Event Hubs | streaming to a SIEM or a third party | |
| Partner solution | Datadog, Elastic... |
$ LAW=$(az monitor log-analytics workspace show -g rg-oncall-lab -n law-sysop --query id -o tsv)
$ az monitor diagnostic-settings create -n to-law --resource $APP --workspace $LAW --logs '[{"category":"AppServiceHTTPLogs","enabled":true},{"category":"AppServiceAppLogs","enabled":true}]' --metrics '[{"category":"AllMetrics","enabled":true}]' -o table
Name ResourceGroup WorkspaceId
------ ------------- ---------------------------------------------------------------------------------------------------------------------------------------------
to-law rg-az5-mon /subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-oncall-lab/providers/Microsoft.OperationalInsights/workspaces/law-sysop
$ for i in $(seq 8); do curl -s -o /dev/null https://shop-mon-5.azurewebsites.net/; done
$ sleep 300
Logs arrive with a few minutes' delay, and only from the moment the setting exists - nothing is backfilled. That is why diagnostic settings belong in the template or a policy (DeployIfNotExists "Configure diagnostic settings ... to Log Analytics") rather than in a ticket after the outage. A resource can have up to five settings; two settings sending the same category to the same workspace duplicate the data (and the bill).
The activity log - who created, changed, deleted or failed to change what, at the control plane - is kept 90 days for free and can be queried with az monitor activity-log list. To keep it longer or query it with KQL, a subscription-level diagnostic setting sends it to a workspace (AzureActivity table):
$ az monitor diagnostic-settings subscription create -n activity-to-law -l westeurope --workspace $LAW --logs '[{"category":"Administrative","enabled":true},{"category":"Policy","enabled":true},{"category":"Autoscale","enabled":true}]' --query "logs[].category" -o tsv
Administrative
Policy
Autoscale
Querying the logs
The rows are in the workspace now. KQL - which Azure III covered in depth - from the CLI:
$ WS=$(az monitor log-analytics workspace show -g rg-oncall-lab -n law-sysop --query customerId -o tsv)
$ az monitor log-analytics query -w $WS --analytics-query "AppServiceHTTPLogs | where TimeGenerated > ago(15m) | summarize requests=count() by CsUriStem, ScStatus" -o table
CsUriStem ScStatus TableName Requests
--------- -------- ------------- --------
/ 200 PrimaryResult 8
$ az monitor log-analytics query -w $WS --analytics-query "AzureMetrics | where MetricName == 'Requests' | summarize total=sum(Total) by bin(TimeGenerated, 5m) | order by TimeGenerated desc | take 3" -o table
TableName TimeGenerated Total
------------- -------------------- -----
PrimaryResult 2026-09-22T20:00:00Z 8
Two things the exam likes: az monitor log-analytics query -w takes the workspace ID (the GUID, customerId), not the resource id; and a query runs in the context of a workspace, a resource (only that resource's rows, "resource-context" access) or a whole resource group, which is how RBAC can let a team read their own logs without seeing everyone else's.
Inside the VM: the agent, data collection rules and VM insights
$ VM=$(az vm show -g rg-az5-mon -n vm-api --query id -o tsv)
$ az monitor log-analytics query -w $WS --analytics-query "InsightsMetrics | where Computer == 'vm-api' | count" -o tsv
0 PrimaryResult
$ az vm extension list -g rg-az5-mon --vm-name vm-api -o table
Nothing: no agent, so nothing from inside the guest. The modern pipeline has three parts:
- the Azure Monitor Agent (AMA), a VM extension (
AzureMonitorLinuxAgent/AzureMonitorWindowsAgent) - it replaced the old Log Analytics (MMA/OMS) agent, retired in August 2024; - a data collection rule (DCR): what to collect (performance counters, syslog, Windows events, text logs) and where to send it;
- a DCR association linking the rule to the VM (one VM can have several rules; one rule many VMs).
VM insights is a DCR that collects a fixed counter set (\VmInsights\DetailedMetrics) into the Microsoft-InsightsMetrics stream:
$ cd ~/oncall-lab/labs/4b-azure/az5/monitor
$ jq '.properties | {sources: .dataSources.performanceCounters[0].counterSpecifiers, flows: .dataFlows}' dcr-vminsights.json
{
"sources": [
"\\VmInsights\\DetailedMetrics"
],
"flows": [
{
"streams": [
"Microsoft-InsightsMetrics"
],
"destinations": [
"vmInsightworkspace"
]
}
]
}
$ az vm extension set -g rg-az5-mon --vm-name vm-api -n AzureMonitorLinuxAgent --publisher Microsoft.Azure.Monitor --enable-auto-upgrade true -o none
$ az monitor data-collection rule create -g rg-az5-mon -n dcr-vminsights --rule-file dcr-vminsights.json --query "{name:name, streams:dataFlows[0].streams}" -o json
{
"name": "dcr-vminsights",
"streams": [
"Microsoft-InsightsMetrics"
]
}
$ az monitor data-collection rule association create --name vminsights-api --rule-id $(az monitor data-collection rule show -g rg-az5-mon -n dcr-vminsights --query id -o tsv) --resource $VM -o none
$ sleep 420
$ az monitor log-analytics query -w $WS --analytics-query "InsightsMetrics | where Computer == 'vm-api' and TimeGenerated > ago(5m) | summarize avg(Val) by Namespace, Name" -o table
Name Namespace TableName Avg_Val
--------------------- ----------- ------------- ------------------
UtilizationPercentage Processor PrimaryResult 3.4163333333333328
AvailableMB Memory PrimaryResult 2911.957
FreeSpacePercentage LogicalDisk PrimaryResult 81.6
ReadBytesPerSecond Network PrimaryResult 167.211
$ az monitor log-analytics query -w $WS --analytics-query "Heartbeat | where Computer == 'vm-api' | summarize last_seen=max(TimeGenerated) by Computer, Category" -o table
Category Computer TableName Last_seen
------------------- -------- ------------- --------------------
Azure Monitor Agent vm-api PrimaryResult 2026-09-22T20:13:00Z
Memory available and disk free space - the two numbers platform metrics cannot give you - now arrive every minute. The VM insights blades in the portal (performance charts, and the Map of processes and connections, which also needs the Dependency agent) are views over InsightsMetrics. When VM insights shows "no data": is the agent installed and healthy (Heartbeat), is a DCR associated, and does the DCR send to the workspace you are looking at?
The other Insights
- Storage insights - a workbook over every storage account's platform metrics: transactions by API and response type (throttling,
ClientOtherError, authorization failures), end-to-end vs server latency (a gap means the client or network is slow, not storage), availability, capacity. No setup; per-request detail needs the account's diagnostic setting (StorageBlobLogs). - Network insights - the topology and health of VNets, gateways, load balancers, Application Gateways; it links to Network Watcher and connection monitor (the next lessons).
- Container insights - AKS, from Azure III.
On the exam (AZ-104 5.1.1-5.1.3, 5.1.5): "collect a resource's logs" = a diagnostic setting (they are not collected by default); "keep the activity log longer than 90 days / query it" = a subscription diagnostic setting to a workspace; "memory usage of a VM" = the Azure Monitor Agent with a DCR (VM insights), not platform metrics; "which aggregation" = Total for counts, Average for utilisation, Maximum for peaks; "query the logs" = Log Analytics with KQL.
In an interview: "During an incident we found the web app's request logs were never collected. How do you stop that happening again?" - "Diagnostic settings are part of the resource's definition: put them in the template or module, and back that with an Azure Policy that deploys them to the central workspace when they are missing, plus a compliance report. Logs only exist from the moment a setting exists, so it can't be an incident-time task."