OnCallReadyStart learning

Azure Lesson 33.41 · Azure V: Compute, App Hosting & the AZ-104 exam · 17 min read

Network Watcher and Connection monitor

In plain words

Network Watcher is a toolbox for the network. One tool asks "would the guard let this packet in?" (IP flow verify). Another asks "which way would this packet be sent?" (next hop). Another tries the whole trip once and tells you where it stopped (connection troubleshoot).

Connection monitor is the same trip, tried every minute, with a notebook of every result - so you can see exactly when the road was closed, and get an alert when it happens.

Network Watcher and Connection monitor

The networking lessons used Network Watcher's tools one at a time to answer one question at a time. This lesson looks at Network Watcher as a toolbox - what each tool answers, so you reach for the right one - and at Connection monitor, which turns a one-off connectivity check into a test that runs every minute and writes its results to Log Analytics, so that "is the app able to reach its database?" becomes a graph and an alert instead of a 2 am SSH session.

Need to know: Network Watcher is a regional service, enabled automatically per region (one NetworkWatcher_<region> resource in NetworkWatcherRG). Its diagnostic tools are IP flow verify (would an NSG allow this 5-tuple, and which rule decides), NSG diagnostics / effective security rules, Next hop (where does the route table send this packet), Connection troubleshoot (test-connectivity: an actual probe from a VM, hop by hop, with the issue found), packet capture, VPN troubleshoot, and flow logs (VNet flow logs - NSG flow logs can no longer be created - with Traffic analytics on top). Connection monitor runs tests from source endpoints (VMs with the Network Watcher agent extension, on-premises machines) to destinations over TCP, HTTP or ICMP on a schedule, with thresholds, and stores results in the NWConnectionMonitorTestResult table.

The toolbox

toolquestion it answerscommand
IP flow verifydoes an NSG allow this flow to/from this VM, and which rule?az network watcher test-ip-flow
Effective security rulesevery NSG rule that applies to this NIC, mergedaz network nic list-effective-nsg
Next hopwhere does a packet from this VM to that IP go?az network watcher show-next-hop
Effective routesthe whole route table of this NICaz network nic show-effective-route-table
Connection troubleshootcan this VM actually connect to that host:port right now?az network watcher test-connectivity
Packet capturewhat is on the wire?az network watcher packet-capture create (portal)
Flow logs + Traffic analyticswhich flows happened, allowed or denied, over time?az network watcher flow-log create --vnet
Connection monitoris this path healthy every minute, and since when not?az network watcher connection-monitor create

The first five are point-in-time. IP flow verify and next hop only read configuration (they work even when the VMs are idle); connection troubleshoot sends real probes from the VM (through the Network Watcher agent extension) and reports per hop.

$ az network watcher list --query "[].{name:name, location:location, group:resourceGroup}" -o table
Name                       Location    Group
-------------------------  ----------  ----------------
NetworkWatcher_westeurope  westeurope  NetworkWatcherRG
$ az network watcher test-ip-flow -g rg-az5-net2 --vm vm-app --direction Outbound --protocol TCP --local 10.91.1.4:50000 --remote 10.92.1.4:5432 -o table
Access  RuleName
------  --------------------------------------
Allow   defaultSecurityRules/AllowVnetOutBound
$ az network watcher show-next-hop -g rg-az5-net2 --vm vm-app --source-ip 10.91.1.4 --dest-ip 10.92.1.4 -o table
NextHopIpAddress  NextHopType  RouteTableId
----------------  -----------  ------------
                  VNetPeering  System Route
$ az network watcher test-connectivity -g rg-az5-net2 --source-resource vm-app --dest-address 10.92.1.4 --dest-port 5432 --query "{status:connectionStatus, avgLatencyMs:avgLatencyInMs, hops:hops[].{type:type, address:address, issues:length(issues)}}" -o json
{
  "avgLatencyMs": 5,
  "hops": [
    {
      "address": "10.91.1.4",
      "issues": 0,
      "type": "Source"
    },
    {
      "address": "10.92.1.4",
      "issues": 0,
      "type": "VirtualMachine"
    }
  ],
  "status": "Reachable"
}

In this lab the spokes are directly peered, so everything is green: the outbound flow is allowed, the next hop is the peering, the probes get through with a latency.

Connection monitor: the same check, every minute

A connectivity check you run once tells you about now. A connection monitor keeps running it and records every result, so you see when it started failing and can alert on it:

$ APP=$(az vm show -g rg-az5-net2 -n vm-app --query id -o tsv); DB=$(az vm show -g rg-az5-net2 -n vm-db --query id -o tsv)
$ az network watcher connection-monitor create -n cm-app-to-db -l westeurope --endpoint-source-name vm-app --endpoint-source-resource-id $APP --endpoint-dest-name vm-db --endpoint-dest-resource-id $DB --test-config-name pg-5432 --protocol Tcp --tcp-port 5432 --test-group-name app-db --workspace-ids $(az monitor log-analytics workspace show -g rg-oncall-lab -n law-sysop --query id -o tsv) --query "{name:name, status:monitoringStatus, tests:testGroups[0].name}" -o json
{
  "name": "cm-app-to-db",
  "status": "Running",
  "tests": "app-db"
}
$ az network watcher connection-monitor list -l westeurope --query "[].{name:name, status:monitoringStatus}" -o table
Name          Status
------------  -------
cm-legacy     Running
cm-app-to-db  Running

The pieces of a connection monitor:

  • endpoints - sources (Azure VMs or scale sets with the Network Watcher agent, on-premises machines with the Azure Monitor agent) and destinations (VMs, addresses, URLs, Azure resources);
  • test configurations - protocol (TCP port, HTTP with method/path/expected status codes, ICMP), the test frequency (default 30 seconds) and thresholds (checks failed %, round-trip time) that decide pass or fail;
  • test groups - which sources test which destinations with which configurations;
  • the workspace that stores the results (NWConnectionMonitorTestResult, NWConnectionMonitorPathResult).

Five minutes later there is history. Then someone tightens the database subnet:

$ sleep 300
$ WS=$(az monitor log-analytics workspace show -g rg-oncall-lab -n law-sysop --query customerId -o tsv)
$ az monitor log-analytics query -w $WS --analytics-query "NWConnectionMonitorTestResult | where TestGroupName == 'app-db' | summarize checks=count() by TestResult" -o table
TableName      TestResult  Checks
-------------  ----------  ------
PrimaryResult  Pass        5
$ az network nsg rule create -g rg-az5-net2 --nsg-name vm-dbNSG -n deny-app-subnet --priority 300 --source-address-prefixes 10.91.1.0/24 --destination-port-ranges 5432 --protocol Tcp --access Deny -o none
$ sleep 300
$ az monitor log-analytics query -w $WS --analytics-query "NWConnectionMonitorTestResult | where TestGroupName == 'app-db' | summarize first_fail=minif(TimeGenerated, TestResult == 'Fail'), last_pass=maxif(TimeGenerated, TestResult == 'Pass'), fails=countif(TestResult == 'Fail')" -o table
TableName      Fails  First_fail                Last_pass
-------------  -----  ------------------------  ------------------------
PrimaryResult  4      2026-09-22T20:06:18.200Z  2026-09-22T20:05:18.200Z

first_fail is the minute the NSG change landed - which the activity log can then attribute to a person or a pipeline (az monitor activity-log list --offset 15m). A log search alert on NWConnectionMonitorTestResult (or the built-in connection monitor metrics, ChecksFailedPercent and RoundTripTimeMs) pages someone the first time the path breaks, instead of when the users notice.

Flow logs and Traffic analytics

To see which flows happened - who talked to whom, allowed or denied - enable VNet flow logs (storage account for the raw logs, optionally Traffic analytics into a workspace for the aggregated view):

$ az network watcher flow-log create -g rg-az5-net2 -n fl-nsg -l westeurope --nsg vm-dbNSG --storage-account stflowaz5
ERROR: (simulator) NSG flow logs can no longer be created: creation stopped on 30 June 2025 and NSG flow logs retire on 30 September 2027. Create a virtual network flow log instead: --vnet <name> (it covers every NIC in the VNet, and records whether the traffic was allowed or denied by the NSGs).
$ az network watcher flow-log create -g rg-az5-net2 -n fl-vnet-data -l westeurope --vnet vnet-data --storage-account stflowaz5 --traffic-analytics true --workspace $(az monitor log-analytics workspace show -g rg-oncall-lab -n law-sysop --query id -o tsv) --query "{name:name, target:targetResourceId, enabled:enabled}" -o json
{
  "enabled": true,
  "name": "fl-vnet-data",
  "target": "/subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-az5-net2/providers/Microsoft.Network/virtualNetworks/vnet-data"
}

NSG flow logs stopped being creatable on 30 June 2025 and retire on 30 September 2027; VNet flow logs cover every NIC in the VNet (or a subnet or a NIC), record flows whether or not an NSG is attached, and include the encryption state. They are not real time (minutes of delay) and cost per GB collected.

On the exam (AZ-104 5.1.6): "which NSG rule blocks this traffic" = IP flow verify (or effective security rules); "why does traffic go to the appliance" = next hop; "test reachability from a VM to a host and port, see the failing hop" = connection troubleshoot; "monitor connectivity between an on-premises server and an Azure VM continuously, with latency" = Connection monitor; "capture packets on a VM" = packet capture; "analyse flows across a VNet" = VNet flow logs + Traffic analytics.

In an interview: "How would you know a network path broke before users tell you?" - "A connection monitor test from the app's VMs to each dependency - database port, the payment API's URL - writing to Log Analytics, with an alert on failed checks or latency. When it fires, the results show the first failing minute, and the activity log shows what changed then."

Why it helps

Most network incidents are "it worked yesterday". A one-off test tells you it is broken now; a connection monitor tells you when it broke, which points straight at the change in the activity log that broke it.

AZ-104 asks which Network Watcher tool answers which question, and to configure a connection monitor. This chapter's last lab uses exactly that path: a monitor that turned red, the minute it turned red, and the NSG rule a pipeline added at that minute.

Commands in this lesson

az sleep

FAQ

Do I have to enable Network Watcher?

It is enabled automatically per region when you create a virtual network there (a NetworkWatcher_<region> resource in NetworkWatcherRG). Some tools need more: connection monitor and connection troubleshoot use the Network Watcher agent extension on VMs.

IP flow verify or effective security rules?

IP flow verify answers one flow: allowed or denied, and by which rule. Effective security rules list every rule that applies to a NIC, subnet and NIC NSGs combined, with service tags expanded. Use the first to answer a question, the second to understand the whole picture.

What does next hop tell me?

Where the route table sends traffic from a VM to an IP: Internet, VirtualNetwork, VNetPeering, VirtualAppliance (with its IP), VirtualNetworkGateway or None (dropped). It is the quickest way to see that a UDR sends traffic to a firewall, or nowhere.

What does a connection monitor measure?

For each test group (sources, destinations, a test configuration such as TCP 5432 every 60 seconds), the checks failed percentage and round-trip time. Results go to a Log Analytics workspace, and alerts can fire on them. It shows the path's history, not only the current state.

What are flow logs for?

Recording every flow through a VNet (VNet flow logs, replacing NSG flow logs) to a storage account, with traffic analytics summarising them in a workspace. Use them for "who talked to what" questions and security reviews, not for a quick reachability check.

In an interview Mid

How would you know a network path broke before users tell you?

A connection monitor test from the app's VMs to each dependency - the database port, the payment API's URL - writing to Log Analytics, with an alert on failed checks or latency. When it fires, the results show the first failing minute, and the activity log shows what changed then. For a one-off check, IP flow verify and next hop answer which rule or route is responsible.

Also asked: Which Network Watcher tool tells you which NSG rule blocks a connection? · How would you check where a VM's traffic to an IP address is routed? · What is the difference between flow logs and a connection monitor?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.