OnCallReadyStart learning

Azure Lesson 33.32 · Azure V: Compute, App Hosting & the AZ-104 exam · 22 min read

Azure DNS and the Standard load balancer

In plain words

Azure DNS is the phone book for your domain: it says which address www.example.com has. Creating the book in Azure is not enough - the registrar must be told that Azure keeps the book now (the NS records).

A load balancer is a receptionist with one public number who passes each caller to one of several identical desks, but only to desks that answer the regular "are you there?" check (the health probe).

Azure DNS and the Standard load balancer

Two pieces of plumbing sit in front of almost every service: the name people type and the load balancer that spreads their connections over healthy servers. When either is wrong, the service is down even though every VM is fine. This lesson hosts a public domain in Azure DNS, then builds a Standard load balancer - public and internal - and breaks it the three ways it breaks in real life: a probe that asks the wrong question, an NSG that drops the probe, and a backend nobody added to the pool.

Need to know: an Azure DNS zone answers for a domain once the registrar delegates it to the zone's four name servers (NS records). Records live in record sets (A, AAAA, CNAME, MX, TXT, NS, SRV, CAA, PTR); a CNAME cannot sit at the apex - use an alias record for Azure resources there. A Standard load balancer (Basic is retired) has a frontend (public IP, or a private IP for an internal LB), a backend pool (NIC IP configurations), health probes (TCP, HTTP, HTTPS) sent from 168.63.129.16 (the AzureLoadBalancer tag) and load-balancing rules joining them. Only healthy backends get new flows; with none healthy, connections time out. DipAvailability is the probe health metric, VipAvailability the data path's.

A public DNS zone

$ az network dns zone create -g rg-az5-lb -n az5-lab.example --query "{zone:name, ns:nameServers}" -o json
{
  "ns": [
    "ns1-03.azure-dns.com.",
    "ns2-03.azure-dns.net.",
    "ns3-03.azure-dns.org.",
    "ns4-03.azure-dns.info."
  ],
  "zone": "az5-lab.example"
}

Creating the zone does not make the internet ask it. At the registrar (or the parent zone) you replace the domain's NS records with these four name servers - delegation - and from then on every resolver on the internet ends up at Azure DNS for az5-lab.example. Forget it and records you create here are answered by nobody (the lab delegates *.example zones automatically, so dig works at once). A child zone (dev.az5-lab.example) is delegated the same way, with NS records in the parent zone.

Records come in record sets: all the records of one name and type, with one TTL:

$ az network dns record-set a add-record -g rg-az5-lb -z az5-lab.example -n www -a 203.0.113.10 -o none
$ az network dns record-set a add-record -g rg-az5-lb -z az5-lab.example -n www -a 203.0.113.11 -o none
$ az network dns record-set cname set-record -g rg-az5-lb -z az5-lab.example -n shop -c www.az5-lab.example -o none
$ az network dns record-set mx add-record -g rg-az5-lb -z az5-lab.example -n @ -e mail.az5-lab.example -p 10 -o none
$ az network dns record-set txt add-record -g rg-az5-lb -z az5-lab.example -n @ -v "v=spf1 -all" -o none
$ az network dns record-set list -g rg-az5-lb -z az5-lab.example --query "[].{name:name, kind:type, ttl:TTL}" -o table
Name  Kind                              Ttl
----  --------------------------------  ------
@     Microsoft.Network/dnszones/NS     172800
@     Microsoft.Network/dnszones/SOA    3600
www   Microsoft.Network/dnszones/A      3600
shop  Microsoft.Network/dnszones/CNAME  3600
@     Microsoft.Network/dnszones/MX     3600
@     Microsoft.Network/dnszones/TXT    3600
$ dig +short www.az5-lab.example
203.0.113.10
203.0.113.11
$ dig +short shop.az5-lab.example
www.az5-lab.example.
203.0.113.10
203.0.113.11

@ is the apex (the zone name itself). A record set's TTL (3600 s by default) is how long resolvers may cache it - lower it a day before a migration, not during. Rules the exam checks:

  • a CNAME must be the only record at its name, so never at the apex (@ already has SOA and NS);
  • set-record for CNAME (one value), add-record for the multi-value types;
  • an alias record set (A/AAAA/CNAME pointing at an Azure resource: a public IP, Traffic Manager, Front Door, a CDN endpoint) follows the resource's IP when it changes and can sit at the apex;
  • private DNS zones (Azure II) answer only inside linked VNets and can auto-register VM names; public zones answer the internet.

A public Standard load balancer

Two web VMs in two zones (vm-web1, vm-web2, nginx on port 80) sit behind nothing yet. The pieces:

$ az network lb create -g rg-az5-lb -n lb-web --sku Standard --public-ip-address pip-lb-web --frontend-ip-name fe-web --backend-pool-name be-web -o none
$ az network lb probe create -g rg-az5-lb --lb-name lb-web -n hp-web --protocol Http --port 80 --path /healthz -o none
$ az network lb rule create -g rg-az5-lb --lb-name lb-web -n http --protocol Tcp --frontend-port 80 --backend-port 80 --frontend-ip-name fe-web --backend-pool-name be-web --probe-name hp-web --idle-timeout 15 --enable-tcp-reset true -o none
$ for v in vm-web1 vm-web2; do az network nic ip-config address-pool add -g rg-az5-lb --nic-name ${v}VMNic --ip-config-name ipconfig$v --lb-name lb-web --address-pool be-web -o none; done
$ az network lb show -g rg-az5-lb -n lb-web --query "{sku:sku.name, frontend:frontendIPConfigurations[0].name, pool:backendAddressPools[0].backendIPConfigurations[].id, rule:loadBalancingRules[0].{fe:frontendPort, be:backendPort, idle:idleTimeoutInMinutes}}" -o json
{
  "frontend": "fe-web",
  "pool": [
    "/subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-az5-lb/providers/Microsoft.Network/networkInterfaces/vm-web1VMNic/ipConfigurations/ipconfigvm-web1",
    "/subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-az5-lb/providers/Microsoft.Network/networkInterfaces/vm-web2VMNic/ipConfigurations/ipconfigvm-web2"
  ],
  "rule": {
    "be": 80,
    "fe": 80,
    "idle": 15
  },
  "sku": "Standard"
}
  • Frontend: the public IP (Standard, zone-redundant) clients connect to.
  • Backend pool: NIC IP configurations (or IP addresses in the VNet). VMs in the pool need no public IP; with no outbound rule or NAT gateway they also get no internet access out (Standard LB is closed by default).
  • Health probe: every 5 seconds by default; an HTTP probe needs a 200; the probe comes from 168.63.129.16.
  • Rule: frontend port -> backend port, with a probe. Distribution is a 5-tuple hash (source IP and port, destination IP and port, protocol); --load-distribution SourceIP gives session persistence. Idle timeout (4-100 minutes) and TCP reset decide what happens to quiet connections - the cause of many "connection reset by peer" mysteries.
$ LBIP=$(az network public-ip show -g rg-az5-lb -n pip-lb-web --query ipAddress -o tsv)
$ curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://$LBIP/
000
$ az monitor metrics list --resource $(az network lb show -g rg-az5-lb -n lb-web --query id -o tsv) --metric DipAvailability --offset 5m --interval PT1M --query "value[0].timeseries[0].data[-1].average" -o tsv
0

000 and a health probe status of 0: both backends are marked down. The probe asks for /healthz, and nginx on these VMs has no such page - it answers 404, and an HTTP probe accepts only

  1. Point the probe at a path that exists (or deploy a real health endpoint):
$ az network lb probe update -g rg-az5-lb --lb-name lb-web -n hp-web --path / -o none
$ curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://$LBIP/
200
$ az monitor metrics list --resource $(az network lb show -g rg-az5-lb -n lb-web --query id -o tsv) --metric DipAvailability --offset 5m --interval PT1M --query "value[0].timeseries[0].data[-1].average" -o tsv
100

An internal load balancer

An internal LB has a private frontend in a subnet: for the tier behind the web servers, or for an appliance pair. Same pieces, no public IP:

$ az network lb create -g rg-az5-lb -n lb-api --sku Standard --vnet-name vnet-lb --subnet snet-web --private-ip-address 10.97.1.100 --frontend-ip-name fe-api --backend-pool-name be-api -o none
$ az network lb frontend-ip list -g rg-az5-lb --lb-name lb-api --query "[].{name:name, privateIp:privateIPAddress, allocation:privateIPAllocationMethod}" -o table
Name    PrivateIp    Allocation
------  -----------  ----------
fe-api  10.97.1.100  Static

A static frontend IP (--private-ip-address) is what clients and DNS records will point at. An internal Standard LB also offers HA ports (a rule with protocol All and port 0: every port, every protocol) - how network virtual appliances are made redundant. Clients in peered VNets can reach an internal LB; on-premises clients too, over VPN/ExpressRoute.

Troubleshooting a load balancer

Symptoms from the client side: connections time out (no healthy backend, or the frontend port has no rule), or some requests fail (one backend unhealthy, or not in the pool). Work through it in order:

checkhowwhat it means
health probe statusmetric DipAvailability per backend IP (dimension BackendIPAddress)0 = the probe fails on that backend
data pathmetric VipAvailability0 = the frontend itself does not deliver
is the probe allowed?az network watcher test-ip-flow from 168.63.129.16 to the probe portan NSG must allow the AzureLoadBalancer tag
does the probe's question have a good answer?curl the probe path on the VM (Bastion, run-command)HTTP probes need 200, not 301 or 404
is the VM in the pool?az network lb show ... backendAddressPools[].backendIPConfigurationsa new VM nobody added gets no traffic
rule and portsaz network lb rule listfrontend port with no rule = timeout

The second classic, an NSG that blocks the probe. Someone saw an unknown address, 168.63.129.16, "hitting port 80 every five seconds" in the nginx logs and blocked it:

$ az network nsg rule create -g rg-az5-lb --nsg-name nsg-web -n deny-scanner --priority 150 --source-address-prefixes 168.63.129.16/32 --destination-port-ranges 80 --protocol Tcp --access Deny -o none
$ az network watcher test-ip-flow -g rg-az5-lb --vm vm-web1 --direction Inbound --protocol TCP --local 10.97.1.4:80 --remote 168.63.129.16:51000 -o table
Access  RuleName
------  --------------------------
Deny    securityRules/deny-scanner
$ curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://$LBIP/
000
$ az network nsg rule delete -g rg-az5-lb --nsg-name nsg-web -n deny-scanner
$ curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://$LBIP/
200

That "scanner" was the load balancer's health probe. With it denied at 150 - above the default AllowAzureLoadBalancerInBound (65001) - every probe failed, both backends were marked down and the whole site timed out, although the VMs, nginx and the client rules were fine. 168.63.129.16 is Azure's platform address (probes, DHCP, DNS, the VM agent's wire server): never block it. If a subnet must deny broadly, put an explicit allow for the AzureLoadBalancer tag above the deny.

On the exam (AZ-104 4.3.1-4.3.3): "the domain does not resolve after creating the zone" = update the NS records at the registrar; "point the apex at a public IP that may change" = alias record; "load balance HTTP with URL paths or TLS offload" = Application Gateway, not a load balancer; "private frontend" = internal LB; "backends unhealthy although the app runs" = the probe (path, port, protocol) or an NSG blocking 168.63.129.16; "only some VMs get traffic" = check the backend pool and per-backend health probe status.

In an interview: "Users time out on a load-balanced service but every VM looks fine. What do you check?" - "Health probe status per backend: if they're all down, the probe is failing - wrong path or port, a 301 instead of a 200, or an NSG that blocks 168.63.129.16. Then whether the VMs are actually in the backend pool and the frontend port has a rule."

Why it helps

Name resolution and load balancing sit in front of every service, so their failures look like outages of the service itself: a domain that does not resolve because nobody updated the registrar, a CNAME at the apex, a probe that gets a redirect instead of a 200, an NSG that blocks the probe's source.

AZ-104 asks you to configure both and to troubleshoot the load balancer; the lesson does both with real metrics.

Commands in this lesson

az dig curl

FAQ

Why does my new zone not resolve?

The registrar (or the parent zone) still points at the old name servers. Each Azure DNS zone gets four name servers (az network dns zone show --query nameServers); put those in the NS records at the registrar. Old answers can stay cached until their TTL expires.

Why can't I put a CNAME at the apex?

DNS forbids a CNAME next to other records with the same name, and the apex (@) always has SOA and NS records. In Azure DNS, use an alias record set: an A record at the apex that points to an Azure resource such as a public IP, Front Door or Traffic Manager.

Where do health probes come from?

From 168.63.129.16, the AzureLoadBalancer service tag. An NSG must allow it (the default AllowAzureLoadBalancerInBound does, unless a custom deny comes first). An HTTP probe needs a 200 from its path; a redirect or 404 marks the backend down.

What happens when no backend is healthy?

The Standard load balancer stops sending new connections, and clients time out - they are not refused. That is why "everything times out but every VM is running" usually means a probe problem, not an application problem.

Public or internal load balancer?

A public load balancer has a public IP frontend and balances internet traffic. An internal one has a private IP frontend in a subnet and balances traffic inside the VNet or from peered and on-premises networks - an app tier in front of a database tier, for example.

In an interview Mid

Users time out on a load-balanced service but every VM looks fine. What do you check?

Health probe status per backend: if they are all down, the probe is failing - wrong path or port, a 301 instead of a 200, or an NSG that blocks 168.63.129.16 (the AzureLoadBalancer tag). Then whether the VMs are actually in the backend pool, and whether the frontend port has a rule. DipAvailability shows the probe health over time.

Also asked: How do you delegate a domain to Azure DNS? · How do you point the apex of a domain at an Azure public IP? · What is the difference between a public and an internal load balancer?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.