Availability sets, zones, scale sets and autoscale
One VM is one failure away from an outage: the host it runs on dies, the rack loses power, Azure patches the host and reboots it, the whole datacenter has a bad day. Azure gives you two ways to spread VMs so that one of those events never takes all of them - availability sets inside a datacenter, availability zones across datacenters - and one way to stop managing VMs one by one: virtual machine scale sets, which an autoscale setting grows and shrinks with the load. This lesson is how to choose between them, and how autoscale decides; the incident later in this chapter is what happens when an autoscale setting is wrong.
Need to know: an availability set spreads VMs across fault domains (racks: power and network, up to 3) and update domains (groups rebooted together during planned maintenance, up to 20, default 5); VM SLA 99.95%. Availability zones are physically separate datacenters in a region (1, 2, 3); VMs in two or more zones: SLA 99.99%. A VM is in one or the other, never both, and the choice is made at creation. A scale set runs identical VMs from one model (Uniform) or VMs you can manage individually (Flexible, the default for new scale sets); its upgrade policy (Manual, Automatic, Rolling) decides when instances pick up a model change. Autoscale evaluates rules on a metric over a time window, scales by a step within min/max, then waits a cooldown.
Availability sets: one datacenter, many racks
$ cd ~/oncall-lab/labs/4b-azure/az5/ha
$ az vm availability-set create -g rg-az5-ha -n as-reports --platform-fault-domain-count 2 --platform-update-domain-count 5 -o table
Location Name PlatformFaultDomainCount PlatformUpdateDomainCount ResourceGroup
---------- ---------- ------------------------ ------------------------- -------------
westeurope as-reports 2 5 rg-az5-ha
$ az vm create -g rg-az5-ha -n vm-rep1 --image Ubuntu2404 --size Standard_B1s --availability-set as-reports --public-ip-address "" --nsg "" --generate-ssh-keys -o none
$ az vm create -g rg-az5-ha -n vm-rep2 --image Ubuntu2404 --size Standard_B1s --availability-set as-reports --public-ip-address "" --nsg "" --generate-ssh-keys -o none
$ for v in vm-rep1 vm-rep2; do az vm get-instance-view -g rg-az5-ha -n $v --query "{vm:name, faultDomain:instanceView.platformFaultDomain, updateDomain:instanceView.platformUpdateDomain}" -o tsv; done
vm-rep1 0 0
vm-rep2 1 1
The two VMs landed in different fault domains (0 and 1: different racks, power and top-of-rack switch) and different update domains (Azure reboots one update domain at a time during maintenance and waits 30 minutes before the next). What an availability set does not protect against: losing the datacenter. Rules:
- the availability set is chosen when the VM is created; an existing VM cannot be added to one (redeploy it from its disks);
- a VM cannot be in an availability set and a zone;
- all VMs of a set use managed disks in Aligned sets (the default), so the disks follow the fault domains too;
- you need at least two VMs in the set for the 99.95% SLA - one VM in a set is still one VM.
$ az vm create -g rg-az5-ha -n vm-rep3 --image Ubuntu2404 --size Standard_B1s --availability-set as-reports --zone 1 -o none
ERROR: usage error: --zone and --availability-set are mutually exclusive
Availability zones: separate datacenters
A region with zones has three or more datacenters with independent power, cooling and networking, a few kilometres apart, with low-latency links between them. Resources are either:
- zonal - pinned to one zone (
--zone 1): a VM, a managed disk, a zonal public IP; - zone-redundant - spread over all zones by Azure: a Standard load balancer frontend, a Standard public IP created without a zone, ZRS storage and disks, zone-redundant App Service plans.
Two VMs in zones 1 and 2 survive a datacenter failure; the price is a little latency between zones and inter-zone data transfer charges. Remember the restrictions from the VM lesson: a size can be missing from a zone (az vm list-skus shows zones and restrictions), and a zonal disk only attaches to a VM in its own zone.
| availability set | availability zones | |
|---|---|---|
| protects against | rack, host and planned-maintenance failures | a whole datacenter |
| SLA (2+ VMs) | 99.95% | 99.99% |
| chosen | at VM creation | at VM creation |
| latency between VMs | lowest (same datacenter) | low (< 2 ms round trip) |
| extra cost | none | inter-zone traffic |
Scale sets: many VMs from one model
A virtual machine scale set creates and manages a group of VMs from one definition (image, size, network, extensions, custom data). You change the definition once and the instances follow; you change the capacity and Azure adds or removes VMs.
$ az vmss create -g rg-az5-ha -n vmss-api --image Ubuntu2404 --vm-sku Standard_B1s --instance-count 2 --zones 1 2 3 --orchestration-mode Uniform --upgrade-policy-mode Manual --admin-username azureuser --generate-ssh-keys --custom-data cloud-init.txt -o none
$ az vmss show -g rg-az5-ha -n vmss-api --query "{capacity:sku.capacity, size:sku.name, zones:zones, mode:orchestrationMode, upgrade:upgradePolicy.mode}" -o json
{
"capacity": 2,
"mode": "Uniform",
"size": "Standard_B1s",
"upgrade": "Manual",
"zones": [
"1",
"2",
"3"
]
}
$ az vmss list-instances -g rg-az5-ha -n vmss-api --query "[].{instance:instanceId, name:name, zone:zones[0], latest:latestModelApplied}" -o table
Instance Name Zone Latest
-------- ---------- ---- ------
0 vmss-api_0 1 True
1 vmss-api_1 2 True
$ az network lb list -g rg-az5-ha --query "[].name" -o tsv
vmss-apiLB
az vmss create also made a VNet, a Standard load balancer in front of the instances (skip it with --lb "" when an Application Gateway or nothing should be in front) and spread the instances across the zones you named. Two choices to understand:
Orchestration mode. Uniform - identical instances addressed by instance id (0, 1 ...), managed only through the scale set: the original model, still what most autoscaled stateless tiers and AKS node pools use. Flexible - each instance is a normal VM resource (az vm commands work on it), you can mix sizes, add existing VMs, and it is the only mode that also spreads across fault domains like an availability set. Flexible is now the default for new scale sets; the mode cannot change after creation.
Upgrade policy (Uniform). After az vmss update changes the model (a new size, image or custom data):
- Manual - nothing happens to existing instances;
latestModelAppliedturnsFalseuntil you runaz vmss update-instances --instance-ids ...(or reimage them); - Automatic - every instance is updated at once, in no particular order: an outage for anything that is not stateless behind a retrying client;
- Rolling - in batches (
maxBatchInstancePercent, 20% by default), waiting for health between batches; needs a health probe or the Application Health extension.
$ az vmss update -g rg-az5-ha -n vmss-api --set virtualMachineProfile.storageProfile.osDisk.diskSizeGb=64 -o none
$ az vmss list-instances -g rg-az5-ha -n vmss-api --query "[].{instance:instanceId, latest:latestModelApplied}" -o table
Instance Latest
-------- ------
0 False
1 False
$ az vmss update-instances -g rg-az5-ha -n vmss-api --instance-ids 0
$ az vmss list-instances -g rg-az5-ha -n vmss-api --query "[].{instance:instanceId, latest:latestModelApplied}" -o table
Instance Latest
-------- ------
0 True
1 False
Updating instance 0 first and watching it is a manual canary. Scaling by hand:
$ az vmss scale -g rg-az5-ha -n vmss-api --new-capacity 3 --query sku.capacity
3
Autoscale: rules, limits and cooldowns
An autoscale setting (Microsoft.Insights/autoscalesettings, one per target) is attached to a scale set or an App Service plan. It holds one or more profiles; each profile has a minimum, maximum and default instance count and a list of rules:
$ az monitor autoscale create -g rg-az5-ha --resource vmss-api --resource-type Microsoft.Compute/virtualMachineScaleSets -n as-api --min-count 2 --max-count 4 --count 2 -o none
$ az monitor autoscale rule create -g rg-az5-ha --autoscale-name as-api --condition "Percentage CPU > 70 avg 5m" --scale out 1 -o none
$ az monitor autoscale rule create -g rg-az5-ha --autoscale-name as-api --condition "Percentage CPU < 25 avg 10m" --scale in 1 -o none
$ az monitor autoscale rule list -g rg-az5-ha --autoscale-name as-api -o table
ID Metric Operator Threshold Aggregation Time Window Direction Value Cooldown
-- -------------- ----------- --------- ----------- ----------- --------- ----- --------
0 Percentage CPU GreaterThan 70 Average PT5M Increase 1 PT5M
1 Percentage CPU LessThan 25 Average PT10M Decrease 1 PT5M
$ az monitor autoscale show -g rg-az5-ha -n as-api --query "profiles[0].capacity" -o json
{
"default": "2",
"maximum": "4",
"minimum": "2"
}
How autoscale reads a rule: every minute it takes the metric (Percentage CPU, averaged across all instances) over the time window (5m), compares it with the threshold, and if it fires, changes the count by the step (out 1, in 1, or a percentage, or to N), never below the minimum or above the maximum. After any scale action it waits the cooldown (5 minutes by default) before it acts again. The behaviour that matters in an incident:
- Scale out happens if any scale-out rule fires; scale in only if all scale-in rules agree. Out fast, in slowly: a short window and a big step out, a long window and a small step in.
- Thresholds must leave room: out at 70%, in at 25% - not 60/50. With a scale-in rule that would push the average straight back over the scale-out threshold, autoscale detects the flapping and skips the scale-in.
--count(default) is used when the metric is missing. The minimum is the floor for everything · set it to 0 and autoscale will happily take a quiet service down to zero instances at night.- A profile can be recurring (weekdays 08:00-18:00 min 4) or fixed date (Black Friday min 10); outside them the default profile applies.
- Every scale action writes to the activity log (
Microsoft.Insights/AutoscaleSettings/Scaleup/ActionandScaledown), and the setting can send notifications by email or webhook.
App Service plans autoscale the same way (--resource-type Microsoft.Web/serverfarms, metric CpuPercentage), within the instance limit of their tier - the App Service part of this chapter.
On the exam (AZ-104 3.2.6, 3.2.7): "protect against a datacenter failure" = zones; "rack failure, no zones in the region" = availability set; "add an existing VM to an availability set" = not possible, recreate it. Autoscale questions are arithmetic: start count, rule, cooldown, min/max
- work out the count minute by minute. "Instances still run the old image after the update" =
upgrade policy Manual.
In an interview: "How would you make a VM-based service survive a datacenter outage?" - "Run at least two instances across availability zones, behind a zone-redundant Standard load balancer, with zone-redundant or replicated data. For more than a couple of VMs, a scale set across zones with autoscale, a health probe and rolling upgrades."