OnCallReadyStart learning

Azure Lesson 33.11 · Azure V: Compute, App Hosting & the AZ-104 exam · 22 min read

Availability sets, zones, scale sets and autoscale

In plain words

If you keep all your eggs in one basket, dropping the basket breaks them all. An availability set puts your VMs in different baskets inside one building: different racks with their own power and network. Availability zones put them in different buildings in the same city.

A scale set is a recipe that bakes as many identical VMs as you need. Autoscale watches how busy they are and bakes more when it is busy and fewer when it is quiet - within the minimum and maximum you allow.

Availability sets, zones, scale sets and autoscale

One VM is one failure away from an outage: the host it runs on dies, the rack loses power, Azure patches the host and reboots it, the whole datacenter has a bad day. Azure gives you two ways to spread VMs so that one of those events never takes all of them - availability sets inside a datacenter, availability zones across datacenters - and one way to stop managing VMs one by one: virtual machine scale sets, which an autoscale setting grows and shrinks with the load. This lesson is how to choose between them, and how autoscale decides; the incident later in this chapter is what happens when an autoscale setting is wrong.

Need to know: an availability set spreads VMs across fault domains (racks: power and network, up to 3) and update domains (groups rebooted together during planned maintenance, up to 20, default 5); VM SLA 99.95%. Availability zones are physically separate datacenters in a region (1, 2, 3); VMs in two or more zones: SLA 99.99%. A VM is in one or the other, never both, and the choice is made at creation. A scale set runs identical VMs from one model (Uniform) or VMs you can manage individually (Flexible, the default for new scale sets); its upgrade policy (Manual, Automatic, Rolling) decides when instances pick up a model change. Autoscale evaluates rules on a metric over a time window, scales by a step within min/max, then waits a cooldown.

Availability sets: one datacenter, many racks

$ cd ~/oncall-lab/labs/4b-azure/az5/ha
$ az vm availability-set create -g rg-az5-ha -n as-reports --platform-fault-domain-count 2 --platform-update-domain-count 5 -o table
Location    Name        PlatformFaultDomainCount  PlatformUpdateDomainCount  ResourceGroup
----------  ----------  ------------------------  -------------------------  -------------
westeurope  as-reports  2                         5                          rg-az5-ha
$ az vm create -g rg-az5-ha -n vm-rep1 --image Ubuntu2404 --size Standard_B1s --availability-set as-reports --public-ip-address "" --nsg "" --generate-ssh-keys -o none
$ az vm create -g rg-az5-ha -n vm-rep2 --image Ubuntu2404 --size Standard_B1s --availability-set as-reports --public-ip-address "" --nsg "" --generate-ssh-keys -o none
$ for v in vm-rep1 vm-rep2; do az vm get-instance-view -g rg-az5-ha -n $v --query "{vm:name, faultDomain:instanceView.platformFaultDomain, updateDomain:instanceView.platformUpdateDomain}" -o tsv; done
vm-rep1	0	0
vm-rep2	1	1

The two VMs landed in different fault domains (0 and 1: different racks, power and top-of-rack switch) and different update domains (Azure reboots one update domain at a time during maintenance and waits 30 minutes before the next). What an availability set does not protect against: losing the datacenter. Rules:

  • the availability set is chosen when the VM is created; an existing VM cannot be added to one (redeploy it from its disks);
  • a VM cannot be in an availability set and a zone;
  • all VMs of a set use managed disks in Aligned sets (the default), so the disks follow the fault domains too;
  • you need at least two VMs in the set for the 99.95% SLA - one VM in a set is still one VM.
$ az vm create -g rg-az5-ha -n vm-rep3 --image Ubuntu2404 --size Standard_B1s --availability-set as-reports --zone 1 -o none
ERROR: usage error: --zone and --availability-set are mutually exclusive

Availability zones: separate datacenters

A region with zones has three or more datacenters with independent power, cooling and networking, a few kilometres apart, with low-latency links between them. Resources are either:

  • zonal - pinned to one zone (--zone 1): a VM, a managed disk, a zonal public IP;
  • zone-redundant - spread over all zones by Azure: a Standard load balancer frontend, a Standard public IP created without a zone, ZRS storage and disks, zone-redundant App Service plans.

Two VMs in zones 1 and 2 survive a datacenter failure; the price is a little latency between zones and inter-zone data transfer charges. Remember the restrictions from the VM lesson: a size can be missing from a zone (az vm list-skus shows zones and restrictions), and a zonal disk only attaches to a VM in its own zone.

availability setavailability zones
protects againstrack, host and planned-maintenance failuresa whole datacenter
SLA (2+ VMs)99.95%99.99%
chosenat VM creationat VM creation
latency between VMslowest (same datacenter)low (< 2 ms round trip)
extra costnoneinter-zone traffic

Scale sets: many VMs from one model

A virtual machine scale set creates and manages a group of VMs from one definition (image, size, network, extensions, custom data). You change the definition once and the instances follow; you change the capacity and Azure adds or removes VMs.

$ az vmss create -g rg-az5-ha -n vmss-api --image Ubuntu2404 --vm-sku Standard_B1s --instance-count 2 --zones 1 2 3 --orchestration-mode Uniform --upgrade-policy-mode Manual --admin-username azureuser --generate-ssh-keys --custom-data cloud-init.txt -o none
$ az vmss show -g rg-az5-ha -n vmss-api --query "{capacity:sku.capacity, size:sku.name, zones:zones, mode:orchestrationMode, upgrade:upgradePolicy.mode}" -o json
{
  "capacity": 2,
  "mode": "Uniform",
  "size": "Standard_B1s",
  "upgrade": "Manual",
  "zones": [
    "1",
    "2",
    "3"
  ]
}
$ az vmss list-instances -g rg-az5-ha -n vmss-api --query "[].{instance:instanceId, name:name, zone:zones[0], latest:latestModelApplied}" -o table
Instance  Name        Zone  Latest
--------  ----------  ----  ------
0         vmss-api_0  1     True
1         vmss-api_1  2     True
$ az network lb list -g rg-az5-ha --query "[].name" -o tsv
vmss-apiLB

az vmss create also made a VNet, a Standard load balancer in front of the instances (skip it with --lb "" when an Application Gateway or nothing should be in front) and spread the instances across the zones you named. Two choices to understand:

Orchestration mode. Uniform - identical instances addressed by instance id (0, 1 ...), managed only through the scale set: the original model, still what most autoscaled stateless tiers and AKS node pools use. Flexible - each instance is a normal VM resource (az vm commands work on it), you can mix sizes, add existing VMs, and it is the only mode that also spreads across fault domains like an availability set. Flexible is now the default for new scale sets; the mode cannot change after creation.

Upgrade policy (Uniform). After az vmss update changes the model (a new size, image or custom data):

  • Manual - nothing happens to existing instances; latestModelApplied turns False until you run az vmss update-instances --instance-ids ... (or reimage them);
  • Automatic - every instance is updated at once, in no particular order: an outage for anything that is not stateless behind a retrying client;
  • Rolling - in batches (maxBatchInstancePercent, 20% by default), waiting for health between batches; needs a health probe or the Application Health extension.
$ az vmss update -g rg-az5-ha -n vmss-api --set virtualMachineProfile.storageProfile.osDisk.diskSizeGb=64 -o none
$ az vmss list-instances -g rg-az5-ha -n vmss-api --query "[].{instance:instanceId, latest:latestModelApplied}" -o table
Instance  Latest
--------  ------
0         False
1         False
$ az vmss update-instances -g rg-az5-ha -n vmss-api --instance-ids 0
$ az vmss list-instances -g rg-az5-ha -n vmss-api --query "[].{instance:instanceId, latest:latestModelApplied}" -o table
Instance  Latest
--------  ------
0         True
1         False

Updating instance 0 first and watching it is a manual canary. Scaling by hand:

$ az vmss scale -g rg-az5-ha -n vmss-api --new-capacity 3 --query sku.capacity
3

Autoscale: rules, limits and cooldowns

An autoscale setting (Microsoft.Insights/autoscalesettings, one per target) is attached to a scale set or an App Service plan. It holds one or more profiles; each profile has a minimum, maximum and default instance count and a list of rules:

$ az monitor autoscale create -g rg-az5-ha --resource vmss-api --resource-type Microsoft.Compute/virtualMachineScaleSets -n as-api --min-count 2 --max-count 4 --count 2 -o none
$ az monitor autoscale rule create -g rg-az5-ha --autoscale-name as-api --condition "Percentage CPU > 70 avg 5m" --scale out 1 -o none
$ az monitor autoscale rule create -g rg-az5-ha --autoscale-name as-api --condition "Percentage CPU < 25 avg 10m" --scale in 1 -o none
$ az monitor autoscale rule list -g rg-az5-ha --autoscale-name as-api -o table
ID  Metric          Operator     Threshold  Aggregation  Time Window  Direction  Value  Cooldown
--  --------------  -----------  ---------  -----------  -----------  ---------  -----  --------
0   Percentage CPU  GreaterThan  70         Average      PT5M         Increase   1      PT5M
1   Percentage CPU  LessThan     25         Average      PT10M        Decrease   1      PT5M
$ az monitor autoscale show -g rg-az5-ha -n as-api --query "profiles[0].capacity" -o json
{
  "default": "2",
  "maximum": "4",
  "minimum": "2"
}

How autoscale reads a rule: every minute it takes the metric (Percentage CPU, averaged across all instances) over the time window (5m), compares it with the threshold, and if it fires, changes the count by the step (out 1, in 1, or a percentage, or to N), never below the minimum or above the maximum. After any scale action it waits the cooldown (5 minutes by default) before it acts again. The behaviour that matters in an incident:

  • Scale out happens if any scale-out rule fires; scale in only if all scale-in rules agree. Out fast, in slowly: a short window and a big step out, a long window and a small step in.
  • Thresholds must leave room: out at 70%, in at 25% - not 60/50. With a scale-in rule that would push the average straight back over the scale-out threshold, autoscale detects the flapping and skips the scale-in.
  • --count (default) is used when the metric is missing. The minimum is the floor for everything · set it to 0 and autoscale will happily take a quiet service down to zero instances at night.
  • A profile can be recurring (weekdays 08:00-18:00 min 4) or fixed date (Black Friday min 10); outside them the default profile applies.
  • Every scale action writes to the activity log (Microsoft.Insights/AutoscaleSettings/Scaleup/Action and Scaledown), and the setting can send notifications by email or webhook.

App Service plans autoscale the same way (--resource-type Microsoft.Web/serverfarms, metric CpuPercentage), within the instance limit of their tier - the App Service part of this chapter.

On the exam (AZ-104 3.2.6, 3.2.7): "protect against a datacenter failure" = zones; "rack failure, no zones in the region" = availability set; "add an existing VM to an availability set" = not possible, recreate it. Autoscale questions are arithmetic: start count, rule, cooldown, min/max

  • work out the count minute by minute. "Instances still run the old image after the update" =

upgrade policy Manual.

In an interview: "How would you make a VM-based service survive a datacenter outage?" - "Run at least two instances across availability zones, behind a zone-redundant Standard load balancer, with zone-redundant or replicated data. For more than a couple of VMs, a scale set across zones with autoscale, a health probe and rolling upgrades."

Why it helps

"Survive a rack failure", "survive a datacenter failure" and "add VMs when CPU is high" are core AZ-104 questions with exact answers. In production the subtle part is autoscale: a schedule profile with a minimum of zero, scale-in that fights scale-out, a cooldown that is too short.

This chapter's incident is a scale set that scaled itself to zero overnight, doing exactly what its autoscale profile said.

Commands in this lesson

cd az jq

FAQ

Fault domain vs update domain?

A fault domain is a group of hardware sharing power and a network switch - a rack. An update domain is a group of VMs Azure restarts together during planned maintenance. An availability set spreads VMs over up to 3 fault domains and up to 20 update domains.

Can a VM be in an availability set and a zone?

No. A VM is in an availability set or in a zone, not both, and you choose when you create it - you cannot add an existing VM to a set. Zones give the higher SLA (99.99% with two or more VMs across zones) because they survive the loss of a whole datacenter.

Uniform or flexible orchestration?

Flexible is the default for new scale sets: instances are normal VMs you can see and manage, they can mix sizes, and they spread over zones and fault domains. Uniform is the older mode with identical instances managed only through the scale set. New designs use flexible.

How does autoscale decide?

Each profile has a minimum, maximum and default capacity and rules: a metric, a time aggregation, an operator and threshold over a window, and an action (increase or decrease by N) with a cooldown. Scale-out happens if any scale-out rule is met; scale-in only if all scale-in rules are met. Recurrence profiles replace the default profile at set times.

Why did my manual scale change back?

Because autoscale owns the capacity. At its next evaluation it applies the active profile's rules and limits again, overwriting a manual az vmss scale. Change the autoscale setting, or disable it, instead of scaling by hand.

In an interview Mid

How would you make a VM-based service survive a datacenter outage?

Run at least two instances across availability zones, behind a zone-redundant Standard load balancer, with zone-redundant or replicated data. For more than a couple of VMs, a scale set across zones with autoscale, a health probe and rolling upgrades - and a minimum capacity that still serves traffic when one zone is gone.

Also asked: What does an availability set protect against that a single VM does not? · How do scale-in and scale-out rules interact in autoscale? · Why would a scale set scale down to zero?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.