OnCallReady

TerraformAzureSRE · 4 min read

Terraform count vs for_each: removing one item destroys the rest of the list

Removing one item from a count list re-indexes everything after it, so Terraform replaces them. Switch to for_each and move state with moved blocks.

A pull request says "removes one unused subnet". The diff is one word, "app", deleted from a list. The plan says something else:

terminal
$ terraform plan
...
  # azurerm_subnet.s[1] must be replaced
-/+ resource "azurerm_subnet" "s" {
      ~ name = "app" -> "db" # forces replacement
    }

  # azurerm_subnet.s[2] must be replaced
-/+ resource "azurerm_subnet" "s" {
      ~ name = "db" -> "cache" # forces replacement
    }

  # azurerm_subnet.s[3] must be replaced
-/+ resource "azurerm_subnet" "s" {
      ~ name = "cache" -> "mgmt" # forces replacement
    }

  # azurerm_subnet.s[4] will be destroyed
  # (because index [4] is out of range for count)
  - resource "azurerm_subnet" "s" {
      - name                 = "mgmt" -> null
      ...
    }

Plan: 3 to add, 0 to change, 4 to destroy.

One subnet removed in the code, four subnets destroyed in Azure. Anything attached to db, cache and mgmt (VMs, private endpoints, a cluster's node pool) loses its network on apply.

Why: count identifies instances by position

The resource looked like this:

hcl
variable "subnets" {
  type    = list(string)
  default = ["web", "app", "db", "cache", "mgmt"]
}

resource "azurerm_subnet" "s" {
  count                = length(var.subnets)
  name                 = var.subnets[count.index]
  resource_group_name  = azurerm_resource_group.main.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = [cidrsubnet("10.0.0.0/16", 8, count.index)]
}

With count, Terraform tracks the instances as azurerm_subnet.s[0] to s[4]: by position in the list, not by what they are.

terminal
$ terraform state list
azurerm_resource_group.main
azurerm_virtual_network.main
azurerm_subnet.s[0]
azurerm_subnet.s[1]
azurerm_subnet.s[2]
azurerm_subnet.s[3]
azurerm_subnet.s[4]

Remove "app" (position 1) and everything after it moves up one place. s[1] used to be app and is now supposed to be db; s[2] was db and should now be cache. A subnet's name cannot be changed in place, so each shifted instance is destroyed and created again, and s[4] disappears because the list is one shorter. Terraform is doing exactly what count means.

How to read a plan so you catch it

  • Do not stop at the summary line. Search for must be replaced and # forces replacement and count them against what the PR claims.
  • A rename with # forces replacement is a destroy and a create, not an edit.
  • (because index [N] is out of range for count) tells you the list got shorter, and that the instance being destroyed is whatever was last, not the one you removed.

Any PR that removes one item from a count list and shows more than one replacement should be blocked until the resource is converted.

The fix: key instances by name with for_each

for_each takes a map or a set and identifies each instance by its key: azurerm_subnet.s["db"] stays db no matter what else is in the map. Give each subnet its CIDR explicitly, so removing one cannot shift the others' addresses either:

hcl
variable "subnets" {
  type = map(string)
  default = {
    web   = "10.0.0.0/24"
    app   = "10.0.1.0/24"
    db    = "10.0.2.0/24"
    cache = "10.0.3.0/24"
    mgmt  = "10.0.4.0/24"
  }
}

resource "azurerm_subnet" "s" {
  for_each             = var.subnets
  name                 = each.key
  resource_group_name  = azurerm_resource_group.main.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = [each.value]
}

(A CIDR computed from index(var.subnets, each.key) would bring the shift right back: remove one name and the later indexes change.)

Changing the code alone is not enough. State still has s[0]..s[4], the configuration now wants s["web"]..s["mgmt"], and Terraform would destroy all five and create five new ones.

Move the state, not the subnets

Tell Terraform the old address and the new one are the same object. Since Terraform 1.1 the reviewable way is a moved block, one per instance, committed with the change:

hcl
moved {
  from = azurerm_subnet.s[0]
  to   = azurerm_subnet.s["web"]
}
moved {
  from = azurerm_subnet.s[1]
  to   = azurerm_subnet.s["app"]
}
# ... and the same for db, cache and mgmt

The plan then shows moves and nothing else:

terminal
$ terraform plan
...
  # azurerm_subnet.s[0] has moved to azurerm_subnet.s["web"]
    resource "azurerm_subnet" "s" {
        id   = "/subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/rg-sysop-dev/providers/Microsoft.Network/virtualNetworks/vnet-sysop-dev/subnets/web"
        name = "web"
        # (3 unchanged attributes hidden)
    }
...
Plan: 0 to add, 0 to change, 0 to destroy.

The older, imperative way is terraform state mv, run by hand against the live state (keep the single quotes around addresses with brackets):

terminal
$ terraform state mv 'azurerm_subnet.s[0]' 'azurerm_subnet.s["web"]'
Move "azurerm_subnet.s[0]" to "azurerm_subnet.s["web"]"
Successfully moved 1 object(s).

It works, but nobody reviews it and every other copy of the configuration has to be run the same way. moved blocks travel with the code to every environment.

Once the move is applied, removing app is what the PR said it was:

terminal
$ terraform plan
...
  # azurerm_subnet.s["app"] will be destroyed
  # (because key ["app"] is not in for_each map)
...
Plan: 0 to add, 0 to change, 1 to destroy.

Keeping it from coming back

  • for_each for anything with a name. Keep count for N identical copies and for the count = var.enabled ? 1 : 0 on/off switch.
  • Fail the pipeline on unexpected replaces. Count the delete actions in terraform show -json tfplan and require an extra approval for any. Treat that JSON as a secret: it holds sensitive values in plain text, one of the ways a database password ends up in a CI log.
  • Protect what must not die: lifecycle { prevent_destroy = true } on subnets that carry production makes a bad plan fail instead of apply.

A plan that destroys more than you changed has two usual suspects: an index shift like this one, or a directory planning against the wrong state. And while you refactor state addresses, make sure no pipeline is applying at the same time: that is what the state lock is for.

Practise it

Incident: removing one subnet destroyed four (12.21) gives you this count-based network, already applied, to plan, convert and re-address without destroying anything. Drill: take one out without touching the rest (13.51) repeats it with different lists.

OnCallReady is free, with no ads and no tracking. RSS · All posts