Skip to content

Instantly share code, notes, and snippets.

@nquayson

nquayson/README.md

Forked from reegnz/README.md
Created May 2, 2023 19:56
Show Gist options
  • Save nquayson/619cb75fc1460597bbf0bbc7b613d5ee to your computer and use it in GitHub Desktop.
Save nquayson/619cb75fc1460597bbf0bbc7b613d5ee to your computer and use it in GitHub Desktop.
Download ZIP
Using terraform for_each and toset instead of count
Raw
README.md

Using terraform for_each and toset instead of count

Using terraform count and the pain of working with it

If you're using terraform extensively you probably ran into an issue like this.

This is a synthetic example but I still hope the problem is recognizable as something that also happens out in the wild.

First, you have a list variable (in terraform.tfvars)

a_list = [
  "a",
  "c",
  "e",
]

You use this list with count to create a resource for each element of this list:

variable a_list {
  type = list(string)
}

resource null_resource example {
  count = length(var.a_list)

  triggers = {
    element = var.a_list[count.index]
  }
}

You do a terraform apply, create the resources, and everything is fine and dandy:

❯ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # null_resource.example[0] will be created
  + resource "null_resource" "example" {
      + id       = (known after apply)
      + triggers = {
          + "element" = "a"
        }
    }

  # null_resource.example[1] will be created
  + resource "null_resource" "example" {
      + id       = (known after apply)
      + triggers = {
          + "element" = "c"
        }
    }

  # null_resource.example[2] will be created
  + resource "null_resource" "example" {
      + id       = (known after apply)
      + triggers = {
          + "element" = "e"
        }
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.example[2]: Creating...
null_resource.example[0]: Creating...
null_resource.example[1]: Creating...
null_resource.example[2]: Creation complete after 0s [id=469513174290260597]
null_resource.example[0]: Creation complete after 0s [id=671487924317915790]
null_resource.example[1]: Creation complete after 0s [id=6678576508957404863]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Then comes a day when you realize you need to add a new item, b to the list, and just so that the list is nice and organized, you add the new element somewhere in the middle.

a_list = [
  "a",
  "b",
  "c",
  "e",
]

You run a terraform apply, thinking you get a new resource applied and be done with it. Instead, you get something like this:

❯ terraform apply
null_resource.example[0]: Refreshing state... [id=671487924317915790]
null_resource.example[1]: Refreshing state... [id=6678576508957404863]
null_resource.example[2]: Refreshing state... [id=469513174290260597]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # null_resource.example[1] must be replaced
-/+ resource "null_resource" "example" {
      ~ id       = "6678576508957404863" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "element" = "c" -> "b"
        }
    }

  # null_resource.example[2] must be replaced
-/+ resource "null_resource" "example" {
      ~ id       = "469513174290260597" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "element" = "e" -> "c"
        }
    }

  # null_resource.example[3] will be created
  + resource "null_resource" "example" {
      + id       = (known after apply)
      + triggers = {
          + "element" = "e"
        }
    }

Plan: 3 to add, 0 to change, 2 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: no

Apply cancelled.

What happened? The order of the list changed, so now the resource belonging to the 2nd item in the list now belongs to the 3rd item in the list.

You can resolve this issue the hard way, by doing terraform state mv, moving backward from the list:

❯ terraform state mv 'null_resource.example[2]' 'null_resource.example[3]'
Move "null_resource.example[2]" to "null_resource.example[3]"
Successfully moved 1 object(s).

❯ terraform state mv 'null_resource.example[1]' 'null_resource.example[2]'
Move "null_resource.example[1]" to "null_resource.example[2]"
Successfully moved 1 object(s).

Then the plan finally only shows one addition:

❯ terraform apply
null_resource.example[3]: Refreshing state... [id=469513174290260597]
null_resource.example[0]: Refreshing state... [id=671487924317915790]
null_resource.example[2]: Refreshing state... [id=6678576508957404863]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # null_resource.example[1] will be created
  + resource "null_resource" "example" {
      + id       = (known after apply)
      + triggers = {
          + "element" = "b"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.example[1]: Creating...
null_resource.example[1]: Creation complete after 0s [id=1892492995738481705]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The whole terraform state mv dance has to be done every time you add an element not to the end of the list.

Using for_each and toset and the feeling that 'it just works'

The solution consists of two parts: first, you should use for_each, but the issue with that is, it doesn't accept lists, only maps. If your list is anyway consisting of unique items, then you can use toset to convert the list to a map, and start using for_each today!

Continuing with the example we started working with count, this is how we migrate it to for_each.

We change the resource to use for_each instead of count.

resource null_resource example {
  for_each = toset(var.a_list)

  triggers = {
    element = each.value
  }
}
❯ terraform state mv 'null_resource.example[0]' 'null_resource.example["a"]'
Move "null_resource.example[0]" to "null_resource.example[\"a\"]"
Successfully moved 1 object(s).

❯ terraform state mv 'null_resource.example[1]' 'null_resource.example["b"]'
Move "null_resource.example[1]" to "null_resource.example[\"b\"]"
Successfully moved 1 object(s).

❯ terraform state mv 'null_resource.example[2]' 'null_resource.example["c"]'
Move "null_resource.example[2]" to "null_resource.example[\"c\"]"
Successfully moved 1 object(s).

❯ terraform state mv 'null_resource.example[3]' 'null_resource.example["e"]'
Move "null_resource.example[3]" to "null_resource.example[\"e\"]"
Successfully moved 1 object(s).``

We have migrated to the new for_each solution.

If we now run terraform apply, the plan is clean:

❯ terraform apply
null_resource.example["e"]: Refreshing state... [id=469513174290260597]
null_resource.example["a"]: Refreshing state... [id=671487924317915790]
null_resource.example["c"]: Refreshing state... [id=6678576508957404863]
null_resource.example["b"]: Refreshing state... [id=1892492995738481705]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Now let's see what happens if w add another element, d in the middle of the list:

a_list = [
  "a",
  "b",
  "c",
  "d",
  "e",
]

We run terraform apply to create a new resource for d.

❯ terraform apply
null_resource.example["b"]: Refreshing state... [id=1892492995738481705]
null_resource.example["e"]: Refreshing state... [id=469513174290260597]
null_resource.example["a"]: Refreshing state... [id=671487924317915790]
null_resource.example["c"]: Refreshing state... [id=6678576508957404863]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # null_resource.example["d"] will be created
  + resource "null_resource" "example" {
      + id       = (known after apply)
      + triggers = {
          + "element" = "d"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.example["d"]: Creating...
null_resource.example["d"]: Creation complete after 0s [id=5807718687674607863]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

No more messing around with terraform state mv with count and lists!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment