Terraform Module Refactoring

README.md

Terraform Module Refactoring

In this lab we are gonna show how to extract Terraform code to modules and update the state to keep your infrastructure configuration.

If testing this out you must export your GCP project id export TF_VAR_project=your_project_id

This would be our original configuration:

// main.tf

variable "project" {
  description = "GCP project."
}

variable "region" {
  description = "GCP region."
  default     = "us-central1"
}

variable "machine_image" {
  description = "GCP machine image."
  default     = "ubuntu-os-cloud/ubuntu-1804-lts"
}

variable "machine_type" {
  description = "GCP instance type."
  default     = "f1-micro"
}

variable "ip_cidr_range" {
  description = "Subnet CIDR"
  default     = "10.0.10.0/24"
}

variable "machine_names" {
  description = "GCP instance names."
  default     = ["my-instance"]
}

provider "google" {
  version = "~> 2.0"
  project = var.project
  region  = var.region
}

resource "google_compute_network" "vpc" {
  name                    = "vpc"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "subnet" {
  name          = "subnet"
  region        = var.region
  network       = google_compute_network.vpc.self_link
  ip_cidr_range = var.ip_cidr_range
}

resource "google_compute_instance" "instamce" {
  for_each = toset(var.machine_names)

  name         = each.key
  zone         = "${var.region}-b"
  machine_type = var.machine_type

  boot_disk {
    initialize_params {
      image = var.machine_image
    }
  }

  network_interface {
    subnetwork = google_compute_subnetwork.subnet.self_link
    access_config {
    }
  }
}

To execute run

terraform init
terraform apply
[...]
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Note that 3 resources were created: the VPC, the subnet, and the VM instance.

Suppose that we have been using this configuration for some time but now we want to encapsulate all these details and expose only the variables that matter. There are many reasons for that, one of them is that the base code could be reused by other projects/teams. Let's see how the config looks after the refactor:

// main.tf

variable "project" {
  description = "GCP project."
}

provider "google" {
  version = "~> 2.0"
  project = var.project
}

module "instances" {
  source        = "./modules/base_infra"
  machine_names = ["my-instance"]
}

In the day-to-day operation, the main.tf file above is where we would make changes to create new instances or whole new setups.

The ./modules/base_infra/main.tf file below is our module with all the infrastructure details.

// ./modules/base_infra/main.tf

variable "region" {
  description = "GCP region."
  default     = "us-central1"
}

variable "machine_image" {
  description = "GCP machine image."
  default     = "ubuntu-os-cloud/ubuntu-1804-lts"
}

variable "machine_type" {
  description = "GCP instance type."
  default     = "f1-micro"
}

variable "ip_cidr_range" {
  description = "Subnet CIDR"
  default     = "10.0.10.0/24"
}

variable "machine_names" {
  description = "GCP instance names."
}

resource "google_compute_network" "vpc" {
  name                    = "vpc"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "subnet" {
  name          = "subnet"
  region        = var.region
  network       = google_compute_network.vpc.self_link
  ip_cidr_range = var.ip_cidr_range
}

resource "google_compute_instance" "instamce" {
  for_each = toset(var.machine_names)

  name         = each.key
  zone         = "${var.region}-b"
  machine_type = var.machine_type

  boot_disk {
    initialize_params {
      image = var.machine_image
    }
  }

  network_interface {
    subnetwork = google_compute_subnetwork.subnet.self_link
    access_config {
    }
  }
}

Now lets see how it looks like when we run terraform plan:

terraform init
terraform plan
[...]
Plan: 3 to add, 0 to change, 3 to destroy.

From the tf plan output you can see that Terrafom wants to replace the resources. This is exactly what we want to avoid. We want to tell Terraform that despite the changes in the configuration the infrastructure is the same.

In order to do that we need to move the Terraform state of our resources to the module. We do it using the terraform state mv command. Check the Terraform docs for a complete view of how this command works and other examples.

terraform state mv 'google_compute_subnetwork.subnet' 'module.instances.google_compute_subnetwork.subnet'
terraform state mv 'google_compute_network.vpc' 'module.instances.google_compute_network.vpc'
terraform state mv 'google_compute_instance.instamce["my-instance"]' 'module.instances.google_compute_instance.instamce["my-instance"]'

To generate these commands you can do something like this:

for item in $(terraform state list); do
  echo terraform state mv \'$item\' \'module.instances.$item\'
done

You can pipe the state list output to grep to filter only what is being refactored. And avoid running the move directly in the loop - see the comments by the end of this lab.

Now run terraform plan again and see that there are no infrasctructure changes to be applied.

terraform plan
[...]
No changes. Infrastructure is up-to-date.

In the context of this lab, moving the state was somewhat trivial. However, you should be very careful when moving the state of production infrastructure. Doing something wrong may require a lot of time to import and fix your state depending on the size of your config. To avoid issues I'd recommend you run it on your CI/CD pipeline after proper peer review, ideally through a PR. Make sure you first run it in dry-run mode and that you save the backup in a safe place that can be recovered in case of failure. See the docs for more details terraform state mv --help.