Sort Terraform (HCL) file by Resource Block Names using GNU `awk`

tf_vars_sort.awk

#!/usr/bin/env -S awk -f
# https://gist.github.com/yermulnik/7e0cf991962680d406692e1db1b551e6
# Tested with GNU Awk 5.0.1, API: 2.0 (GNU MPFR 4.0.2, GNU MP 6.2.0)
# Usage: /path/to/tf_vars_sort.awk < variables.tf | tee sorted_variables.tf
# Note: "chmod +x /path/to/tf_vars_sort.awk" before use
# No licensing; yermulnik@gmail.com, 2021-2023
{
	# skip blank lines at the beginning of file
	if (!resource_type && length($0) == 0) next

	# pick only known Terraform resource definition block names of the 1st level
	# https://github.com/hashicorp/terraform/blob/main/internal/configs/parser_config.go#L55-L163
	switch ($0) {
		case /^[[:space:]]*(locals|moved|terraform)[[:space:]]+{/:
			resource_type = $1
			resource_ident = resource_type "|" block_counter++
		case /^[[:space:]]*(data|resource)[[:space:]]+("?[[:alnum:]_-]+"?[[:space:]]+){2}{/:
			resource_type = $1
			resource_subtype = $2
			resource_name = $3
			resource_ident = resource_type "|" resource_subtype "|" resource_name
		case /^[[:space:]]*(module|output|provider|variable)[[:space:]]+"?[[:alnum:]_-]+"?[[:space:]]+{/:
			resource_type = $1
			resource_name = $2
			resource_ident = resource_type "|" resource_name
	}
	arr[resource_ident] = arr[resource_ident] ? arr[resource_ident] RS $0 : $0
} END {
	# exit if there was solely empty input
	# (input consisting of multiple empty lines only, counts in as empty input too)
	if (length(arr) == 0) exit
	# declare empty array (the one to hold final result)
	split("", res)
	# case-insensitive string operations in this block
	# (primarily for the `asort()` call below)
	IGNORECASE = 1
	# sort by `resource_ident` which is a key in our case
	asort(arr)

	# blank-lines-fix each block
	for (item in arr) {
		split(arr[item],new_arr,RS)

		# remove multiple blank lines at the end of resource definition block
		while (length(new_arr[length(new_arr)]) == 0) delete new_arr[length(new_arr)]

		# add one single blank line at the end of the resource definition block
		# so that blocks are delimited with a blank like to align with TF code style
		new_arr[length(new_arr)+1] = RS

		# fill resulting array with data from each resource definition block
		for (line in new_arr) {
			# trim whitespaces at the end of each line in resource definition block
			gsub(/[[:space:]]+$/, "", new_arr[line])
			res[length(res)+1] = new_arr[line]
		}
	}

	# ensure there are no extra blank lines at the beginning and end of data
	while (length(res[1]) == 0) delete res[1]
	while (length(res[length(res)]) == 0) delete res[length(res)]

	# print resulting data to stdout
	for (line in res) {
		print res[line]
	}
}

Hi!!

Thank you very much for your function, it works like a charm for variables.tf and outputs.tf files. We would love to use it for other files (or rather all of them), but we saw a problem.

Given the following block e.g.:

provider "aws" {
  region = "eu-west-1"

  assume_role {
    role_arn = "arn:aws:iam::0123456789:role/changemelol"
  }

  default_tags {
    tags = local.tags
  }
}

this program removes all the newlines and results in the following block:

provider "aws" {
  region = "eu-west-1"
  assume_role {
    role_arn = "arn:aws:iam::0123456789:role/changemelol"
  }
  default_tags {
    tags = local.tags
  }
}

Is it possible to modify this program to not strip the newlines found into the blocks?

Thanks!!

@davidcaste I updated the script by reworking it a little bit to simplify and streamline code where applicable along with rectifying bug which was affecting provider block. Please let me know if it works for you now as expected and/or there any new bugs sneaked in.

Thanks @yermulnik !! I tried again the script with our codebase and it works very well. It respects the newlines into the blocks, so great!

But I saw sometimes the awk program stuck, and after some digging around I found it happens with empty files. This is the minimal example to see the problem I'm talking about:

echo "" | awk -f tf_vars_sort.awk

Thanks!! :-)

@davidcaste Good catch! Thanks. I've updated the script to ignore empty input (including input which consists of multiple empty lines).
Bear in mind this script is more of a "at bench scale" effort hence please test and verify it properly and thoroughly before using in production CI/CD 😉
And thanks for the contribution 🙇‍♂️

Sorry for the late reply, but now works great!!

Thank you!!

hi again!

I'm doing some tests with our big IaC repository, and I found the latest version of the script (rev 12) fails with this resource:

resource "aws_imagebuilder_component" "agens" {
  name    = "agensgraph"
  data    = file("${path.module}/components/agensgraph-2.1.3.yaml")
  version = "2.1.3"

  platform = "Linux"
  supported_os_versions = [
    "Amazon Linux 2"
  ]

  tags = merge(
    local.tags, {
      Name = "agensgraph"
    }
  )
}

If I copy this resource to a file and I execute the script, the output is the following:

$ cat foo.tf  | awk -f tf_vars_sort.awk
  data    = file("${path.module}/components/agensgraph-2.1.3.yaml")
  version = "2.1.3"

  platform = "Linux"
  supported_os_versions = [
    "Amazon Linux 2"
  ]

  tags = merge(
    local.tags, {
      Name = "agensgraph"
    }
  )
}

resource "aws_imagebuilder_component" "agens" {
  name    = "agensgraph"

I don't know how many resources we have (many!!) but this is the only one I saw that was mangled.

Thank you!!

@davidcaste That's a good catch. I've updated the gist to make it be more specific on Terraform identifiers. Thanks for continuously assessing this script 👍

hi!! Thank you very much for your quick response, and sorry for bothering you!!

I did a quick test and the resource that failed now is rendered correctly 👍 . This Monday I'll try again with the entire repo, if I find more problems I'll let you know!

Thanks!!

and sorry for bothering you!!

No worries at all!

if I find more problems I'll let you know!

Thank you! I'm glad this script helps =)

Thanks for this bit of brilliant :)
Can confirm it works on MacOS (BigSur) running GNU Awk 5.1.1, API: 3.1 (GNU MPFR 4.1.0, GNU MP 6.2.1)

thanks @yermulnik for this! I've integrated it in a git commit hook so I can enforce it on some files.
was wondering; would it be possible to process data that's only between specific comments? For example:

variable "foo" {}

# tf-sort: begin

variable "baz" {}

variable "bar" {}

# tf-sort: end

Result would be:

variable "foo" {}

# tf-sort: begin

variable "bar" {}

variable "baz" {}

# tf-sort: end

@yohanb Thanks for the comment, though I don't think I'm up to implement such an exemption logic, since it will require quite an essential change in the whole snippet code.
This snippet was meant as an example of a simple implementation of how to sort TF HCL file in scope of some thread in issue (where you came here from probably), and hence it doesn't even support comments which are placed e.g. before definition blocks (which to be honest I discovered by an accident).
In your case I'd go with keeping "protected" (as in "those to be not sorted") vars in a dedicated file(s), which don't get processed by your commit hook at all.
And of course I'd be happy to accept contributions in the form of git diff patches. Else please feel free to fork/copy and improve and leave a link to your gist so that others can benefit from the expanded/improved version(s) of this snippet 👍

@yermulnik thanks for the quick answer.
Makes more sense to manage this with folders like you mentioned.
I'll let you know what I come up with! 👍

Just wanted to express my thanks for this. Saves me trouble all the time.

Is it possible to add a -recursive feature?

Do it for every file in my subdirectories as well. Looks like great work and a big value add but can't find the repo in chocolatey or homebrew!

@CGAndrej This script isn't mean to be run on its own, but rather to read from stdin and print result to stdout — this is why a feature to process directories recursively isn't doable.
To achieve your goal you may leverage e.g. find like this:

#!/usr/bin/env bash
set -e # abort on errors

chmod +x /path/to/tf_vars_sort.awk

find path/to/dir/with/tf/files/ -name "*.tf" -print0 | while read -d $'\0' FILE; do
    echo "Processing: $FILE"
    cp "$FILE" "$FILE.bak" # preserve backup of original file
    /path/to/tf_vars_sort.awk < "$FILE" > "${FILE}.sorted"
    mv "${FILE}.sorted" "$FILE" # overwrite original file
done

Hope this helps.

JFYI: recently there was tfsort utility released, which provides the alike functionality, but is written in Golang — https://github.com/AlexNabokikh/tfsort

tfsort utility released, which provides the alike functionality

@yermulnik This awk script supports more than variables.tf and output.tf. I've been unable to use tfsort for regular resource files whereas this works for a greater context.

@yermulnik This awk script supports more than variables.tf and output.tf. I've been unable to use tfsort for regular resource files whereas this works for a greater context.

@camilosantana Glad to hear this script makes value 👍

Still works, macos Sonoma 14.2.1, GNU Awk 5.3.0, API 4.0, (GNU MPFR 4.2.1, GNU MP 6.3.0), just needed to change awk to 'gawk' in the first line of the script. https://formulae.brew.sh/formula/gawk

#!/usr/bin/env -S gawk -f
...