Padstone v2: Multiple targets, like a Makefile

.padstone.hcl

# Variables are set on the command line:
#    padstone build version=v0.0.1 ami
variable "version" {
  default = "dev"
}

provider "aws" {
  region = "us-west-2"
}

default_build_targets = ["ami"]

target "ami" {
  resource "aws_ami_from_instance" "result" {
    name = "myapp-${var.version}"
    
    # This interpolation creates a dependency on the "source_instance" target,
    # which causes "padstone build ami" to build "source_instance" first, then
    # build "ami", and then destroy the resources created for "source_instance".
    source_instance_id = "${target.source_instance.id}"
  }

  # By default, depended targets are considered temporary and cleaned up
  # automatically once the user-specified targets are complete. However, this
  # can be overridden where the dependent target resources are required to
  # remain in order for this target to function:
  #  kept_targets = ["source_instance"]
    
  # Interpolations of target.NAME.ATTR are the primary way to declare a
  # "support dependency" but this can also be explicitly requested in cases
  # where a dependency cannot be expressed as an interpolation:
  #  supporting_targets = ["source_instance"]
    
  outputs {
    id = "${aws_ami_from_instance.result.id}"
    
    # Interpolating a target attribute into an output is another way
    # to declare a "kept target", which will be retained after the build
    # of this target is complete.
    # instance_id = "${target.source_instance.id}"
  }
}

# The user could potentially run "padstone build source_instance" to directly target this,
# in which case padstone will create all of these resources and then exit with them still
# existing.
#
# However, the intention here is to use "source_instance" as a supporting target when
# running "padstone build ami", which means Padstone will create the resources within this
# target, then create the resources within "ami" target with the source instance results
# interpolated, and will then *destroy* the resources in this block before the build process
# is considered complete.
target "source_instance" {
  data "aws_ami" "source" {
    select = "latest"

    owner = "self"
    tags {
      Name = "ubuntu-wheezy"
    }
  }
  
  data "aws_subnet" "build" {
    tags {
      Name = "build"
    }
  }
  
  resource "aws_instance" "result" {
    instance_type = "t2.micro"
    ami = "${data.aws_ami.source.id}"
    subnet_id = "${data.aws_subnet.build.id}"
    
    tags {
      Name = "myapp build ${var.version}"
    }
  }

  resource "aws_security_group" "provisioning" {
    name = "myapp-${var.version}-${unique_id(7)}"
    vpc_id = "${aws_subnet.build.vpc_id}"
    
    ingress {
      from_port = 22
      to_port = 22
      protocol = "tcp"
      cidr_blocks = ["0.0.0.0/0"]
    }
    
    egress {
      from_port = 0
      to_port = 0
      protocol = "-1"
      cidr_blocks = ["0.0.0.0/0"]
    }
  }

  resource "aws_key_pair" "provisioning" {
    name = "myapp-${var.version}-${unique_id(7)}"
    public_key = "${tls_private_key.provisioning.public_key_openssh}"    
  }

  resource "tls_private_key" "provisioning" {
    algorithm = "RSA"
  }

  outputs {
    id = "${aws_instance.result.id}"
  }
}

Padstone-Notes.md

Padstone v2 Notes

Padstone is a development environment and build automation tool that is based on Terraform. This is a second pass at a design for it, this time adopting a UI concept similar to that of make.

Concepts

As well as the usual Terraform concepts of managed resources, data resources, modules, providers and provisioners, Padstone introduces the concept of a "target". A target is a named set of resources (which may also include nested Terraform modules) which are created or destroyed together.

Targets have outputs in a similar way as Terraform modules have outputs. One target can interpolate the value of another target's output, which then creates a "supporting" dependency. A target's supporting targets are the targets that must exist during that target's creation, but that can be destroyed once the goal target has been completed.

A target may also have kept targets, which are similar to supporting targets except that their resources are retained until the goal target has been destroyed.

Modes

Padstone has two main "modes" of operation, each with their own commands:

Development mode commands are used when the user is actively working on a development task, and Padstone is managing resources that are supporting that development task such as local application virtual machines/containers. In this case Padstone is operating in a similar way to Terraform, keeping track of resources in a state file and applying diffs to them in response to changes to the configuration. When the developer has finished work, these resources can be cleaned up.

Build mode commands are used when the user wishes to produce artifacts for testing or deployment. In this case, Padstone ignores the development mode state and produces a pristine set of resources based on the chosen goal targets, recording the resulting resources and outputs in a result file that can then be used for deployment. When the created artifacts are no longer required (e.g. because new artifacts have superseded them) they can optionally be destroyed using the result file.

Development Workflow

The development workflow consists of commands that somewhat resemble Vagrant commands:

• padstone up [targets] creates or updates the named targets and then retains their resources in the development state file.
• padstone down [targets] destroys any extant resources for the named targets.
• padstone provision [targets] re-runs the provisioner steps for any resources in the named targets or any dependent targets.
• padstone respin [targets] destroys the named targets and then builds them again. Essentially the same as down followed by up on the given targets.

All of these commands take an optional set of goal target names. If no targets are specified then the default_dev_targets setting in the configuration provides a default set to use for up and provision, and down defaults to the full set of targets that are currently instantiated.

padstone up will also instantiate any supporting targets for the goal targets, but will destroy them before it returns since they are needed only during creation.

The development state retains the current set of resources for each target that has been instantiated successfully. It also includes "tainted" target instances that indicate that an error occured during their creation or deletion, thus allowing them to be cleaned up on a subsequent run.

Most of the time there will be little reason to use build-oriented targets in development mode, but this is allowed because it's likely to be useful when making iterative improvements to the build process or to the artifacts. padstone respin can be used to quickly re-run the creation steps without leaving a trail of unnecessary build artifacts.

Build Workflow

The build workflow has its own smaller set of commands:

• padstone build [targets] creates and provisions the named targets and produces a build result file.
• padstone destroy <build-result-file> destroys the resources recorded in the given build result file.

The build subcommand always starts with a pristine empty state, ignoring any resources that might be present from development mode. If no targets are listed on its command line, the configuration setting default_build_targets is used as a default set.

The build result file records the output values of each target along with the state for any resource instances that were created. If the build failed, it will also include tainted target instances which are not used to set outputs but are destroyed by padstone destroy, allowing the user to clean up after a failed build.

When a given goal target has supporting targets, the supporting targets are instantiated before the goal targets are and then they are destroyed before the build process is considered to be complete. Thus after a successful build only the goal targets their kept targets (if any) are retained and recorded in the build result file. A failed build make contain partially-created, tainted instances of the supporting targets.

"local" provider

Terraform doesn't have much support for interacting with local resources because in its usual mode of operation it is being run on many different machines.

The development mode of Padstone can make more reasonable use of local resources since it is always strictly used by one developer in one work directory.

Thus Padstone has an extra Terraform provider called local that provides the following resources:

• resource "local_file" writes a file to local filesystem whose contents are a configured string. Updates it when necessary, and deletes it on destroy.
• resource "local_file_generated" is a special case of resource "local_file" where the file contents are specified not by literal string but rather as a command to execute on the local system to generate the file. In this case, only the presence of the file is monitored and the file will be re-generated only if it is deleted (or if explicitly rebuilt using padstone respin).
• resource "local_daemon" forks a child process which daemonizes itself and then runs a given command. Tracks the pid of the process and starts a fresh copy if it goes away. Sends it SIGTERM when destroyed.
• data "local_file" reads a file from the local filesystem.
• data "local_exec" runs a given command on the local system and exports the stdout/stderr from it if it exits successfully.