The opsman appliance will be built to some degree using the built in tooling on the machine running the packer script. This means that either your computer or the Dockerfile will have an effect on your build. Beware version changes from your workstation to the ci build. Specifically we version lock Go, virtualbox, packer, node, phantomjs, and terraform within the docker image.
Once you have an environment capable of running the packer scripts you can run be rake "vm:create[$IAAS]" Each IAAS works a bit differently. Some appliances are built using local VMs where others are built using the API of the IAAS to spin up virtual machines native to that IAAS. In the case of both google and aws you will need to provide a means to authenticate with the IAAS. Running the rake command will quickly error out with errors for missing ENV variables.
The rake command vm:create must be run from ./installation/vm. It will start from this rake file This will fetch any new chef recipies and send your request to this file Image.rb serves mostly as a try catch block and passes your request onto the ImageExporter Image exporter breaks down specific task by IAAS. You can functionally think about this step as a setup for running packer. It set's the filename, the output directory, and other IAAS specific arguments that may be needed by packer. We now travel to image builder which actually calls packer and some other build steps. This step verifies that your Packerfile is proper JSON and that mandatory arguments are present in the JSON.
Each IAAS inherits a set of generic tasks from base provider. Many of the IAASes overwrite these methods with specific tasks for that IAAS. For example the google provider overwrites the validate! method in base provider to fail if ENV['GOOGLE_PROJECT_ID'] is undefined. These methods are all named well and provide a nice entry point for methods that either should happen before or after packer actually runs.
The build step will interpret the command from your Packerfile and attempt to compile an artifact. The start of this packerfile sets variables These user defined variables are set in the IAAS provider files in the packer_user_variables section which you can see an example of in the azure provider.
The builders section pull from a set of packer provided builders. Each of these builders has unique requirements. They do all a base set of needs however. Specifially, each builder needs a base image to start the work from, a way to communicate with the vm during the build phase (usually ssh), and some information about the desired specs of the machine. You may want to use higher than necessary VM CPU/disk/memory specs to build your images. Human wait time is far more expensive that large machine time on any IAAS.
Gotcha! Beware that on some IAASes the communication medium packer defaults to may be ephemeral SSH keys. These keys are placed on your vm by an IAAS specific daemon running on the VM. Since we are using bosh developed hardened stemcell that daemon may or may not be running. This is why you see alternative communicators in the google builder.
You will know what you are well on your way to success when you can see that your packer can communicate with your VM over SSH. In many of our builds on hardened stemcells this was the largest barrier to entry.
Once communication is established packer will start running provisioners in the order specified by the Packerfile. Chef has a set number of provisioners that we can take advantage of. The basic options are shell scripts, chef, and file transfers. The file transfers are straightforward and function using scp. The shell scripts are wrapped in a larger script that argues working directory. It is also important to note that the names of your scripts will not carry over to the vm. Each script or command is named in GUID fashion and sent to the vm. Thus you cannot depend on that script to be accessible from any other script.
Chef The chef recipies are run from this entry point which calls all the recipes in the repo. These scripts will run with a minimal amount of debug logs. If you find yourself having trouble understanding a failure you can change the execute_command to include a more verbose level of logging. Be careful when editing these scripts, the recipies folder contains the files you want to modify, the cookbooks folder is generated and will not persist change
There are a number of scripts that run in packer, they range from downloading and untarring stemcells, to IAAS specific setup and cleanup, as well as generic setup and cleanup.
Most IAASes have some sort of agent that runs on the vm. This agent's responsibilities will vary from IAAS to IAAS, but without exception they will have some sort of credential management function. This means that even if we create a VM that has no ssh keys on it the agent should still be able to create an ephemeral ssh key using the agent. This method is considered to be the preferred method of authentication on supported IAASes (not vshphere) because access is easily managed by our customers. Beware that some agents may either not be installed or have errors that prevent them from starting because we are not running the hardened stemcell as a bosh deployment.
Most of the information above is stable and unlikely to change quickly. The following is more likely to change over time os beware
The scripts are run by the vcap user. Many bosh-agent scripts have hardcoded the vcap user. We grant this user passwordless sudo for the duration of the install on every IAAS. In a packer cleanup script we revoke vcap's passwordless sudo access and grant the ubuntu user passwordless sudo. Be mindful of this with your expectations of the when you will have access to sudo without a password. (Vsphere is an exception to this as it doesn't have ssh-key metadata and always has a password)
The stemcell is intended for bosh deployments. Because of this we get a bosh agent and some other bosh specific tooling. The bosh agent reads specific metadata from the IAAS on the bosh_settings key and will place an SSH key on the vm if provided there. In some IAASes we use this feature to gain initial access. After this point we rely on the IAAS agents for access and turn off the bosh agent and remove the key it may have placed on the VM.