Create an AWS AMI from scratch

aws-ami-scratch.md

Create an AWS AMI from scratch

• Create (or import) a virtual machine
  • Any hypervisor would work though KVM does not easily support exporting to an OVA
    • The challenge with KVM is not the disk conversion from qcow2 but the generation of an OVF file (TODO Investigate options)
  • Note that the virtual machine should not be EFI-based, which EC2 does not support
• Install and configure cloud-init (minimally for deploying the ssh keypair and resetting the sshd host keys)
  • Assuming a Centos / Red Hat based system:

yum -y install cloud-init

mv /etc/cloud/cloud.cfg /etc/cloud/cloud.cfg.orig
cat <<EOF >/etc/cloud/cloud.conf
# Instance metadata is available from 169.254.169.254
# 169.254.0.0/16 is reserved for link-local addresses
# Cf: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
# Ex: curl http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key
datasource:
  Ec2:
    metadata_urls: ["http://169.254.169.254:80"]

users:
 - default

 # By default all user accounts will have lock_passwd set preventing password logins.
 # ssh_redirect_user will disable ssh logins and prompt with a message to login as the default user.
 # Ref: https://cloudinit.readthedocs.io/en/latest/topics/modules.html#users-and-groups

 - name: <account>
   ssh_redirect_user: true

 # The root login is also disabled by default

system_info:
  default_user:
    name: ec2-user
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
  distro: centos

# Globally disable ssh password auth (to include accounts not explicitly listed above)
ssh_pwauth: false

# By default ssh_deletekeys is enabled (ie delete existing ssh host keys) but must specify ssh_genkeytypes (to regenerate them)
ssh_genkeytypes: [rsa, dsa, ecdsa, ed25519]
# Ref: https://cloudinit.readthedocs.io/en/latest/topics/modules.html#host-keys

# Necessary to support logging only to /var/log/cloud-init
syslog_fix_perms: null


# TODO Investigate, likely not all of these are necessary

cloud_init_modules:
 - disk_setup
 - migrator
 - bootcmd
 - write-files
 - growpart
 - resizefs
 - set_hostname
 - update_hostname
 - update_etc_hosts
 - rsyslog
 - users-groups
 - ssh

cloud_config_modules:
 - mounts
 - locale
 - set-passwords
 - rh_subscription
 - yum-add-repo
 - package-update-upgrade-install
 - timezone
 - puppet
 - chef
 - salt-minion
 - mcollective
 - disable-ec2-metadata
 - runcmd

cloud_final_modules:
 - rightscale_userdata
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - scripts-user
 - ssh-authkey-fingerprints
 - keys-to-console
 - phone-home
 - final-message
 - power-state-change 
EOF

• Export the virtual machine as an OVA for upload to an arbitrary S3 bucket

aws s3 cp <image.ova> s3://<bucket>/<image.ova>
aws ec2 import-image --cli-input-json '{ "DiskContainers": [ { "UserBucket": { "S3Bucket": "<bucket>", "S3Key": "<image.ova>" } } ] }'
# Monitor the import status, Progress indicates percentage completion
aws ec2 describe-import-image-tasks # | jq '.ImportImageTasks[] | select(.ImportTaskId == "<task-id>")'

• Copy the imported AMI (typically named import-ami-<hexstring>) to change its name and deregister (delete) the original AMI
• Optional: Set the image as public under "Modify Image Permissions" to make it available under Community AMIs

cloud-init notes

• Re-run datasource detection and cloud-init to simulate a first boot

# Force datasource identification
DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force

# Remote artifacts to simulate a clean instance
cloud-init clean --logs

cloud-init init --local
cloud-init init

• cloud-init query --all or cat /run/cloud-init/instance-data.json
• cloud-init status --long
• cloud-id
• Log files: /var/log/cloud-init{,-output}.log
• Config files: /etc/cloud/{cloud.cfg,cloud.cfg.d/*.cfg}

Packer

Packer provides AWS AMI builders though most require building from an existing AMI and may not allow making the created AMI public. Ref: https://www.packer.io/docs/builders/amazon

TODO Investigate the use of the amazon-ebssurrogate to build from scratch