Packer Vagrant VirtualBox Ubuntu 22.04 install with Cloud-Init

README.md

Ubuntu Packer VirtualBox

A simple Packer + Vagrant VirtualBox install of Ubuntu live-server 22.04 (x86) using Cloud-Init, and documentation of my host desktop.

host setup

A fresh install of any supported LTS or the latest Ubuntu desktop version is good. I have tested this on ubuntu 18.04.6 running on an i5-4570 (4 core 3.20GHz) with 16GB of RAM, a nearly 10 year old computer. Get a dedicated machine, it need not be fancy or new.

Update and get setup with some tools + deps first:

sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install -y \
    apt-transport-https ca-certificates tcpdump wireshark nmap software-properties-common build-essential \
    htop nethogs iotop ccze tmux git vim make curl wget jq expect dos2unix lsof strace tree entr bats  \
    python3 python3-pip python3-dev python3-venv python3-passlib squashfs-tools minicom screen \
    libssl-dev p7zip-full unrar fakeroot whois pwgen binutils debootstrap mtools \
    cloud-init xorriso genisoimage mkisofs isolinux syslinux-common grub-pc-bin grub-efi-amd64-bin;

Install VS-Code:

snap install --classic code

Install Hashicorp tools:

mkdir ~/src/
curl --output ~/src/apt-hashicorp.gpg -fsSL https://apt.releases.hashicorp.com/gpg
gpg --show-keys ~/src/apt-hashicorp.gpg
# check key ID
sudo cp -v ~/src/apt-hashicorp.gpg /usr/local/share/keyrings/hashicorp.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/local/share/keyrings/hashicorp.asc] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install packer vagrant -y
packer version
vagrant --version

Install Ansible from PPA if you wish:

sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install ansible -y
ansible --version

VMs

Virtual Box

Install Virtual Box from either

Option 1) Ubuntu mirror

Use Ubuntu package:

sudo apt-get install virtualbox -y

Option 2) Oracle mirror

Get direct from Oracle:

curl --output ~/src/oracle_vbox_2016.asc -fsSL https://www.virtualbox.org/download/oracle_vbox_2016.asc
curl --output ~/src/oracle_vbox.asc -fsSL https://www.virtualbox.org/download/oracle_vbox.asc
gpg --show-keys ~/src/oracle_vbox_2016.asc
gpg --show-keys ~/src/oracle_vbox.asc
# check key ID
sudo cp -v ~/src/oracle_vbox_2016.asc /usr/local/share/keyrings/oracle_vbox_2016.pgp
sudo cp -v ~/src/oracle_vbox.asc /usr/local/share/keyrings/oracle_vbox.pgp
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/local/share/keyrings/oracle_vbox_2016.pgp] https://download.virtualbox.org/virtualbox/debian $(lsb_release -cs) contrib" | sudo tee /etc/apt/sources.list.d/virtualbox.list
sudo apt-get update
sudo apt-get install virtualbox-6.1 -y

file layout

Create files:

git clone https://gist.github.com/craig-m/eb4a0840fa03a82398423769676ef426 ubuntu-2204-vm
cd ubuntu-2204-vm/
touch meta-data
mkdir upload

Create a .gitignore file with:

.vagrant
packer_cache/
output-*
local/
venv/
*.iso
*.log
*.box

Install some Python tools into the folder venv/:

python3 -m venv venv && source venv/bin/activate
python3 -m pip install -U setuptools pip tox
python3 -m pip install \
  ansible-runner ansible-lint molecule ansible-core molecule-vagrant paramiko invoke

The end result:

ubuntu-vm/
        ├── upload/
        ├── venv/
        ├── .gitignore
        ├── meta-data
        ├── user-data
        ├── README.md
        ├── playbook-packer.yml
        ├── playbook-vagrant.yml
        ├── Ubuntu.pkr.hcl
        └── Vagrantfile

using

If you want to change the password in user-data you can run mkpasswd -m sha-512 but note what the documentation has to say about this password hash:

#     Please note: while the use of a hashed password is better than
#     plain text, the use of this feature is not ideal. Also,
#     using a high number of salting rounds will help, but it should
#     not be relied upon.
#
#     In other words, this feature is a potential security risk and is
#     provided for your convenience only. If you do not fully trust the
#     medium over which your cloud-config will be transmitted, then you
#     should not use this feature.

Building:

Validate the Cloud-Init user-data file:

cloud-init schema --config-file user-data

Create the Vagrant ubuntu.box file:

packer init .
export PACKER_LOG=1
export PACKER_LOG_PATH=$(pwd)/packer_$(date +%H%M%d%m).log
packer validate . && packer build Ubuntu.pkr.hcl

Running:

The Vagrantfile supports multiple VM, each with own static IP. Edit as needed.

vagrant validate Vagrantfile && vagrant up
vagrant status

This will start the virtual machine/s and log you into them.

Cleaning:

Rinse and repeat.

vagrant destroy -f
vagrant box remove ubuntu.box -f
rm -fv -- *.log ubuntu.box

playbook-packer.yml

---

- name: "packer playbook"

  hosts: all
  gather_facts: yes

  vars:
    package_install: [
      { add_item: 'lsof' },
      { add_item: 'dos2unix' },
      { add_item: 'vim' },
      { add_item: 'tmux' },
      { add_item: 'git' },
      { add_item: 'make' },
      { add_item: 'build-essential' },
      { add_item: 'tasksel' },
      { add_item: 'jq' },
      { add_item: 'tcpdump' },
      { add_item: 'socat' },
      { add_item: 'expect' },
      { add_item: 'sshfs' },
      { add_item: 'gnupg2' },
      { add_item: 'inotify-tools' },
      { add_item: 'apparmor-utils' },
      { add_item: 'monitoring-plugins-common' },
      { add_item: 'monitoring-plugins-basic' },
      { add_item: 'ca-certificates' },
      { add_item: 'lsb-release' },
      { add_item: 'python3-dev' },
      { add_item: 'acl' }
    ]
    package_remove: [
      { del_item: 'telnet' }
    ]
    create_folders: [
      { dir: '/opt/datas/', mode: '0775', owner: 'root', group: 'oper' },
      { dir: '/home/oper/downloads/', mode: '0700', owner: 'oper', group: 'oper' },
      { dir: '/home/oper/upload/', mode: '0700', owner: 'oper', group: 'oper' },
      { dir: '/root/tmp/', mode: '0700', owner: 'root', group: 'root' },
      { dir: '/mnt/sshfs/', mode: '0775', owner: 'root', group: 'root' }
    ]
    newuser_name: "cluster"
    newuser_home: "/home/cluster"

  tasks:

    - name: "checks and debug"
      block:
        - name: "OS check"
          assert:
            that:
              - ansible_os_family == "Debian"
        - name: "Display hostname"
          debug:
            msg: "inventory_hostname {{ inventory_hostname }}"
        - name: "cloud init files"
          block:
              - name: "wait for cloud-init vmhost.fact"
                wait_for:
                  path: /root/welcome.txt
                  search_regex: Welcome
                become: true
              - name: "check ust-blacklist.conf"
                wait_for:
                  path: /etc/modprobe.d/cust-blacklist.conf
                  search_regex: blacklist
              - name: "wait for cloud-init vmhost.fact"
                wait_for:
                  path: /etc/ansible/facts.d/vmhost.fact
              - name: "wait for sudoers file"
                wait_for:
                  path: /etc/sudoers.d/basebox
        - name: "cloud init mod"
          block:
              - name: "Gather all facts of cloud init"
                cloud_init_data_facts:
                  filter: status
                register: res
                until: "res.cloud_init_data_facts.status.v1.stage is defined and not res.cloud_init_data_facts.status.v1.stage"
                retries: 50
                delay: 5

    - name: "user and file config"
      block:
        - name: "create new user"
          ansible.builtin.user:
            name: "{{ newuser_name }}"
            comment: "clust user"
            home: "{{ newuser_home }}"
            shell: /usr/bin/bash
            create_home: false
        - name: "homedir for {{ newuser_name }}"
          ansible.builtin.file:
            path: "{{ newuser_home }}"
            state: directory
            mode: "0775"
            owner: "{{ newuser_name}}"
            group: "{{ newuser_name }}"
        - name: "files and folders"
          block:
            - name: "create common folders"
              ansible.builtin.file:
                path: "{{ item.dir }}"
                state: directory
                mode: "{{ item.mode }}"
                owner: "{{ item.owner }}"
                group: "{{ item.group }}"
              with_items: "{{ create_folders }}"
      become: true

    - name: "software"
      block:
        - name: "install OS packages"
          ansible.builtin.package:
            name: "{{ item.add_item }}"
            state: present
          retries: 3
          with_items: "{{ package_install }}"
          become: true
          tags:
            - software
        - name: "remove OS packages"
          ansible.builtin.package:
            name: "{{ item.del_item }}"
            state: absent
          with_items: "{{ package_remove }}"
          become: true
          tags:
            - software
        - name: "upgrade apt packages"
          ansible.builtin.apt:
            name: "*"
            state: latest
            update_cache: yes
          become: true
          retries: 3
          tags:
            - software

    - name: "os setup"
      block:
        - name: Disable problematic udev service
          ansible.builtin.systemd:
            name: systemd-udev-settle
            state: stopped
            enabled: no
          become: true

    - name: "Finish up"
      block:
        - name: "reboot machine"
          ansible.builtin.reboot:
            reboot_timeout: 3600
          become: true
        - name: "create readme"
          ansible.builtin.copy:
            dest: /home/oper/base-image.txt
            mode: 0644
            content: |
              ---=== packer image details ===---
              distro:   {{ ansible_distribution }} {{ ansible_distribution_version }} {{ ansible_architecture }}
              ansible:  {{ ansible_version }}
              python:   {{ ansible_playbook_python }}
        - name: "debug note"
          debug:
            msg: "-- playbook.yml has finished --"

playbook-vagrant.yml

---

- name: "Vagrant VM playbook"

  hosts: all
  gather_facts: yes

  vars:
    testvar123: "testvar123"

  tasks:

- name: "node0 only"
  hosts: controlnode
  gather_facts: yes
  tasks:
    - name: "cockpit"
      block:
        - name: "setup cockpit"
          block:
            - name: "install cockpit package"
              ansible.builtin.package:
                name: cockpit
                state: present
              tags:
                - software
              become: true
            - name: "turn on cockpit socket"
              service:
                name: cockpit.socket
                state: started
                enabled: yes
              become: true
            - name: "Pause until web ui is up"
              uri:
                url: "http://localhost:9090/"
                follow_redirects: none
                method: GET
              register: _result
              until: _result.status == 200
              retries: 30
              delay: 5 # seconds
          become: true


- name: "worker nodes only"
  hosts: allnodes
  vars:
    foobar: "foobar"
  gather_facts: yes
  tasks:
    - name: "nodes"
      block:
        - name: "group 3 test"
          debug:
            msg: "test"

Ubuntu.pkr.hcl

packer {
  required_version = ">= 1.8.3"

  required_plugins {
    virtualbox = {
      version = ">= 0.0.1"
      source = "github.com/hashicorp/virtualbox"
    }
  }
}


variable "iso_checksum" {
  type    = string
  default = "10f19c5b2b8d6db711582e0e27f5116296c34fe4b313ba45f9b201a5007056cb"
}

variable "iso_url" {
  type    = string
  default = "https://releases.ubuntu.com/22.04/ubuntu-22.04.1-live-server-amd64.iso"
}


source "virtualbox-iso" "ubn-vb" {
  boot_command            = ["<wait5><wait5>yes<enter><wait>"]
  boot_wait               = "2m30s"
  communicator            = "ssh"
  cpus                    = "4"
  disk_size               = "32768"
  guest_os_type           = "Ubuntu_64"
  guest_additions_mode    = "disable"
  iso_checksum            = "${var.iso_checksum}"
  iso_url                 = "${var.iso_url}"
  memory                  = "4096"
  shutdown_command        = "sudo -S shutdown now"
  ssh_username            = "ubuntu"
  ssh_password            = "ubuntu"
  ssh_port                = "22"
  ssh_timeout             = "2h"
  vm_name                 = "ubuntu-vb"
  cd_files                = ["user-data", "meta-data"]
  cd_label                = "cidata"
}


build {
  sources = ["source.virtualbox-iso.ubn-vb"]
  
  provisioner "shell" {
    inline            = ["uptime"]
    pause_before      = "1m"
    timeout           = "15m0s"
  }

  provisioner "shell" {
    inline            = ["sudo -S cloud-init status --wait"]
    timeout           = "15m0s"
  }

  provisioner "ansible" {
    playbook_file   = "playbook-packer.yml"
    pause_before    = "10s"
    timeout         = "10m0s"
  }
  
  provisioner "shell" {
    expect_disconnect = true
    inline            = ["sudo -S reboot now"]
    pause_before      = "10s"
    timeout           = "10m0s"
  }


  post-processor "vagrant" {
    keep_input_artifact  = false
    compression_level    = 0
    output               = "ubuntu.box"
  }

}

user-data

#cloud-config
#
# Cloud Init user-data config. 
# Do not alter first line of this file.
# https://cloudinit.readthedocs.io/en/latest/
autoinstall:
  version: 1
  early-commands:
    - systemctl stop ssh
    - sed -i "s/#Cache=.*/Cache=yes/g" /etc/systemd/resolved.conf
    - sed -i "s/#DNS=.*/DNS=4.2.2.1 4.2.2.2 208.67.220.220/g" /etc/systemd/resolved.conf
    - sed -i "s/#FallbackDNS=.*/FallbackDNS=4.2.2.1 4.2.2.2 208.67.220.220/g" /etc/systemd/resolved.conf
    - systemctl restart systemd-resolved
  locale: en_US.UTF-8
  network:
      network:
          version: 2
          ethernets:
              eth0:
                 dhcp4: yes
                 dhcp6: no
  apt:
    geoip: false
    primary:
      - arches: [default]
        search:
          - http://au.archive.ubuntu.com/ubuntu/
          - http://archive.ubuntu.com
  keyboard:
    layout: en
    variant: us
  storage:
    layout:
      name: lvm
  identity:
    hostname: ubnbox
    username: oper
    password: $6$lg4c2ZWWNopoR$ym2Ac0LAvuRTWde4l3aqZ2I0G54jy7aGO8IcFdkmrvlAVKcsk/YU3fZTybIkucd6GZLZnweC0ywINWTZ3VD1M/
  ssh:
    install-server: yes
    allow-pw: true
  packages:
    - ca-certificates
    - uuid
    - software-properties-common
    - python3-pip
    - python3-venv
  output:
    init: "> /var/log/cloud-init.log"
    config: [ ">> /tmp/foo.out", "> /var/log/cloud-config.log" ]
    final:
      output: "| tee /tmp/final.stdout | tee /var/log/cloud-final.log"
      error: "&1"
  final_message: "System installation complete."
  user-data:
    disable_root: true
    package_update: true
    package_upgrade: false
    write_files:
      - path: /etc/sudoers.d/basebox
        owner: 'root:root'
        permissions: '0644'
        defer: true
        content:
          oper ALL=(ALL) NOPASSWD:ALL
      - path: /root/welcome.txt
        owner: 'root:root'
        permissions: '0644'
        defer: true
        content:
          Welcome to root.
      - encoding: b64
        path: /etc/modprobe.d/cust-blacklist.conf
        owner: 'root:root'
        permissions: '0644'
        defer: true
        content: YmxhY2tsaXN0IGpveWRldgpibGFja2xpc3Qgc25kX2hkYV9pbnRlbApibGFja2xpc3Qgc25kX2hkYV9jb2RlYwpibGFja2xpc3Qgc25kX2hkYV9jb2RlY19nZW5lcmljCmJsYWNrbGlzdCBzbmRfaGRhX2NvcmUKYmxhY2tsaXN0IHNuZF9wY20KYmxhY2tsaXN0IHNuZF9od2RlcApibGFja2xpc3Qgc25kX3RpbWVyCmJsYWNrbGlzdCBzbmQKYmxhY2tsaXN0IHNvdW5kY29yZQ==
      - encoding: b64
        path: /etc/ansible/facts.d/vmhost.fact
        owner: 'root:root'
        permissions: '0755'
        defer: true
        content: IyEvdXNyL2Jpbi9lbnYgc2gKZWNobyAie1widm0tY2xvdWQtaW5pdFwiIDogXCJ0cnVlXCJ9Ig==
  late-commands:
    - curtin in-target --target=/target -- add-apt-repository --yes --update ppa:ansible/ansible
    - curtin in-target --target=/target -- apt install ansible -y -q
    - mkdir /target/etc/ansible/facts.d
    - sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="\(.*\)"$/GRUB_CMDLINE_LINUX_DEFAULT=="\1 net.ifnames=0 biosdevname=0"/g' /target/etc/default/grub
    - sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=" net.ifnames/GRUB_CMDLINE_LINUX_DEFAULT="net.ifnames/g' /target/etc/default/grub
    - curtin in-target --target=/target -- update-grub
    - mkdir /target/etc/opt/img/
    - curtin in-target --target=/target -- logger "finished late-commands"

Vagrantfile

Vagrant.require_version ">= 2.2.14"

servers = [
    { 
        :hostname => "vmnode0",
        :ip => "192.168.50.10",
        :ram => 4096,
        :cpu => 2,
        :ansible_var => { myvm_var_test: 'foobar', desk_package: 'ubuntu-gnome-desktop' }
    }, { 
        :hostname => "vmnode1",
        :ip => "192.168.50.11",
        :ram => 4096,
        :cpu => 2,
        :ansible_var => { myvm_var_test: 'foobar', desk_package: 'kubuntu-desktop' }
    }
]

Vagrant.configure("2") do |config|
    config.vm.box = "ubuntu.box"
    config.vm.box_check_update = false

    config.vm.synced_folder "upload/", "/home/oper/upload", type: "rsync", disabled: true

    config.ssh.username = "oper"
    config.ssh.password = "foobar123"
    config.ssh.insert_key = true
    config.ssh.keep_alive = true
    config.ssh.compression = false
    config.ssh.forward_agent = false

    servers.each do |machine|
        config.vm.define machine[:hostname] do |node|

            node.vm.hostname = machine[:hostname]
            node.vm.network "private_network", ip: machine[:ip]

            # VB provider settings
            node.vm.provider "virtualbox" do |vb|
                vb.memory = machine[:ram]
                vb.cpus = machine[:cpu]
                vb.check_guest_additions = false
                vb.gui = false
                vb.customize ["modifyvm", :id, "--vram", "128"]
                vb.customize ["modifyvm", :id, "--accelerate3d", "on"]
            end

            # ansible must be installed on host machine
            node.vm.provision :ansible do |ansible|
                ansible.playbook = "playbook-vagrant.yml"
                ansible.verbose = "vv"
                ansible.become = false
                ansible.extra_vars = machine[:ansible_var]
                ansible.groups = { controlnode: 'vmnode0', allnodes: 'vmnode[0-100]' }
            end
            
        end
    end

end

Very interesting... But it does not work. I had to change the keyboard layout and update to the last ubuntu LTS image (+ checksum), but that is not enough. There is a SSH issue after initial build. Indeed, once the VMs is built and after cloud-init ran, the machine reboot, and then ubuntu user is not available anymore.

I thought to use a multi stage build with virtualbox, but that is not easy... I also tried to push my SSH key, but the examples I saw are using either preseed or kickstart and I would like to stay with cloud-init.

Any update here would be useful !

BTW, the playbook vagrant does not seem to be used here, and I think a true github repository would be much appreciated, allowing us to create real issue.

Anyway, thanks for sharing.

This was a great resource, but I needed a different boot command. Was there a step you did to customize the installer, because on bare metal that short boot command would be really handy!