Ubuntu 20.04 user-data files for autoinstall ISO

gistfile1.txt

#cloud-config
autoinstall:
  version: 1
  refresh-installer:  # start with an up-to-date installer
  update: yes
  interactive-sections:  # Install groups listed here will wait for user input
    - storage
  storage:  # should set the interactive default but doesn't seem to work??
    layout:
      name: direct
  locale: en_US.UTF-8
  keyboard:
    layout: us
  identity:  # This is section you may want to add to interactive-sections  (user name and password are ubuntu here)
    hostname: puget-000
    password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
    username: ubuntu
  ssh:
    allow-pw: true
    install-server: true
  apt:
    sources:
      ignored1:  # This is here to get the yaml formatting right when adding a ppa
        source: ppa:graphics-drivers/ppa
  packages: 
    - build-essential
    - network-manager
    - dkms
    - emacs-nox
    - ubuntu-desktop-minimal
  package_update: true
  package_upgrade: true
  late-commands:
    # Changing from networkd to NetworkManager
    # move existing config out of the way
    - find /target/etc/netplan/ -name "*.yaml" -exec sh -c 'mv "$1" "$1-orig"' _ {} \;
    # Create a new netplan and enable it
    - |
      cat <<EOF | sudo tee /target/etc/netplan/01-netcfg.yaml
      network:
        version: 2
        renderer: NetworkManager
      EOF
    - curtin in-target --target /target netplan generate
    - curtin in-target --target /target netplan apply
    - curtin in-target --target /target systemctl enable NetworkManager.service
    # Write a script that can take care of some post install setup "late-commands" cannot be interactive unfortunately"
    # - |
    #   cat <<EOF | sudo tee /target/etc/finish-install-setup.sh
    #   #!/usr/bin/env bash
    #   echo *************************
    #   echo ****  Finish Setup   ****
    #   echo ************************* 
    #   echo 'Enter the hostname for this system: '
    #   read NEW_HOSTNAME
    #   hostnamectl set-hostname \${NEW_HOSTNAME}
    #   echo
    #   echo 'Enter the timezone for this system: '
    #   echo 'America/Los_Angeles America/Denver America/Chicago America/New_York'
    #   read NEW_TIMEZONE
    #   timedatectl set-timezone \${NEW_TIMEZONE}
    #   echo *************************
    #   echo
    #   echo *************************
    #   echo 'Restarting to finish ...'
    #   shutdown -r 3
    #   EOF
    # - curtin in-target --target /target chmod 744 /etc/finish-install-setup.sh
    - cp /target/cdrom/extras/Puget_Systems.png /target/usr/share/backgrounds/
  user-data: # Commands here run during first boot (cannot be interactive)
    runcmd:
      # Install the NVIDIA driver from the ppa we setup earlier
      - [apt-get, update]
      - [apt-get, dist-upgrade, --yes]
      - [apt, autoremove, --yes]
      - [apt-get, install, --yes,  nvidia-driver-470] #, --no-install-recommends]
      - [sudo, -u, ubuntu, dbus-launch, gsettings, set, org.gnome.desktop.background, picture-uri, file:///usr/share/backgrounds/Puget_Systems.png]
      # - |
      #   #!/usr/bin/env bash
      #   echo ''
      #   echo '***************************************'
      #   echo ' To complete install setup please run, '
      #   echo ' sudo /etc/finish-install-setup.sh'
      #   echo '***************************************'
      #   echo ''

user-data-server

#cloud-config
autoinstall:
  version: 1
  refresh-installer:  # start with an up-to-date installer
  update: yes
  interactive-sections:  # Install groups listed here will wait for user input
    - storage
  storage:  # should set the interactive default but doesn't seem to work??
    layout:
      name: direct
  locale: en_US.UTF-8
  keyboard:
    layout: us
  identity:  # This is section you may want to add to interactive-sections  (user name and password are ubuntu here)
    hostname: puget-000
    password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
    username: ubuntu
  ssh:
    allow-pw: true
    install-server: true
  apt:
    sources:
      ignored1:  # This is here to get the yaml formatting right when adding a ppa
        source: ppa:graphics-drivers/ppa
  packages: 
    - build-essential
    - network-manager
    - dkms
    - emacs-nox
    #- ubuntu-desktop-minimal^
  package_update: true
  package_upgrade: true
  late-commands:
    # Changing from networkd to NetworkManager
    # move existing config out of the way
    - find /target/etc/netplan/ -name "*.yaml" -exec sh -c 'mv "$1" "$1-orig"' _ {} \;
    # Create a new netplan and enable it
    - |
      cat <<EOF | sudo tee /target/etc/netplan/01-netcfg.yaml
      network:
        version: 2
        renderer: NetworkManager
      EOF
    - curtin in-target --target /target netplan generate
    - curtin in-target --target /target netplan apply
    - curtin in-target --target /target systemctl enable NetworkManager.service
    # Write a script that can take care of some post install setup "late-commands" cannot be interactive unfortunately"
    # - |
    #   cat <<EOF | sudo tee /target/etc/finish-install-setup.sh
    #   #!/usr/bin/env bash
    #   echo *************************
    #   echo ****  Finish Setup   ****
    #   echo ************************* 
    #   echo 'Enter the hostname for this system: '
    #   read NEW_HOSTNAME
    #   hostnamectl set-hostname \${NEW_HOSTNAME}
    #   echo
    #   echo 'Enter the timezone for this system: '
    #   echo 'America/Los_Angeles America/Denver America/Chicago America/New_York'
    #   read NEW_TIMEZONE
    #   timedatectl set-timezone \${NEW_TIMEZONE}
    #   echo *************************
    #   echo
    #   echo *************************
    #   echo 'Restarting to finish ...'
    #   shutdown -r 3
    #   EOF
    # - curtin in-target --target /target chmod 744 /etc/finish-install-setup.sh
    - ls -l > /target/root/ls.out
    - pwd > /target/root/pwd.out
    - mount > /target/root/mount.out
    - touch afile
    - cp afile /target/root/
    - ls -l / > /target/root/ls-root.out
    - ls -l /target > /target/root/ls-target.out
    - ls -l /target/cdrom > /target/root/ls-target-cdrom.out
  user-data: # Commands here run during first boot (cannot be interactive)
    runcmd:
      # Install the NVIDIA driver from the ppa we setup earlier
      - [apt-get, install, --yes,  nvidia-driver-470, --no-install-recommends]
      - |
        #!/usr/bin/env bash
        echo ''
        echo '***************************************'
        echo ' To complete install setup please run, '
        echo ' sudo /etc/finish-install-setup.sh'
        echo '***************************************'
        echo ''

is there any example of setting basic LVM ?

is there any example of setting basic LVM ?

This stuff is really hard to write manually. The easiest solution I've found is to configure one machine manually, then use the storage section from the file generated at /var/log/installer/autoinstall-user-data.

that's the hard part - it does not shows the structure I created in logs, and documentation is fuzzy and not clear at all: Not with examples and not with explanation itself

In general I find cloud init difficult to use, but preseed wasn't simple either. When I did my post (where this gist came from) I had to do a lot of trial and error. But, I think the more it gets used folks will work out the rough edges. I'm going to try it again in a few weeks with Ubuntu 22.04 and write up another post. Hopefully it will be less of a struggle to use this next time :-) (I'm not betting on that)

What cantoni suggested is probably the best way to get a storage config worked out.

• Do a basic auto install with c-i using interactive storage config
• Log into that system and pull the config from /var/log/installer/autoinstall-data
• Use that in your new autoinstall-user-data file

I found this post which gives a more detailed example of doing exactly that.

https://www.golinuxcloud.com/generate-user-data-file-ubuntu-20-04/

The storage section with LVM config is below. You would probably need to modify something like that to be more general, and there may some parts that can be left out. But it would be a good start for a workable config.

  storage:
    config:
    - ptable: gpt
      serial: 3600508b1001c576619b6670156e25877
      wwn: '0x600508b1001c576619b6670156e25877'
      path: /dev/sda
      wipe: superblock
      preserve: false
      name: ''
      grub_device: true
      type: disk
      id: disk-sda
    - device: disk-sda
      size: 1048576
      flag: bios_grub
      number: 1
      preserve: false
      grub_device: false
      type: partition
      id: partition-3
    - device: disk-sda
      size: 1073741824
      wipe: superblock
      flag: ''
      number: 2
      preserve: false
      grub_device: false
      type: partition
      id: partition-4
    - fstype: ext4
      volume: partition-4
      preserve: false
      type: format
      id: format-2
    - device: disk-sda
      size: 899074228224
      wipe: superblock
      flag: ''
      number: 3
      preserve: false
      grub_device: false
      type: partition
      id: partition-5
    - name: ubuntu-vg
      devices:
      - partition-5
      preserve: false
      type: lvm_volgroup
      id: lvm_volgroup-1
    - name: ubuntu-lv
      volgroup: lvm_volgroup-1
      size: 107374182400B
      wipe: superblock
      preserve: false
      type: lvm_partition
      id: lvm_partition-1
    - fstype: ext4
      volume: lvm_partition-1
      preserve: false
      type: format
      id: format-3
    - path: /
      device: format-3
      type: mount
      id: mount-3
    - path: /boot
      device: format-2
      type: mount
      id: mount-2

first of thank you for the answer.
I had chat with the golinux post writer, and he didn't have too much information to share either, yet I was able to find eventually a small example that enabled me to build my LVM as follows below:

   config:
     - {ptable: msdos, wipe: superblock-recursive, preserve: false, grub_device: true, path: /dev/ROOT_DEV, type: disk, id: disk-sda}
     - {type: partition, wipe: superblock-recursive, number: 1, device: disk-sda, flag: bios_grub, size: 1M, id: sda-grub}
     - {type: partition, wipe: superblock-recursive, number: 2, device: disk-sda, flag: boot, size: 1G, id: sda-boot}
     - {type: partition, wipe: superblock-recursive, number: 3, device: disk-sda, size: -1, id: sda-lvm}
     - {type: lvm_volgroup, name: vg-my, devices: [sda-lvm], id: vg-my}
     - {type: lvm_partition, volgroup: vg-my, id: lv-root, name: lv-root, size: 15G}
     - {type: lvm_partition, volgroup: vg-my, id: lv-var, name: lv-var, size: 10G}
     - {type: lvm_partition, volgroup: vg-my, id: lv-var-lib-docker, name: lv-var-lib-docker, size: 20G}
     - {type: lvm_partition, volgroup: vg-my, id: lv-var-log, name: lv-var-log, size: 10G}
     - {type: lvm_partition, volgroup: vg-my, id: lv-var-log-audit, name: lv-var-log-audit, size: 5G}
     - {type: lvm_partition, volgroup: vg-my, id: lv-var-tmp, name: lv-var-tmp, size: 10G}
     - {type: lvm_partition, volgroup: vg-my, id: lv-home, name: lv-home, size: -1} # this means to use all the left storage
     - {type: format, fstype: ext4, volume: sda-boot, id: sda-boot-fs}
     - {type: format, fstype: ext4, volume: lv-root, id: lv-root-fs}
     - {type: format, fstype: ext4, volume: lv-home, id: lv-home-fs}
     - {type: format, fstype: ext4, volume: lv-var, id: lv-var-fs}
     - {type: format, fstype: ext4, volume: lv-var-lib-docker, id: lv-var-lib-docker-fs}
     - {type: format, fstype: ext4, volume: lv-var-log, id: lv-var-log-fs}
     - {type: format, fstype: ext4, volume: lv-var-log-audit, id: lv-var-log-audit-fs}
     - {type: format, fstype: ext4, volume: lv-var-tmp, id: lv-var-tmp-fs}
     - {type: mount, path: /, id: m-root, device: lv-root-fs}
     - {type: mount, path: /boot, id: m-boot, device: sda-boot-fs}
     - {type: mount, path: /home, id: m-home, device: lv-home-fs}
     - {type: mount, path: /var, id: m-var, device: lv-var-fs}
     - {type: mount, path: /var/lib/docker, id: m-varllibdocker, device: lv-var-lib-docker-fs}
     - {type: mount, path: /var/log, id: m-varlog, device: lv-var-log-fs}
     - {type: mount, path: /var/log/audit, id: m-var-log-audit, device: lv-var-log-audit-fs}
     - {type: mount, path: /var/tmp, id: m-vartmp, device: lv-var-tmp-fs}

ROOT_DEV is a variable that i am substituting with sed with shell script that am running with early-commands of cloud-init. it was only way I was able to make it dynamic.I have tried to use manual install c-i file, but for some reason, it does not work, mainly lvm, write-file and additional parameters for the user, were not implemented with c-i.
Anyway - thanks again for response and guidance.

Thanks for posting that back here! That's a nice use of early-commands and good reference for a difficult aspect of using c-i

Hi all. @dbkinghorn I used your How-to make a custom iso using cloud-init for a on prem server. We would like to simplify installing a crashed server obviously. Everything is fine and thank you for that. I cannot find any other how to and I have searched.
I have a few questions (because I am also new to yaml & cloud-init) and was wondering if you would assist?
My first question is simple: Does cloud-init (c-i) allow adding and installing from other repositories at initilal install with a yaml like yours?
Question 2, can I add multiple users at the same time during initial install.
Thanks in advance.

Does cloud-init (c-i) allow adding and installing from other repositories at initilal install with a yaml like yours?
yes: you can add apt tag with address to your repository e.g.

apt:
    sources:
        source: deb.address.to.your.repo

Thanks for the info @zero-pytagoras

Hello, @zero-pytagoras !
ROOT_DEV is a variable that i am substituting with sed with shell script that am running with early-commands of cloud-init.
Could you tell me more about how to use this variable and give me please some examples of it.
Thank you very much!

I am using shell script that runs sed command to substitute the ROOT_DEV as part of early_commands tags.
essentially i run this script

ROOT_DEV=$(lsblk -x TYPE|grep disk|awk '{print $1}')

sed -i 's/ROOT_DEV/${ROOT_DEV}/g'

as part of user-data file

- early-commands:
  - swap-drive-name.sh  

for my use case that was good enough, but might not suit a system where you have more then one drive

@zero-pytagoras: After I have successfully added the repo, how do I update it, or install the software from it? In my case, I want to install the latest postgresql-client.

Hello, @zero-pytagoras !
What file do you change seed to make to work the variable in the config section?
let's give an example, it doesn't work

#cloud-config
autoinstall:
  version: 1
  early-commands:
    - /bin/bash /tmp/hdd.sh
    - /usr/bin/echo "Disk for install - $DISK"

  storage:
    layout:
      name: lvm
      match:
        path: $DISK

Thank you very much!

@vgervais
use tag packages and add to it list of packages you would like to install, for example:

apt:
  repo: link-toyour-repo
packages:
  - ipython
  - postgresql-client

@finalls
not sure i understand what you are trying to do. who is giving you value of $DISK ?
in my case i used early-commands to run script and to edit with sed the user-data file itself. kind of editing script from i am running myself

@zero-pytagoras Thanks for the info. My yaml is now working 100%.

@zero-pytagoras Thanks for the info. My yaml is now working 100%.

@finalls not sure i understand what you are trying to do. who is giving you value of ? in my case i used to run script and to edit with the user-data file itself. kind of editing script from i am running myself$DISK``early-commands``sed

#cloud-config
autoinstall:
early-commands:
- ROOT_DEV=$(lsblk -x TYPE|grep disk|grep G|awk '{print $1}')
it is my script , but the ROOT_DEV does not worker ? why

because it is yaml - it can not run shell scripts directly.
you need to run it separate file/script and redirect its value to your current user-data file

@zero-pytagoras
I'm also trying to define a path using a script in the early commands. Would you know which user-data file to edit with sed so that it is used in the installation? I've tried some of the files in
/var/lib/cloud/instances/nocloud and /var/lib/cloud/instance
but I've had no luck yet.

the default file to be update needs to /automation, but you could test it on failed run.

the default file to be update needs to /automation, but you could test it on failed run.

I needed to update the /autoinstall.yaml but thanks a lot for pointing me in the right direction! Works great now!

Thank you so much for this template.