cloud-init "nocloud" networking setup

README.md

cloud-init is absolute cancer. Its code is horrible. It has no documentation at all.

It took me 5 fucking hours to figure out how to properly configure networking on recent cloud-init (Ubuntu 16.04 cloud image) with local datasource.

It's not mentioned anywhere you need to provide dsmode: local. (but only if you need network-config, besides that everything is fine; someone below noted that -m flag does the same thing, good to know) Of course nobody needs documentation for network-config format either. (cloudinit/net/__init__.py is a protip, enjoy the feces dive)

Oh, and by the way - no, it's not possible to provide network-config to uvt-kvm without patching shit.

Use -N flag for cloud-localds for network-config.

meta-data

dsmode: local
# ↑ THIS. SHIT.

network-config

---
version: 1
config:
 - type: physical
   name: ens3
   subnets:
    - control: auto
      type: static
      address: |
         192.168.21.37
             pre-up echo xD > /tmp/yesitsthereyo # quality configuration injection, real thing

user-data

#cloud-config
chpasswd:
  list: |
    root:password
  expire: False
write_files:
  - content: |
    You don't really need that, just a bloody test.
  path: /root/test.txt

Thank you for posting this! You literally saved our day. The Cloud Init project's documentation is utter crap right now. Unfortunately, since it's open-source, the usual response is "so work on the docs" :P

You say:

Use -H flag for cloud-localds for network-config.

But the -H flag for cloud-localds is to set the hostname -

How do I get the 'network-config' section above into cloud-localds? Do I just append it to meta-data?
(That would make sense, since it starts with '----', the yaml separator)

j

Instead of network-config, you can pass dsmode: local and network-interfaces config to uvt-kvm using the --meta-data option.

I am using Ubuntu 16.04 server on a Raspberry Pi from here, I believe the cloud-init configuration is keeping a lot of my network configs from working such as creating a VLAN interface.

Just a note that I do have the kernel module for 802.1q loaded.

Everything works fine if I manually configure the VLAN interfaces after boot:

ip link add link eth0 name eth0.13 type vlan id 13
ip addr add 192.168.13.11/24 dev eth0.13
ip link set eth0.13 up

networking.service fails if I configure the interface in /etc/network/interfaces.d/10-eth0.cfg:

auto eth0
iface eth0 inet static
  address 192.168.1.11
  netmask 255.255.255.0
  gateway 192.168.1.1

auto eth0.13
iface eth0.13 inet static
    vlan-raw-device eth0
    address 192.168.13.12
    netmask 255.255.255.0

So my question is where should I change the meta-data and user-data files or add the network-config file?

There are multiple files in the /var/lib/cloud/ directory:

root@shawn-node02:/var/lib/cloud# tree

├── data
│   ├── instance-id
│   ├── previous-datasource
│   ├── previous-hostname
│   ├── previous-instance-id
│   ├── result.json
│   └── status.json
├── handlers
├── instance -> /var/lib/cloud/instances/nocloud
├── instances
│   └── nocloud
│       ├── boot-finished
│       ├── cloud-config.txt
│       ├── datasource
│       ├── handlers
│       ├── obj.pkl
│       ├── scripts
│       ├── sem
│       │   ├── config_apt_configure
│       │   ├── config_apt_pipelining
│       │   ├── config_byobu
│       │   ├── config_ca_certs
│       │   ├── config_chef
│       │   ├── config_disk_setup
│       │   ├── config_fan
│       │   ├── config_grub_dpkg
│       │   ├── config_keys_to_console
│       │   ├── config_landscape
│       │   ├── config_locale
│       │   ├── config_lxd
│       │   ├── config_mcollective
│       │   ├── config_mounts
│       │   ├── config_ntp
│       │   ├── config_package_update_upgrade_install
│       │   ├── config_phone_home
│       │   ├── config_power_state_change
│       │   ├── config_puppet
│       │   ├── config_rightscale_userdata
│       │   ├── config_rsyslog
│       │   ├── config_runcmd
│       │   ├── config_salt_minion
│       │   ├── config_scripts_per_instance
│       │   ├── config_scripts_user
│       │   ├── config_scripts_vendor
│       │   ├── config_seed_random
│       │   ├── config_set_hostname
│       │   ├── config_set_passwords
│       │   ├── config_snap_config
│       │   ├── config_snappy
│       │   ├── config_ssh
│       │   ├── config_ssh_authkey_fingerprints
│       │   ├── config_ssh_import_id
│       │   ├── config_timezone
│       │   ├── config_ubuntu_init_switch
│       │   ├── config_users_groups
│       │   ├── config_write_files
│       │   └── consume_data
│       ├── user-data.txt
│       ├── user-data.txt.i
│       ├── vendor-data.txt
│       └── vendor-data.txt.i
├── scripts
│   ├── per-boot
│   ├── per-instance
│   ├── per-once
│   └── vendor
├── seed
│   └── nocloud-net
│       ├── meta-data
│       └── user-data
└── sem
    ├── config_apt_pipelining.once
    └── config_scripts_per_once.once

and in /etc/cloud:

root@shawn-node02:/etc/cloud# tree
.
├── build.info
├── cloud.cfg
├── cloud.cfg.d
│   ├── 05_logging.cfg
│   ├── 90_dpkg.cfg
│   ├── 99-fake_cloud.cfg
│   └── README
└── templates
    ├── chef_client.rb.tmpl
    ├── hosts.debian.tmpl
    ├── hosts.freebsd.tmpl
    ├── hosts.redhat.tmpl
    ├── hosts.suse.tmpl
    ├── ntp.conf.debian.tmpl
    ├── ntp.conf.fedora.tmpl
    ├── ntp.conf.rhel.tmpl
    ├── ntp.conf.sles.tmpl
    ├── ntp.conf.ubuntu.tmpl
    ├── resolv.conf.tmpl
    ├── sources.list.debian.tmpl
    └── sources.list.ubuntu.tmpl

Thank you in advance for any advice or help, sorry for such a long post, I wanted to make sure I shared as much information as possible.

Regards,
Shawn

@jowolf yup, I made a mistake, thanks for noting that, it was supposed to be -N...
@willbryant meta-data - fine, but network-config is (at least was back when I wrote this document) supposed to be a separate file, and that was missing from uvt-kvm. After all I just made my own tooling for that.

Too bad gist doesn't notify about comments by default...

maybe this helps another person stumbling over this.. http://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html#datasource-nocloud

I feel your frustration bro... im definitely getting a stroke few years earlier than expected after working with juju and cloud-init

You're lucky you figured it out in 5 hours. I've been struggling with this for 2 days now trying to figure out why my network config is being ignored. I didn't need the cloud-localds -m or to set dsmode: local to get it to work though. Specifying cloud-localds --network-config=... did the trick for me. This is what I ended up with if it helps anyone else: https://github.com/brennancheung/playbooks/tree/master/cloud-init-lab

+1 about cancer. I never managed to figure out where the hell it got its name from - always being reset to a wrong value in /etc/hosts when I restarted the container in lxc.

Finally I gave up (I hate to say this) and just set

preserve_hostname: true
manage_etc_hosts: false

in /etc/cloud/cloud.cfg

+1111 on the cancer. Just wasted 2 hours of my life on this, even following all the tips above still nothing. Giving up and fucking consoling in to set a static IP the manual way.

Fuck it this shit works in AWS / Azure and even good old Kickstart and I'm not a fucking KVM admin thank fucking God

EDIT aaaaaand the issue was... drumroll.... completely remove and purge cloud-init from a baseline ubuntu 18.04 server iso, re-install the package, and then it works. So whatever it shipped with is borked (for networking only, my user-data went fine which was so frustrating). Got it from another comment somewhere!

I've started a ansilbe role to did my labs more confortable and I suffering with this configuration for ubuntu 16.04 , ubuntu don't get the static ips and it's so problematic.

I hope don't have any trouble with old and good kickstart.

After I was posted my frustation here this exactly configuration bellow works for me without any extra tags.

network-interfaces: |
auto ens3
iface ens3 inet static
address 192.168.15.201
netmask 255.255.255.0
gateway 192.168.15.1

Just want to express my deepest gratitude, I've spent several days fighting with this on vsphere. Thanks!

How do I get the --network-config option in cloud-localds? I am in Centos7 and it said option not recognized when run cloud-localds. Do I need to patch it and where I can get the patch? thanks!

@barrowkwan, I am also having the same issue in CentOS 7. is it working for you or still struggling? I have spent full day but no luck.
Anyone please help. Thanks.

In 2022 the docs still suck. There's one ridiculously complex example using every possible option and a three liner for dhcp.

I also was stumped by this but if you delete the /etc/cloud/cloud.cfg.d/99-fake-cloud.cfg (I may have the name wrong slightly) file, Cloud-Init will then respect your user-data network configuration. I came about the solution from following @wintermute000's advice (Neuromancer is my favorite book BTW!) about deleting and re-installing cloud-config and noticed that file is missing when you do that.

Thank you everyone for your help with this! It definitely pointed me in the right direction. I was chasing my tail with Proxmox VE with a custom version 2 (Netplan using the systemd-networkd renderer) network-config on an Ubuntu 22.04 VM. Everything pointed to cloud-init being the problem, but despite modifying the PVE code to add the dsmode: local setting to the meta-data-generating source code, it just did not work. What I learned was cloud-init would tell netplan to render the networkd configuration to /run/systemd/network but it wouldn't actually start the network. It depended on systemd to do that.

I found out my network-config was just bad. I was trying to use set-name: eth0 without matching on a MAC address. Which strangely wouldn't work on bootup, but would work by running netplan apply manually.

If you find your network just isn't coming up, check out what networkctl is telling you and see if the interface is managed or not. Check your configuration and the system journal and make sure systemd-networkd is happy.

@chris-mccoy I'm running into a similar issues to what you were experiencing.

I found out my network-config was just bad. I was trying to use set-name: eth0 without matching on a MAC address. Which strangely wouldn't work on bootup, but would work by running netplan apply manually.

One thing I've found is instead of matching on mac address it's simpler to match by name

#cloud-config

network:
  version: 2
  ethernets:
    eth0:
      set-name: eth0
      match:     
        name: en*

@chris-mccoy

Thank you everyone for your help with this! It definitely pointed me in the right direction. I was chasing my tail with Proxmox VE with a custom version 2 (Netplan using the systemd-networkd renderer) network-config on an Ubuntu 22.04 VM. Everything pointed to cloud-init being the problem, but despite modifying the PVE code to add the dsmode: local setting to the meta-data-generating source code, it just did not work. What I learned was cloud-init would tell netplan to render the networkd configuration to /run/systemd/network but it wouldn't actually start the network. It depended on systemd to do that.

According to the proxmox docs (see --citype) the nocloud datasource is used for linux which doesn't support the dsmode setting. Only the configdrive datasource supports dsmode: local as far as I can tell.

My workaround has been to conditionally call netplan apply from the bootcmd which is called early in the (source) cloud-init execution order. That way the package sources are updated on first boot and packages can be installed as standard.

#cloud-config

bootcmd:
  - if [ ! -f /root/first-run-complete ]; then netplan apply; fi

runcmd:
  - touch /root/first-run-complete

apt:
  preserve_sources_list: true
package_update: true
package_upgrade: true
package_reboot_if_required: true
packages:
  - git

@StudioLE nice, thank you! 🙏

@StudioLE
I want to set a static ip address with nocloud but that doesn't work, on proxmox.
I'm mounting the cloud-init data using an ISO cd drive.
Do you have a workaround?

I have put this in metadata and user data, but that doesn't work.

network:
    version: 2
    ethernets:
        eth0:
            match:     
              name: en*
            set-name: eth0
            addresses: [10.10.10.84/32]
            gateway4: 10.10.10.1
            nameservers:
              addresses:
                - 8.8.8.8
                - 8.8.4.4

@mcbenjemaa

Instead of relying on cloud-init's network configuration I've opted to write the netplan configuration from the user-config.ymlusing write_files and it's applied from a per-instance script.

Full disclosure though I have ditched Proxmox in favour of Multipass but the cloud-init logic will be the same.

Here are the relevant lines:

#cloud-config

write_files:
- path: /etc/netplan/10-custom.yaml
  content: |
    network:
      version: 2
      ethernets:
        extra0:
          dhcp4: no
          match:
            macaddress: C8:36:54:11:11:11
          addresses:
          - 10.0.1.2/24
          - fc00::0:1:2/112
          routes:
          - to: default
            via: 10.0.1.1
            metric: 50
          - to: default
            via: fc00::0:1:1
            metric: 50
- path: /var/lib/cloud/scripts/per-instance/30-network-configure
  permissions: 0o500
  content: |
    #!/bin/bash
    set -uo pipefail

    echo-success () {
      echo -e "\e[32m ✔  $1\e[0m" >&2
    }

    echo-information () {
      echo -e "\e[34m ⓘ  $1\e[0m" >&2
    }

    echo-error () {
      echo -e "\e[31m !  $1\e[0m" >&2
    }

    echo-warning () {
      echo -e "\e[33m ⚠  $1\e[0m" >&2
    }

    echo-subsidiary () {
      echo -e "\e[37m    $1\e[0m" >&2
    }

    echo-step () {
      echo -e "\e[0m »  $1\e[0m" >&2
    }

    echo-step "Applying Netplan configuration"
    if netplan apply
    then
      echo-success "Applied Netplan configuration"
    else
      echo-error "Failed to apply netplan configuration"
    fi

@StudioLE I want to set a static ip address with nocloud but that doesn't work, on proxmox. I'm mounting the cloud-init data using an ISO cd drive. Do you have a workaround?

I have put this in metadata and user data, but that doesn't work.

network:
    version: 2
    ethernets:
        eth0:
            match:     
              name: en*
            set-name: eth0
            addresses: [10.10.10.84/32]
            gateway4: 10.10.10.1
            nameservers:
              addresses:
                - 8.8.8.8
                - 8.8.4.4

The indentation isn't consistent in your example, I'm not sure if that may be causing issues? In some places you're using 4 spaces and others 2 spaces.

Also worth noting that gateway4: is deprecated you should use default routes instead.

Rewriting your example would look something like this:

network:
  version: 2
  ethernets:
    eth0:
      match:     
        name: en*
      set-name: eth0
      addresses: 
      - 10.10.10.84/32
      routes:
      - to: default
        via: 10.10.10.1
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4

@mcbenjemaa

I think the CIDR you're using for your address is incorrect too. 10.10.10.84/32 should be 10.10.10.84/24 It should match the network range of the gateway whereas /32 is restricted to just a single IP.

Netplan example:
https://github.com/canonical/netplan/blob/main/examples/static.yaml#L7
Docs:
https://netplan.readthedocs.io/en/stable/netplan-tutorial/#using-static-ip-addresses

should i put this in the network-config file?

The network-config is not working, I only see the default cloud-init dhcp config.

the script you provided adds the 10-custom.yaml but there's still 50-cloud-init.yaml by default.

it doesn't change the IP, should I disable it?

@StudioLE , thanks for your suggestion.

I have this working now. 💯

#cloud-config
hostname: proxmox-test-control-plane
manage_etc_hosts: true
user: ionos
chpasswd:
  expire: false
users:
  - default
package_upgrade: true

write_files:
  - path: /etc/cloud/cloud.cfg.d/99-custom-networking.cfg
    permissions: '0644'
    content: |
      network: {config: disabled}

  - path: /etc/netplan/99-custom.yaml
    content: |
      network:
        version: 2
        renderer: networkd
        ethernets:
          eth0:
            dhcp4: no
            addresses:
            - 10.10.10.66/24
            routes:
            - to: default
              via: 10.10.10.1
            nameservers:
              addresses: [8.8.8.8, 8.8.4.4]

  - path: /var/lib/cloud/scripts/per-instance/30-network-configure
    permissions: 0o500
    content: |
        #!/bin/bash
        set -uo pipefail
        
        echo-success () {
          echo -e "\e[32m ✔  $1\e[0m" >&2
        }
        
        echo-information () {
          echo -e "\e[34m ⓘ  $1\e[0m" >&2
        }
        
        echo-error () {
          echo -e "\e[31m !  $1\e[0m" >&2
        }
        
        echo-warning () {
          echo -e "\e[33m ⚠  $1\e[0m" >&2
        }
        
        echo-subsidiary () {
          echo -e "\e[37m    $1\e[0m" >&2
        }
        
        echo-step () {
          echo -e "\e[0m »  $1\e[0m" >&2
        }
        
        echo-step "Applying Netplan configuration"
        if netplan apply
        then
          echo-success "Applied Netplan configuration"
        else
          echo-error "Failed to apply netplan configuration"
        fi

@StudioLE I want to set a static ip address with nocloud but that doesn't work, on proxmox. I'm mounting the cloud-init data using an ISO cd drive. Do you have a workaround?
I have put this in metadata and user data, but that doesn't work.

network:
    version: 2
    ethernets:
        eth0:
            match:     
              name: en*
            set-name: eth0
            addresses: [10.10.10.84/32]
            gateway4: 10.10.10.1
            nameservers:
              addresses:
                - 8.8.8.8
                - 8.8.4.4

The indentation isn't consistent in your example, I'm not sure if that may be causing issues? In some places you're using 4 spaces and others 2 spaces.

Also worth noting that gateway4: is deprecated you should use default routes instead.

Rewriting your example would look something like this:

network:
  version: 2
  ethernets:
    eth0:
      match:     
        name: en*
      set-name: eth0
      addresses: 
      - 10.10.10.84/32
      routes:
      - to: default
        via: 10.10.10.1
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4

Thank you!
Saved me hours of hair pulling trying to build images without dhcpd.