Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@meetmatt
Last active June 27, 2023 10:25
Show Gist options
  • Star 8 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save meetmatt/da9e7364dec04a43465e20c430fb58df to your computer and use it in GitHub Desktop.
Save meetmatt/da9e7364dec04a43465e20c430fb58df to your computer and use it in GitHub Desktop.
Scripted version of the how-to article by Rodrigo Nascimento "OpenStack Single-Node (MicroStack)" https://connection.rnascimento.com/2021/03/08/openstack-single-node-microstack/ Part 2 with Kubernetes can be found here https://gist.github.com/yurgol/92167f8cc61e85346bbb97b4501d8d22

Openstack

OS configuration

Add user to sudoers without password

echo 'user ALL=(ALL) NOPASSWD: ALL' | sudo tee /etc/sudoers.d/user

Kernel optimizations

echo fs.inotify.max_queued_events=1048576 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_instances=1048576 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_watches=1048576 | sudo tee -a /etc/sysctl.conf
echo vm.max_map_count=262144 | sudo tee -a /etc/sysctl.conf
echo vm.swappiness=1 | sudo tee -a /etc/sysctl.conf

Disable ipv6

echo net.ipv6.conf.all.disable_ipv6=1 | sudo tee -a /etc/sysctl.conf
echo net.ipv6.conf.default.disable_ipv6=1 | sudo tee -a /etc/sysctl.conf
echo net.ipv6.conf.lo.disable_ipv6=1 | sudo tee -a /etc/sysctl.conf

Persist to grub:

sudo vim /etc/default/grub
# find these options and replace them with
# speed up boot
GRUB_TIMEOUT=1
# disable IPv6
GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"
GRUB_CMDLINE_LINUX="ipv6.disable=1"
sudo update-grub
sudo reboot

Remove LXD

sudo snap remove lxd

Microstack installation

Install microstack from snap

sudo snap install microstack --devmode --beta

Add CLI aliases

sudo snap alias microstack.openstack openstack
sudo snap alias microstack.ovs-vsctl ovs-vsctl

Initialize microstack services

sudo microstack init --auto --control

Persist network configuration

Configure Open vSwitch bridge

Move host IP address from the physical interface to Open vSwitch managed bridge.

Save script that restores the IP address and default route

sudo tee /usr/local/bin/microstack-br-workaround > /dev/null << EOL
#!/bin/bash
ovs-vsctl add-port br-ex enp5s0 || :
ip addr flush dev enp5s0 || :
ip address add 192.168.1.100/24 dev br-ex || :
ip link set br-ex up || :
ip route add default via 192.168.1.1 || :
EOL
sudo chmod +x /usr/local/bin/microstack-br-workaround
sudo /usr/local/bin/microstack-br-workaround

Create systemd startup service which runs the script on boot

sudo tee /etc/systemd/system/microstack-br-workaround.service > /dev/null << EOL
[Unit]
Description=Service for adding physical ip to microstack bridge
Requires=snap.microstack.external-bridge.service
After=snap.microstack.external-bridge.service

[Service]
ExecStart=/usr/local/bin/microstack-br-workaround
SyslogIdentifier=microstack-br-workaround
Restart=no
WorkingDirectory=/usr/local/bin
TimeoutStopSec=30
Type=oneshot

[Install]
WantedBy=multi-user.target
EOL

Enable the service

sudo systemctl daemon-reload
sudo systemctl enable microstack-br-workaround.service

Restore dnsmasq ability to forward DNS after network manipulations

sudo tee /etc/systemd/resolved.conf > /dev/null << EOL
[Resolve]
DNS=1.1.1.1
EOL
```shell
Restart the systemd service
```shell
sudo systemctl restart systemd-resolved.service

Reboot to test

sudo reboot

Prepare Openstack

Clean-up default networks and router

Delete default router

openstack router remove subnet test-router test-subnet
openstack router unset --external-gateway test-router
openstack router delete test-router

Delete default networks

openstack subnet  delete test-subnet external-subnet
openstack network delete test        external

Extend quotas

openstack quota set \
    --secgroups -1 \
    --cores 128 \
    --instances 100 \
    --ram 52000 \
    admin

Setup networking

Create a public network

openstack network create \
    --enable \
    --project admin \
    --external \
    --default \
    --provider-network-type flat \
    --provider-physical-network physnet1 \
    public

Subnet without DHCP:

openstack subnet create \
    --project admin \
    --subnet-range 192.168.1.0/24 \
    --no-dhcp \
    --gateway 192.168.1.1 \
    --network public \
    --allocation-pool start=192.168.1.200,end=192.168.1.250 \
    public

Create a private network

openstack network create \
    --enable \
    --project admin \
    --internal \
    private

Subnet with DHCP:

openstack subnet create \
    --project admin \
    --subnet-range 10.10.0.0/24 \
    --dhcp \
    --network private \
    private

Create the router as NAT gateway for private network

openstack router create \
    --disable \
    --project admin \
    router

Attach the router to private network:

openstack router add subnet router private

Set the router gateway through public network and enable SNAT:

openstack router set \
    --enable \
    --enable-snat \
    --external-gateway public \
    router

Pre-allocate floating IPs

for i in $(seq 1 50)
do
    openstack floating ip create public >/dev/null
done

Create keypair

ssh-keygen -q -t rsa -f ~/.ssh/id_rsa -N ''
openstack keypair create --public-key ~/.ssh/id_rsa.pub default 

Replace default security group

Delete default security group:

SEC_GROUP=$(openstack security group list --project admin -c ID -f value)
openstack security group delete $SEC_GROUP

Create allow-all security group:

openstack security group create --project admin allow

Delete default rules:

openstack security group rule list allow -f value -c ID \
    | xargs -n1 -I{} openstack security group rule delete {}

Add allow-all rules.
Ingress:

openstack security group rule create allow \
    --project admin \
    --ethertype IPv4 \
    --ingress

Egress:

openstack security group rule create allow \
    --project admin \
    --ethertype IPv4 \
    --egress

Prepare virtual machine flavors

Delete default flavours

openstack flavor list -c Name -f value \
    | xargs -n1 -I{} openstack flavor delete {}

Create new flavours

1,2,4,8 VCPUs; 1024, 2048, 4096, 8192 RAM; 5, 10, 20 disk.

for i in 1 2 4 8
do
    for j in 1024 2048 4096 8192
    do
        for k in 5 10 20
        do
            openstack flavor create "$i.$j.$k" --vcpus $i --ram $j --disk $k >/dev/null
        done
    done
done
openstack flavor list --sort-column VCPUs --sort-column RAM --sort-column Disk -c Name -c VCPUs -c RAM -c Disk

Import VM OS image

Download the cloud Ubuntu OS image

Redefine the SERIES with another Ubuntu release (e.g. bionic, xenial) if necessary.

SERIES=focal
wget https://cloud-images.ubuntu.com/${SERIES}/current/${SERIES}-server-cloudimg-amd64.img

Create the image to Openstack Glance

openstack image create ubuntu.${SERIES} \
      --public \
      --disk-format=qcow2 \
      --container-format=bare \
      --property os_distro='ubuntu' \
      --file=${SERIES}-server-cloudimg-amd64.img

TODO

Install Openstack Barbican, Barbican Vault, Octavia

https://docs.openstack.org/barbican/ussuri/
https://docs.openstack.org/octavia/ussuri/

Add DNS support

https://docs.openstack.org/designate/latest/intro/index.html

TODO: investigate possibility to use let's encrypt

Test Openstack

Launch instance

openstack server create \
    --image ubuntu.focal \
    --flavor 8.2048.5 \
    --security-group allow \
    --key-name default \
    --network private \
    --wait \
    test

Assign floating IP

FLOAT_IP=$(openstack floating ip list -f value | grep None | head -n1 | awk '{print $2}')
openstack server add floating ip test $FLOAT_IP

SSH

ssh ubuntu@$FLOAT_IP

Kill instance

openstack server delete test
@meetmatt
Copy link
Author

meetmatt commented Jul 4, 2021

Next step: deploy Kubernetes cluster to Single Node Microstack

Using Juju: https://opendev.org/x/microstack/src/branch/master/DEMO.md

Or alternative from the series of articles about running K8S on Devstack:
https://berndbausch.medium.com/running-a-kubernetes-cluster-on-devstack-533d579bb2f9
(hosted at https://github.com/berndbausch/Devstack-Kubernetes)

And finally, probably the most involved, following offical Charmed Kubernetes on OpenStack documentation from Ubuntu:
https://ubuntu.com/kubernetes/docs/openstack-integration

@meetmatt
Copy link
Author

meetmatt commented Jul 4, 2021

Quick research note how to use netplan instead of systemd workaround from the original article.
Reference: https://askubuntu.com/questions/1278670/ovsintport-with-netplan

This is what I have on the host after Ubuntu 20.04 LTS installation (manually configured in the install GUI with static IPv4):

sudo cat /etc/netplan/00-installer-config.yaml

# This is the network config written by 'subiquity'
network:
  ethernets:
    enp5s0:
      addresses:
      - 192.168.1.100/24
      gateway4: 192.168.1.1
      nameservers:
        addresses:
        - 192.168.1.1
        - 1.1.1.1
        - 8.8.8.8
        search: []
  version: 2

And this is how the config presumably should look to reach the same result as in the article:

network:
    version: 2
    renderer: networkd
    openvswitch:
        ports:
            - [enp5s0]
    ethernets:
        enp5s0: {}
    bridges:
        br-ex:
            addresses: [192.168.1.100/24]
            interfaces: [enp5s0]
            mtu: 9000
            openvswitch: {}

Haven't tested, but I think it will remove the port to the br-int, which will definitely impact the OpenStack installation

@anazeer-netstratum
Copy link

Hi Matt,

Thanks for putting this together. The script seem to work without errors. I see the UI with right network and routers. But I still see a br-ex interface with the original 10.20.x ip address when issuing an ifconfig.

br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.20.20.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::907a:c8ff:fe5b:6348 prefixlen 64 scopeid 0x20
ether b8:2a:72:cf:cb:9e txqueuelen 1000 (Ethernet)
RX packets 14 bytes 680 (680.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 15 bytes 1146 (1.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

  Project Network Name Subnets Associated DHCP Agents Shared External Status Admin State Availability Zones Actions
  admin private private_subnet 10.10.0.0/24 0 No No Active UP - Edit Network
  admin public public_subnet 10.251.251.0/24 0 No Yes Active UP -

When creating an instance I get the following error (from UI):

Message
Build of instance 0d9c8161-849e-4e06-a68d-4ae5c9c5b8e5 aborted: Volume 160f8afa-ba68-409c-aa4f-7ddbdc184f35 did not finish being created even after we waited 0 seconds or 1 attempts. And its status is error.
Code
500

and on command line:

microstack launch cirros --name test --availability-zone nova:pmatulis-ss-mstack-2.project.serverstack
Creating local "microstack" ssh key at /home/root/snap/microstack/common/.ssh/id_microstack
Launching server ...
No Network found for test
Traceback (most recent call last):
File "/snap/microstack/233/bin/microstack", line 11, in

@meetmatt
Copy link
Author

Hi @anazeer-netstratum,

It's hard to say what happened in your case just by looking at ifconfig. Have you checked the routing table?
Also I did a restart after the last line just to check that network indeed works as expected.

I'm planning to rebuild that workaround with a proper netplan configuration.

And there's another issue btw that you may encounter later related to default ubuntu systemd resolv.conf.
The issue is that openstack, by default, will inherit the /etc/resolv.conf which just contains a link to local dnsmasq systemd service which is obviously not available in the VM's private network.
The solution is to replace the systemd service with static configuration via resolvconf snap or with some other similar workarounds.
Stay tuned for updates.

Best regards,
Matt

@meetmatt
Copy link
Author

Hey @anazeer-netstratum,

I've read your message again and actually it's alright that you still see the 10.20.x.x bridge, don't touch it!
When I was preparing this gist I noticed the same and deleted it, and then it broke everything, not sure what exactly but basically nothing was working for me.
In the initial revision of this gist I had this comment:

# remove previous IP from the bridge
# DON'T DO IT! IT WILL KILL ABILITY TO USE HORIZON
# sudo ip addr del 10.20.20.1/24 dev br-ex

May be this is somehow related to your case? Can you show the output ofip r?

Bests, Matt

@meetmatt
Copy link
Author

Netplan idea failed because it conflicts with integrated systemd script from the microstack distribution:
https://opendev.org/x/microstack/src/branch/master/snap-overlay/bin/setup-br-ex

@meetmatt
Copy link
Author

Extracted part 2 with Kubernetes to separate gist https://gist.github.com/meetmatt/92167f8cc61e85346bbb97b4501d8d22

@tomchean
Copy link

Hi meetmatt:

Thanks for your useful guide, but I encountered some problem in network.

My local LAN range is 192.168.50.0/24, and machine IP is 192.168.50.140, gateway is 192.168.50.1, and I launch a cirros instance with IP 192.168.50.212.
In cirros, I can ping 192.168.50.140, 192.168.50.250 (gateway of openstack router), but I can't ping 192.169.50.1 and 8.8.8.8

Following is my network info
image
image
221554

Should I delete the second default routing rule, or is there anything wrong in my setting.

Thanks in advance.

@protosam
Copy link

protosam commented Apr 1, 2022

Had connectivity issues because it was trying to add the route to the gateway through a 2nd nic. Had to explicitly define where the route was to go through.

ip route add default via 192.168.86.1 dev br-ex

@ada-devops
Copy link

Hi, in Ubuntu22.04 how can configure the network for Multi-Node? I try with Netplan and OVS but nothing, can u help me?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment