Skip to content

Instantly share code, notes, and snippets.

@meetmatt

meetmatt/01-openstack.md

Last active June 27, 2023 10:25
Show Gist options
  • Star 8 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save meetmatt/da9e7364dec04a43465e20c430fb58df to your computer and use it in GitHub Desktop.
Save meetmatt/da9e7364dec04a43465e20c430fb58df to your computer and use it in GitHub Desktop.
Download ZIP
Scripted version of the how-to article by Rodrigo Nascimento "OpenStack Single-Node (MicroStack)" https://connection.rnascimento.com/2021/03/08/openstack-single-node-microstack/ Part 2 with Kubernetes can be found here https://gist.github.com/yurgol/92167f8cc61e85346bbb97b4501d8d22
Raw
01-openstack.md

Openstack

OS configuration

Add user to sudoers without password

echo 'user ALL=(ALL) NOPASSWD: ALL' | sudo tee /etc/sudoers.d/user

Kernel optimizations

echo fs.inotify.max_queued_events=1048576 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_instances=1048576 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_watches=1048576 | sudo tee -a /etc/sysctl.conf
echo vm.max_map_count=262144 | sudo tee -a /etc/sysctl.conf
echo vm.swappiness=1 | sudo tee -a /etc/sysctl.conf

Disable ipv6

echo net.ipv6.conf.all.disable_ipv6=1 | sudo tee -a /etc/sysctl.conf
echo net.ipv6.conf.default.disable_ipv6=1 | sudo tee -a /etc/sysctl.conf
echo net.ipv6.conf.lo.disable_ipv6=1 | sudo tee -a /etc/sysctl.conf

Persist to grub:

sudo vim /etc/default/grub
# find these options and replace them with
# speed up boot
GRUB_TIMEOUT=1
# disable IPv6
GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"
GRUB_CMDLINE_LINUX="ipv6.disable=1"
sudo update-grub
sudo reboot

Remove LXD

sudo snap remove lxd

Microstack installation

Install microstack from snap

sudo snap install microstack --devmode --beta

Add CLI aliases

sudo snap alias microstack.openstack openstack
sudo snap alias microstack.ovs-vsctl ovs-vsctl

Initialize microstack services

sudo microstack init --auto --control

Persist network configuration

Configure Open vSwitch bridge

Move host IP address from the physical interface to Open vSwitch managed bridge.

Save script that restores the IP address and default route

sudo tee /usr/local/bin/microstack-br-workaround > /dev/null << EOL
#!/bin/bash
ovs-vsctl add-port br-ex enp5s0 || :
ip addr flush dev enp5s0 || :
ip address add 192.168.1.100/24 dev br-ex || :
ip link set br-ex up || :
ip route add default via 192.168.1.1 || :
EOL
sudo chmod +x /usr/local/bin/microstack-br-workaround
sudo /usr/local/bin/microstack-br-workaround

Create systemd startup service which runs the script on boot

sudo tee /etc/systemd/system/microstack-br-workaround.service > /dev/null << EOL
[Unit]
Description=Service for adding physical ip to microstack bridge
Requires=snap.microstack.external-bridge.service
After=snap.microstack.external-bridge.service

[Service]
ExecStart=/usr/local/bin/microstack-br-workaround
SyslogIdentifier=microstack-br-workaround
Restart=no
WorkingDirectory=/usr/local/bin
TimeoutStopSec=30
Type=oneshot

[Install]
WantedBy=multi-user.target
EOL

Enable the service

sudo systemctl daemon-reload
sudo systemctl enable microstack-br-workaround.service

Restore dnsmasq ability to forward DNS after network manipulations

sudo tee /etc/systemd/resolved.conf > /dev/null << EOL
[Resolve]
DNS=1.1.1.1
EOL
```shell
Restart the systemd service
```shell
sudo systemctl restart systemd-resolved.service

Reboot to test

sudo reboot

Prepare Openstack

Clean-up default networks and router

Delete default router

openstack router remove subnet test-router test-subnet
openstack router unset --external-gateway test-router
openstack router delete test-router

Delete default networks

openstack subnet  delete test-subnet external-subnet
openstack network delete test        external

Extend quotas

openstack quota set \
    --secgroups -1 \
    --cores 128 \
    --instances 100 \
    --ram 52000 \
    admin

Setup networking

Create a public network

openstack network create \
    --enable \
    --project admin \
    --external \
    --default \
    --provider-network-type flat \
    --provider-physical-network physnet1 \
    public

Subnet without DHCP:

openstack subnet create \
    --project admin \
    --subnet-range 192.168.1.0/24 \
    --no-dhcp \
    --gateway 192.168.1.1 \
    --network public \
    --allocation-pool start=192.168.1.200,end=192.168.1.250 \
    public

Create a private network

openstack network create \
    --enable \
    --project admin \
    --internal \
    private

Subnet with DHCP:

openstack subnet create \
    --project admin \
    --subnet-range 10.10.0.0/24 \
    --dhcp \
    --network private \
    private

Create the router as NAT gateway for private network

openstack router create \
    --disable \
    --project admin \
    router

Attach the router to private network:

openstack router add subnet router private

Set the router gateway through public network and enable SNAT:

openstack router set \
    --enable \
    --enable-snat \
    --external-gateway public \
    router

Pre-allocate floating IPs

for i in $(seq 1 50)
do
    openstack floating ip create public >/dev/null
done

Create keypair

ssh-keygen -q -t rsa -f ~/.ssh/id_rsa -N ''
openstack keypair create --public-key ~/.ssh/id_rsa.pub default

Replace default security group

Delete default security group:

SEC_GROUP=$(openstack security group list --project admin -c ID -f value)
openstack security group delete $SEC_GROUP

Create allow-all security group:

openstack security group create --project admin allow

Delete default rules:

openstack security group rule list allow -f value -c ID \
    | xargs -n1 -I{} openstack security group rule delete {}

Add allow-all rules.
Ingress:

openstack security group rule create allow \
    --project admin \
    --ethertype IPv4 \
    --ingress

Egress:

openstack security group rule create allow \
    --project admin \
    --ethertype IPv4 \
    --egress

Prepare virtual machine flavors

Delete default flavours

openstack flavor list -c Name -f value \
    | xargs -n1 -I{} openstack flavor delete {}

Create new flavours

1,2,4,8 VCPUs; 1024, 2048, 4096, 8192 RAM; 5, 10, 20 disk.

for i in 1 2 4 8
do
    for j in 1024 2048 4096 8192
    do
        for k in 5 10 20
        do
            openstack flavor create "$i.$j.$k" --vcpus $i --ram $j --disk $k >/dev/null
        done
    done
done
openstack flavor list --sort-column VCPUs --sort-column RAM --sort-column Disk -c Name -c VCPUs -c RAM -c Disk

Import VM OS image

Download the cloud Ubuntu OS image

Redefine the SERIES with another Ubuntu release (e.g. bionic, xenial) if necessary.

SERIES=focal
wget https://cloud-images.ubuntu.com/${SERIES}/current/${SERIES}-server-cloudimg-amd64.img

Create the image to Openstack Glance

openstack image create ubuntu.${SERIES} \
      --public \
      --disk-format=qcow2 \
      --container-format=bare \
      --property os_distro='ubuntu' \
      --file=${SERIES}-server-cloudimg-amd64.img

TODO

Install Openstack Barbican, Barbican Vault, Octavia

https://docs.openstack.org/barbican/ussuri/
https://docs.openstack.org/octavia/ussuri/

Add DNS support

https://docs.openstack.org/designate/latest/intro/index.html

TODO: investigate possibility to use let's encrypt

Test Openstack

Launch instance

openstack server create \
    --image ubuntu.focal \
    --flavor 8.2048.5 \
    --security-group allow \
    --key-name default \
    --network private \
    --wait \
    test

Assign floating IP

FLOAT_IP=$(openstack floating ip list -f value | grep None | head -n1 | awk '{print $2}')
openstack server add floating ip test $FLOAT_IP

SSH

ssh ubuntu@$FLOAT_IP

Kill instance

openstack server delete test
@meetmatt
Copy link
Author

Hi @anazeer-netstratum,

It's hard to say what happened in your case just by looking at ifconfig. Have you checked the routing table?
Also I did a restart after the last line just to check that network indeed works as expected.

I'm planning to rebuild that workaround with a proper netplan configuration.

And there's another issue btw that you may encounter later related to default ubuntu systemd resolv.conf.
The issue is that openstack, by default, will inherit the /etc/resolv.conf which just contains a link to local dnsmasq systemd service which is obviously not available in the VM's private network.
The solution is to replace the systemd service with static configuration via resolvconf snap or with some other similar workarounds.
Stay tuned for updates.

Best regards,
Matt

@meetmatt
Copy link
Author

Hey @anazeer-netstratum,

I've read your message again and actually it's alright that you still see the 10.20.x.x bridge, don't touch it!
When I was preparing this gist I noticed the same and deleted it, and then it broke everything, not sure what exactly but basically nothing was working for me.
In the initial revision of this gist I had this comment:

# remove previous IP from the bridge
# DON'T DO IT! IT WILL KILL ABILITY TO USE HORIZON
# sudo ip addr del 10.20.20.1/24 dev br-ex

May be this is somehow related to your case? Can you show the output ofip r?

Bests, Matt

@meetmatt
Copy link
Author

More info about netplan instead of workaround.
Found here: https://itnext.io/a-single-server-devstack-cloud-as-a-kubernetes-platform-cd30e28e405a
https://github.com/berndbausch/Devstack-Kubernetes/blob/main/00-installer-config.yaml

@meetmatt
Copy link
Author

Netplan idea failed because it conflicts with integrated systemd script from the microstack distribution:
https://opendev.org/x/microstack/src/branch/master/snap-overlay/bin/setup-br-ex

@meetmatt
Copy link
Author

Extracted part 2 with Kubernetes to separate gist https://gist.github.com/meetmatt/92167f8cc61e85346bbb97b4501d8d22

@tomchean
Copy link

Hi meetmatt:

Thanks for your useful guide, but I encountered some problem in network.

My local LAN range is 192.168.50.0/24, and machine IP is 192.168.50.140, gateway is 192.168.50.1, and I launch a cirros instance with IP 192.168.50.212.
In cirros, I can ping 192.168.50.140, 192.168.50.250 (gateway of openstack router), but I can't ping 192.169.50.1 and 8.8.8.8

Following is my network info
image
image
221554

Should I delete the second default routing rule, or is there anything wrong in my setting.

Thanks in advance.

@protosam
Copy link

protosam commented Apr 1, 2022

Had connectivity issues because it was trying to add the route to the gateway through a 2nd nic. Had to explicitly define where the route was to go through.

ip route add default via 192.168.86.1 dev br-ex

@ada-devops
Copy link

Hi, in Ubuntu22.04 how can configure the network for Multi-Node? I try with Netplan and OVS but nothing, can u help me?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment