Skip to content

Instantly share code, notes, and snippets.

@pcn
Last active August 29, 2015 14:05
Show Gist options
  • Save pcn/d18de4d40413e9e3c42f to your computer and use it in GitHub Desktop.
Save pcn/d18de4d40413e9e3c42f to your computer and use it in GitHub Desktop.
Nova and Neutron commands

In simple cases, you've created a network but dhcp and network ranges aren't showing up in the web ui. For that, re-start the dhcp agent:

sudo /etc/init.d/neutron-dhcp-agent

When dhcp doesn't seem to be working, or when the uuids in the logs and device names in the logs on the hypervisors don't seem to match existing names of tunnels and bridges, do this:

stop the dhcp agent, stop any dnsmasq processes, remove the ports
 (ovs-vsctl del-port br-int tap83ba5659-b4 and also in neutron),
 delete the namespace, start dhcp agent

So deleting all internal, external, and tunnels on the controller seems to be a good thing, then re-starting the services on the controller (and re-running chef-runs was part of my latest attempt) seems to be the right thing to do.

Jeez.

root@ubuntu:/etc/neutron# nova boot --flavor 1 --image b3f14546-0aa3-46f4-ad04-749f9554d31b --availability-zone nova --nic net-id=778d312b-2fdd-4c4e-a718-0dbaf55d48ec foo

Get the network IDs from the PoV of nova:

root@ubuntu:/etc/neutron# nova net-list
+--------------------------------------+-------------------+------+
| ID                                   | Label             | CIDR |
+--------------------------------------+-------------------+------+
| 7848df6d-315d-450b-bfc4-9409d947b991 | 172-16-3-external | -    |
| 778d312b-2fdd-4c4e-a718-0dbaf55d48ec | 172-16-2-internal | -    |
+--------------------------------------+-------------------+------+

And the image:

root@ubuntu:/etc/neutron# nova image-list
+--------------------------------------+--------+--------+--------+
| ID                                   | Name   | Status | Server |
+--------------------------------------+--------+--------+--------+
| b3f14546-0aa3-46f4-ad04-749f9554d31b | cirros | ACTIVE |        |
+--------------------------------------+--------+--------+--------+

List the flavors:

root@ubuntu:/etc/neutron# nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1  | m1.tiny   | 512       | 1    | 0         |      | 1     | 1.0         | True      |
| 2  | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      |
| 3  | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      |
| 4  | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      |
| 5  | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+

On a single-node install

/etc/network/interfaces needs to be reconfigured so that eth0 is a bridged port on the br-ex bridge, or it’ll lose access to the network once in a while.

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
# iface eth0 inet static
#       address 192.168.144.21
#       netmask 255.255.240.0
#       broadcast 192.168.159.254
#       gateway 192.168.144.1
#       dns-nameservers 8.8.8.8 8.8.4.4
iface eth0 inet manual
        up ifconfig $IFACE up

auto br-ex
iface br-ex inet static
        address 192.168.144.21
        netmask 255.255.240.0
        network 192.168.144.0
        broadcast 192.168.159.254
        gateway 192.168.144.1
        dns-nameservers 8.8.8.8 8.8.4.4
        bridge_ports eth0
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off

Cleaning and re-creating an installation:

Remove all interfaces from neutron networks

neutron router-gateway-clear ext-to-int
neutron router-interface-delete ext-to-int 172-16-2

Remove all of the neutron networks

neutron net-delete $(neutron net-list -f csv --quote minimal | tail -1 | cut -d, -f1)

Remove a floating ip (happens via the nova API)

nova floating-ip-delete 172.16.21.20

Remove all routers (use this iteratively until it fails, for instance):

neutron router-delete $(neutron router-list -f csv --quote minimal | tail -1 | cut -d, -f1)

At times, the config gets really jacked. You need to:

Stop all neutron services on the controller

for s in /etc/init.d/neutron-* ; do $s stop; done

Remove ovs bridges and interfaces (only bridge interfaces)

ovs-vsctl del-port qr-284298ee-89

That doesn’t work for me. I can also get desparate and shutdown the openvswitch-switch service on both the controller and the client systems, then run:

rm /etc/openvswitch/conf.db.*

Start all neutron services on the controller again

for s in /etc/init.d/neutron-* ; do $s start; done

Creating a new network

neutron net-create 172-16-2-internal
neutron net-create 172-16-3-external -- --router:external=True

Now create and allocate IP addresses

neutron subnet-create --name '172-16-2' --allocation-pool start=172.16.2.20,end=172.16.2.240 --dns-nameserver 192.168.94.1 172-16-2-internal 172.16.2.0/24
neutron subnet-create --name '172-16-3' --allocation-pool start=172.16.3.20,end=172.16.3.240 --dns-nameserver 192.168.94.1 --enable_dhcp=False 172-16-3-external 172.16.3.0/24

Create a router, and associate the internal and external networks via this router

neutron router-create ext-to-int
neutron router-gateway-set ext-to-int 172-16-3-external
neutron router-interface-add  ext-to-int 172-16-2

We're going to be using the controller as a NAT concentrator for now. Based on the tutorial at http://openstack.redhat.com/Networking_in_too_much_detail this should work?

Add the router to the default namespace's br-ex interface:

ip addr add 172.16.3.2/24 dev br-ex

Then handle iptables:

iptables -A FORWARD -d 172.16.3.0/24 -j ACCEPT 
iptables -A FORWARD -s 172.16.3.0/24 -j ACCEPT 
iptables -t nat -I POSTROUTING 1 -s 172.3.0/24 -j MASQUERADE

Last add a security group that allows ssh in.

nova secgroup-add-rule default tcp  22 22 0.0.0.0/0

That'll make the default secuirty group allow port 22.

Cleaning and re-creating an installation:

Remove all interfaces from neutron networks

root@ubuntu:~# neutron router-gateway-clear ext-to-int
Removed gateway from router ext-to-int
root@ubuntu:~# neutron router-interface-delete ext-to-int 172-16-2
Removed interface from router ext-to-int.

Remove all of the neutron networks

neutron net-delete $(neutron net-list -f csv --quote minimal | tail -1 | cut -d, -f1)

Remove a floating ip (happens via the nova API)

nova floating-ip-delete 172.16.21.20

Remove all routers (use this iteratively until it fails, for instance):

neutron router-delete $(neutron router-list -f csv --quote minimal | tail -1 | cut -d, -f1)

Kill neutron dhcp agents, kill dnsmasq daemon, delete all bridges and tunnels, re-start all neutron services per the other note in this gist.

Creating a new network

For testing, let’s start with pairs of networks at 172.16.{even,odd}.0/24. The even will be internal and the odd will be external.

For example, 172.16.2.0/24 and 172.16.3.0/24 would provide 172.16.2.0/24 as internal ip addresses for instances, and 172.16.3.0/24 would provide external addresses.

For each /24, dhcp will allocate addresses from .20-.240 dynamically and as NATable floating IPs.

root@ubuntu:/var/log/nova# neutron net-create 172-16-2-internal
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 38dc0ca5-a517-4d89-907a-10006e4c42a9 |
| name                      | 172-16-2-internal                    |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 8                                    |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | 42f00e87dd1d433384c977f01baf8bca     |
+---------------------------+--------------------------------------+
root@ubuntu:/var/log/nova# neutron net-create 172-16-3-external -- --router:external=True
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 2323d480-5586-40cb-b90e-e95e394b8dff |
| name                      | 172-16-3-external                    |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 7                                    |
| router:external           | True                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | 42f00e87dd1d433384c977f01baf8bca     |
+---------------------------+--------------------------------------+

Now create and allocate IP addresses

root@ubuntu:/var/log/nova# neutron subnet-create --name '172-16-2' --allocation-pool start=172.16.2.20,end=172.16.2.240 --dns-nameserver 192.168.94.1 172-16-2-internal 172.16.2.0/24
Created a new subnet:
+------------------+-------------------------------------------------+
| Field            | Value                                           |
+------------------+-------------------------------------------------+
| allocation_pools | {"start": "172.16.2.20", "end": "172.16.2.240"} |
| cidr             | 172.16.2.0/24                                   |
| dns_nameservers  | 192.168.94.1                                    |
| enable_dhcp      | True                                            |
| gateway_ip       | 172.16.2.1                                      |
| host_routes      |                                                 |
| id               | fc32b323-04c5-447c-a621-10dd7640dbb6            |
| ip_version       | 4                                               |
| name             | 172-16-2                                        |
| network_id       | 38dc0ca5-a517-4d89-907a-10006e4c42a9            |
| tenant_id        | 42f00e87dd1d433384c977f01baf8bca                |
+------------------+-------------------------------------------------+
root@ubuntu:/var/log/nova# neutron subnet-create --name '172-16-3' --allocation-pool start=172.16.3.20,end=172.16.3.240 --dns-nameserver 192.168.94.1 --enable_dhcp=False 172-16-3-external 172.16.3.0/24
Created a new subnet:
+------------------+-------------------------------------------------+
| Field            | Value                                           |
+------------------+-------------------------------------------------+
| allocation_pools | {"start": "172.16.3.20", "end": "172.16.3.240"} |
| cidr             | 172.16.3.0/24                                   |
| dns_nameservers  | 192.168.94.1                                    |
| enable_dhcp      | False                                           |
| gateway_ip       | 172.16.3.1                                      |
| host_routes      |                                                 |
| id               | 4682ca4e-4643-49c2-864a-5d2e9cd47805            |
| ip_version       | 4                                               |
| name             | 172-16-3                                        |
| network_id       | 2323d480-5586-40cb-b90e-e95e394b8dff            |
| tenant_id        | 42f00e87dd1d433384c977f01baf8bca                |
+------------------+-------------------------------------------------+

Create a router, and associate the internal and external networks via this router

root@ubuntu:/var/log/nova# neutron router-create ext-to-int
Created a new router:
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| admin_state_up        | True                                 |
| external_gateway_info |                                      |
| id                    | 29d18b31-b047-46c5-b512-0df5d09de193 |
| name                  | ext-to-int                           |
| status                | ACTIVE                               |
| tenant_id             | 42f00e87dd1d433384c977f01baf8bca     |
+-----------------------+--------------------------------------+
root@ubuntu:/var/log/nova# neutron router-gateway-set ext-to-int 172-16-3-external
Set gateway for router ext-to-int
root@ubuntu:/var/log/nova# neutron router-interface-add  ext-to-int 172-16-2
Added interface e6f16494-25cd-4e77-9d17-277ff510d5e7 to router ext-to-int.

Re-building from complete scratch

If reverting to a snapshot that still requires a base chef run, the following should be done:

  • Re-install openvswitch 2.3.0

  • Make sure to install some extra packages:

$ sudo apt-get install conntrack module-assistant debhelper ipsec-tools racoon python-twisted-web  html2t
ext po-debconf intltool-debian

The key above is conntrack. External router port status can’t be determined without it.

I got a snapshot of a dev config from fyb3r that confirmed that a lot of the documentation out there for havana contained un-necessary extra work in /etc/network/interfaces, etc. when using ovs and 3 interfaces. Here are the pastes (s)he provided:

/etc/network/interfaces to go with the below:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 10.1.10.155
        netmask 255.255.255.0
        gateway 10.1.10.99
        broadcast 10.1.10.255
        dns-nameservers 10.1.10.10 8.8.8.8

auto eth1
iface eth1 inet static
        address 10.1.11.155
        netmask 255.255.255.0
        network 10.1.11.0

# The external network interface
auto eth2
iface eth2 inet manual
        up ip link set dev $IFACE up
        down ip link set dev $IFACE down

output of ovs-vsctl show, displaying which interfaces are in use, and how

69b4faef-c40d-4874-bbb3-953b91dad0ab
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth2"
            Interface "eth2"
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.0.2"

showing ml2_conf.ini, showing the IP address for eth1 in use for ovs

    [ml2]
# (ListOpt) List of network type driver entrypoints to be loaded from
# the neutron.ml2.type_drivers namespace.
#
type_drivers = gre,local,flat,vlan,vxlan
# Example: type_drivers = flat,vlan,gre,vxlan

# (ListOpt) Ordered list of network_types to allocate as tenant
# networks. The default value 'local' is useful for single-box testing
# but provides no connectivity between hosts.
#
tenant_network_types = gre
# Example: tenant_network_types = vlan,gre,vxlan

# (ListOpt) Ordered list of networking mechanism driver entrypoints
# to be loaded from the neutron.ml2.mechanism_drivers namespace.
mechanism_drivers = openvswitch,l2population,linuxbridge
# Example: mechanism drivers = openvswitch,mlnx
# Example: mechanism_drivers = arista
# Example: mechanism_drivers = cisco,logger
# Example: mechanism_drivers = openvswitch,brocade
# Example: mechanism_drivers = linuxbridge,brocade

[ml2_type_flat]
# (ListOpt) List of physical_network names with which flat networks
# can be created. Use * to allow flat networks with arbitrary
# physical_network names.
#
# flat_networks =
# Example:flat_networks = physnet1,physnet2
# Example:flat_networks = *

[ml2_type_vlan]
# (ListOpt) List of <physical_network>[:<vlan_min>:<vlan_max>] tuples
# specifying physical_network names usable for VLAN provider and
# tenant networks, as well as ranges of VLAN tags on each
# physical_network available for allocation as tenant networks.
#
# network_vlan_ranges =
# Example: network_vlan_ranges = physnet1:1000:2999,physnet2

[ml2_type_gre]
# (ListOpt) Comma-separated list of <tun_min>:<tun_max> tuples enumerating ranges of GRE tunnel IDs that are available for tenant network allocation
tunnel_id_ranges = 1:1000

[ovs]
local_ip = 10.1.11.155
tunnel_type = gre
enable_tunneling = True

[agent]
tunnel_types = gre,vxlan
l2_population = True

[ml2_type_vxlan]
# (ListOpt) Comma-separated list of <vni_min>:<vni_max> tuples enumerating
# ranges of VXLAN VNI IDs that are available for tenant network allocation.
#
# vni_ranges =

# (StrOpt) Multicast group for the VXLAN interface. When configured, will
# enable sending all broadcast traffic to this multicast group. When left
# unconfigured, will disable multicast VXLAN mode.
#
# vxlan_group =
# Example: vxlan_group = 239.1.1.1

[securitygroup]
# Controls if neutron security group is enabled or not.
# It should be false when you use nova security group.
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True

Totally preventable things I've done that cost me time

  • After each reboot, neutron seems to need to be restarted once - I guess it gives up on contacting rabbitmq. (!?!?! why not retry more?). This causes an error when logging into the web GUI. /etc/init.d/neutron-server restart
  • [ESX specific] the esx host is in US/Pacific, the VM is in UTC, every reboot the clock is off (I should just give up and put the VM into US/Pacific). apt-get install ntp; /etc/init.d/ntp stop; ntpdate 0.pool.ntp.org; /etc/init.d/ntp start
  • After re-starting neutron-server, gotta give it a minute, then re-start the l3 agent and the dhcp agent for the appropriate net namespaces to be created, and for any needed dhcp servers to start up and router interfaces to get going. service neutron-l3-agent restart, service neutron-dhcp-agent restart
  • After running the chef cookbook, eth1 may be placed on the br-ex bridge on the controller node, which makes it unreachable. ovs-vsctl del-port br-ex eth1.
  • If eth1 was placed on br-ex, eth2 won't be on br-ex. ovs-vsctl add-port br-ex eth2
  • Eth1 need to be manually added to br-ex, and eth2 needs to be brought up post-install
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment