Skip to content

Instantly share code, notes, and snippets.

@arcolife
Last active September 8, 2018 11:09
Show Gist options
  • Save arcolife/6db55d44695195c20cb5ade48f919189 to your computer and use it in GitHub Desktop.
Save arcolife/6db55d44695195c20cb5ade48f919189 to your computer and use it in GitHub Desktop.
iptables hack for openstack (no floating ip for keystone)

CFME side

iptables -A OUTPUT -t nat -d 192.0.2.1 -j DNAT --to-destination 10.16.154.6
iptables -A OUTPUT -t nat -d 172,21,0.10 -j DNAT --to-destination 10.16.154.9

OSP side

Overcloud (heat admin):

  • Floating IP: 10.16.154.141
  • Bridge: br-ex
  • Private IP (Keystone): 192.0.2.11

Undercloud:

  • Floating IP: 10.16.154.126
  • Bridge: em1
  • Private IP: 192.0.2.1

^check which interface has a floating IP for both undercloud and overcloud

Repeat following steps:

  • after sourcing stackrc for stack user of undercloud,
  • and then sourcing overcloudrc for head-admin user of overcloud-controller:

list all endpoint ports

$ for i in `openstack endpoint list | awk -F'| ' '{ print $2}' | sed 1,3d | sed '/^$/d'`; do openstack endpoint show $i | grep admin; done > endpoint_ports

extract endpoint ports

$ cat endpoint_ports |  sed '/^$/d' | awk -F':' '{ print $3}' |  sed '/^$/d' | awk -F'/| ' '{ print $1}' > ports_new

apply settings

# undercloud
$ for i in `cat ports_new`; do sudo iptables -t nat -A PREROUTING -i em1 -p tcp --dport $i -j DNAT --to-destination 192.0.2.1:$i; done
# overcloud
$ for i in `cat ports_new`; do sudo iptables -t nat -A PREROUTING -i br-ex -p tcp --dport $i -j DNAT --to-destination 192.0.2.11:$i; done

do an additional step for 5000 port (not included in ports_new from above steps)

# undercloud
$ iptables -t nat -A PREROUTING -i em1 -p tcp --dport 5000 -j DNAT --to-destination 192.0.2.1:5000
# overcloud
$ iptables -t nat -A PREROUTING -i br-ex -p tcp --dport 5000 -j DNAT --to-destination 192.0.2.11:5000

optionally, if you messup, delete ports from iptables like this:

# for undercloud
for i in {1..13}; do sudo iptables -t nat -D PREROUTING 4; done

# overcloud
for i in {1..13}; do sudo iptables -t nat -D PREROUTING 2; done
@arcolife
Copy link
Author

arcolife commented Sep 26, 2017

overcloud controller (heat admin) iptables

[heat-admin@overcloud-controller-0 ~]$ sudo iptables -t nat -L -n -v --line-numbers
Chain PREROUTING (policy ACCEPT 387K packets, 17M bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1    2253K  103M nova-api-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
2        0     0 DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:35357 to:192.0.2.11:35357
3        0     0 DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8000 to:192.0.2.11:8000
4       15   900 DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8004 to:192.0.2.11:8004
5        0     0 DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8041 to:192.0.2.11:8041
6        0     0 DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8042 to:192.0.2.11:8042
7       27  1620 DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8080 to:192.0.2.11:8080
8      120  7200 DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8774 to:192.0.2.11:8774
9      142  8520 DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8776 to:192.0.2.11:8776
10   11858  711K DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8777 to:192.0.2.11:8777
11      46  2760 DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9292 to:192.0.2.11:9292
12     247 14820 DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9696 to:192.0.2.11:9696
13    3637  218K DNAT       tcp  --  br-ex  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5000 to:192.0.2.11:5000

Chain INPUT (policy ACCEPT 19521 packets, 1308K bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 1005K packets, 60M bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1    5670K  340M nova-api-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 1005K packets, 60M bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1    5670K  340M nova-api-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
2    5673K  341M nova-postrouting-bottom  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain nova-api-OUTPUT (1 references)
num   pkts bytes target     prot opt in     out     source               destination         

Chain nova-api-POSTROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination         

Chain nova-api-PREROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination         

Chain nova-api-float-snat (1 references)
num   pkts bytes target     prot opt in     out     source               destination         

Chain nova-api-snat (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1    5670K  340M nova-api-float-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain nova-postrouting-bottom (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1    5670K  340M nova-api-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

@arcolife
Copy link
Author

undercloud iptables

[stack@gprfc043 ~]$ sudo iptables -t nat -L -n -v --line-numbers
Chain PREROUTING (policy ACCEPT 30447 packets, 2510K bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1     153K 9191K REDIRECT   tcp  --  br-ctlplane *       0.0.0.0/0            169.254.169.254      tcp dpt:80 redir ports 8775
2     194K   20M nova-api-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
3     156K 9903K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
4        0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8774 to:192.0.2.1:8774
5        0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:6385 to:192.0.2.1:6385
6        0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9000 to:192.0.2.1:9000
7        0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9696 to:192.0.2.1:9696
8        0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:35357 to:192.0.2.1:35357
9        0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8888 to:192.0.2.1:8888
10       0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8080 to:192.0.2.1:8080
11       0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8777 to:192.0.2.1:8777
12       0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5050 to:192.0.2.1:5050
13       0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8989 to:192.0.2.1:8989
14       0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8004 to:192.0.2.1:8004
15       0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9292 to:192.0.2.1:9292
16       0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8042 to:192.0.2.1:8042
17       0     0 DNAT       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5000 to:192.0.2.1:5000

Chain INPUT (policy ACCEPT 58213 packets, 4176K bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 124K packets, 8152K bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1     712K   47M nova-api-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
2     452K   27M DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 124K packets, 8152K bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1     711K   47M BOOTSTACK_MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
2     712K   47M nova-api-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
3     712K   47M nova-postrouting-bottom  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
4        0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
5        0     0 MASQUERADE  all  --  *      eth0    192.0.2.0/24         0.0.0.0/0           

Chain BOOTSTACK_MASQ (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 RETURN     all  --  *      *       192.0.2.0/24         192.168.122.1       
2        2   684 MASQUERADE  all  --  *      *       192.0.2.0/24        !192.0.2.0/24        

Chain DOCKER (2 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0           

Chain nova-api-OUTPUT (1 references)
num   pkts bytes target     prot opt in     out     source               destination         

Chain nova-api-POSTROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination         

Chain nova-api-PREROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination         

Chain nova-api-float-snat (1 references)
num   pkts bytes target     prot opt in     out     source               destination         

Chain nova-api-snat (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1     712K   47M nova-api-float-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain nova-postrouting-bottom (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1     712K   47M nova-api-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment