Skip to content

Instantly share code, notes, and snippets.

@dlresende

dlresende/why_om_ui_is_not_accessible.md

Last active October 20, 2018 12:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dlresende/024a1e9d8d9f87998c0c97565a5600f3 to your computer and use it in GitHub Desktop.
Save dlresende/024a1e9d8d9f87998c0c97565a5600f3 to your computer and use it in GitHub Desktop.
Download ZIP
Debug VirtualBox not correctly forwarding traffic on a multi-user machine
Raw
why_om_ui_is_not_accessible.md

Problem

Cannot access OM running inside VirtualBox

Facts

box

  • The box has 2 interfaces one for each IP range

    eth0      Link encap:Ethernet  HWaddr 08:00:27:00:fb:33
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8246 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5784 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:651908 (651.9 KB)  TX bytes:628051 (628.0 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:4d:7a:44
          inet addr:192.168.50.50  Bcast:192.168.50.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

  • SSHing into the box results in an SSH connections being established (the box can then correctly establish connections with the outside world)

    vagrant@opsmanager-2-3:~$ netstat -tn 2>/dev/null
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        1      0 127.0.0.1:52602         127.0.0.1:8080          CLOSE_WAIT
tcp        1      0 127.0.0.1:52614         127.0.0.1:8080          CLOSE_WAIT
tcp        0      0 127.0.0.1:44904         127.0.0.1:5432          ESTABLISHED
tcp        1      0 127.0.0.1:52608         127.0.0.1:8080          CLOSE_WAIT
tcp        0    180 10.0.2.15:22            10.0.2.2:62768          ESTABLISHED
tcp        0      0 127.0.0.1:5432          127.0.0.1:44904         ESTABLISHED

  • There are no iptables rules inside the box dropping inbound packages (so packages might have being dropped outside the box)

    vagrant@opsmanager-2-3:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (2 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

mac

  • The domain name pcf.releng.rocks resolves correctly to 192.168.50.50 (so no issues with DNS resolution)

    ± dlr |master ?:1 ✗| → dig +short pcf.releng.rocks
192.168.50.50

  • packages sent to range 192.168.50 are handled by vboxnet interface

    ± dlr |master ?:1 ✗| → netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            192.168.1.1        UGSc           92        4     en0
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH              4    55749     lo0
169.254            link#10            UCS             0        0     en0
192.168.1          link#10            UCS             1        0     en0
192.168.1.1/32     link#10            UCS             1        0     en0
192.168.1.1        88:d2:74:89:3f:5c  UHLWIir        35     5392     en0   1199
192.168.1.105/32   link#10            UCS             0        0     en0
192.168.1.106      c4:b3:1:d9:59:f5   UHLWI           0        0     en0    339
192.168.50         link#22            UC              1        0 vboxnet
224.0.0/4          link#10            UmCS            2        0     en0
224.0.0.251        1:0:5e:0:0:fb      UHmLWI          0        0     en0
239.255.255.250    1:0:5e:7f:ff:fa    UHmLWI          0      528     en0
255.255.255.255/32 link#10            UCS             0        0     en0

Hypethesis

  1. There is 1 VirtualBox installation and 2 accounts on this machine that are using VirtualBox. Both accounts have installed the same box (vagrant-ops-manager), therefore theyt might have the same network configuration. Some IP address or network configuration might be colliding.

  • in this case, why vagrant ssh still works? Vagrant seems to be able to SSH, but direct SSH does not work:

    ± dlr |master ?:1 ✗| → ssh -i .vagrant/machines/default/virtualbox/private_key vagrant@192.168.50.50
ssh: connect to host 192.168.50.50 port 22: Host is down

2018-10-20 12:23:08 ⌚ ruby 2.5.1p57 (system) diegos-mbp-2 in ~/workspace/vagrant-ops-manager
± dlr |master ?:1 ✗| → vagrant ssh
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.15.0-33-generic x86_64)

* Documentation:  https://help.ubuntu.com
* Management:     https://landscape.canonical.com
* Support:        https://ubuntu.com/advantage
Last login: Sat Oct 20 10:41:05 UTC 2018 from 10.0.2.2 on pts/1
Last login: Sat Oct 20 11:23:17 2018 from 10.0.2.2

    Vagrant seems to have some sort of priviledged route to SSH, like a tunnel, or GuestAdditions, etc.

    ± dlr |master ?:1 ✗| → vagrant ssh --debug
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
DEBUG virtualbox_5_2:   - [1, "ssh", 2222, 22, "127.0.0.1"]
DEBUG ssh: Checking key permissions: /Users/pcf-rabbitmq/workspace/vagrant-ops-manager/.vagrant/machines/default/virtualbox/private_key
 INFO ssh: Invoking SSH: /usr/bin/ssh ["vagrant@127.0.0.1", "-p", "2222", "-o", "LogLevel=FATAL", "-o", "Compression=yes", "-o", "DSAAuthentication=yes", "-o", "IdentitiesOnly=yes", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "-o", "IdentityFile=\"/Users/pcf-rabbitmq/workspace/vagrant-ops-manager/.vagrant/machines/default/virtualbox/private_key\""]
 Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.15.0-33-generic x86_64)

  • what happens when the box is shut down and restarted? OM seems not to be responding still after a restart.

  • what happens when the box is shut down, we quit VirtualBox, we reopen VirtualBox and we restart the box? Interestingly, some VirtualBox process seem to still be running for an account which is not the account being used (the one presenting these issues):

    ± dlr |master ?:1 ✗| → ps aux | grep VBox
pcf-redis        57080  13.4  0.1  4441336  23364   ??  S    11:39am   4:45.79 /Applications/VirtualBox.app/Contents/MacOS/VBoxSVC --auto-shutdown
pcf-redis        57074  11.8  0.0  4327916  10908   ??  S    11:39am   3:33.44 /Applications/VirtualBox.app/Contents/MacOS/VBoxXPCOMIPCD
pcf-rabbitmq     40937   0.0  0.0  4267792   1048 s006  S+   12:31pm   0:00.00 grep --color=auto VBox

± dlr |master ?:1 ✗| → whoami
pcf-rabbitmq

    Trying to kill these processes does not work, they keep coming back. Trying to restart the machine.

    After a reboot everything worked fine and I could access OM running inside the box. Here are the VirtualBox processess I have now:

    ± dlr |master U:2 ✗| → ps aux | grep -i vbox
pcf-rabbitmq      5481   7.0  3.8  5705884 1258500   ??  S    12:45pm   0:20.80 /Applications/VirtualBox.app/Contents/MacOS/VBoxHeadless --comment vagrant-ops-manager_default_1539988030446_79541 --startvm 710bd607-36c6-4107-9b0c-b65d817a2c6b --vrde config
pcf-rabbitmq      4121   0.1  0.1  4416668  18064   ??  S    12:45pm   0:00.31 /Applications/VirtualBox.app/Contents/MacOS/VBoxSVC --auto-shutdown
pcf-rabbitmq      7611   0.0  0.0  4267792   1056 s001  S+   12:46pm   0:00.00 grep --color=auto -i vbox
pcf-rabbitmq      5557   0.0  0.0  4322152  12044   ??  S    12:45pm   0:00.02 /Applications/VirtualBox.app/Contents/MacOS/VBoxNetDHCP --ip-address 192.168.56.100 --lower-ip 192.168.56.101 --mac-address 08:00:27:5B:2A:05 --netmask 255.255.255.0 --network HostInterfaceNetworking-vboxnet0 --trunk-name vboxnet0 --trunk-type netadp --upper-ip 192.168.56.254
pcf-rabbitmq      4119   0.0  0.0  4295148  10684   ??  S    12:45pm   0:00.09 /Applications/VirtualBox.app/Contents/MacOS/VBoxXPCOMIPCD

Conclusion

It seems having the VBoxSVC process running as another user was causing the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment