Skip to content

Instantly share code, notes, and snippets.

@dannysauer

dannysauer/gist:b4f8ba1a021a7488d137c7fc14a7d20b

Last active August 3, 2018 14:52
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dannysauer/b4f8ba1a021a7488d137c7fc14a7d20b to your computer and use it in GitHub Desktop.
Save dannysauer/b4f8ba1a021a7488d137c7fc14a7d20b to your computer and use it in GitHub Desktop.
Download ZIP
Overriding network namespace
Raw
gistfile1.md

To override name resolution, create a mount namespace and bind mount a special resolv.conf over the top of the original. The IP address will be the first address in the range given to the virtual network. I used 192.168.68.0/22, so the DNS server (dnsmasq) is started by KVM on 192.168.68.1. You can "discover" that by getting the information for the network, then looking at the IP address assigned to the bridge device:

sauer@lightning:~> sudo virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 caasp-dev-net        active     no            yes

sauer@lightning:~> sudo virsh net-info caasp-dev-net
Name:           caasp-dev-net
UUID:           4fb06ff8-897d-4683-b13c-478f5572a51c
Active:         yes
Persistent:     yes
Autostart:      no
Bridge:         virbr0

sauer@lightning:~> ip addr show dev virbr0
89: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:50:d6:3a brd ff:ff:ff:ff:ff:ff
    inet 192.168.68.1/22 brd 192.168.71.255 scope global virbr0
       valid_lft forever preferred_lft forever

Having gotten the IP address (which could easily be put into a script using virsh xml output and adding -o to the ip command), create an alternate resolv.conf which will override the system file. I put this in /etc/netns because I was originally going to use ip netns (which dind't pan out), but a more reasonable place to put it is within the devel directory structure with the other terraform vars and stuff.

sauer@lightning:~/kubic/automation> umask 022
sauer@lightning:~/kubic/automation> sudo mkdir -p /etc/netns/caasp_dev/
[sudo] password for root:
sauer@lightning:~/kubic/automation> { echo 'nameserver 192.168.68.1'; echo 'search devenv.caasp.suse.net'; } | sudo tee /etc/netns/caasp_dev/resolv.conf
nameserver 192.168.68.1
search devenv.caasp.suse.net
sauer@lightning:~/kubic/automation> sudo unshare -m sh -c 'mount --bind /etc/netns/caasp_dev/resolv.conf /etc/resolv.conf; sudo -u $SUDO_USER host admin'
admin.devenv.caasp.suse.net has address 192.168.69.0
sauer@lightning:~/kubic/automation> host admin
Host admin not found: 3(NXDOMAIN)

Note that nost and nslookup go straight to the DNS server in resolv.conf, while the nss gethost*() syscalls will check nscd first. That means just about everything other than nslookup will still fail if the "bad" result is still cached. :) So you probably also need to disable nscd. An easy way to do that is to also map the socket to /dev/null with another bind mount.

Here's ping with and without nscd bypassed:

sauer@lightning:~/kubic/automation> sudo unshare -m sh -c 'mount --bind /etc/netns/caasp_dev/resolv.conf /etc/resolv.conf; sudo -u $SUDO_USER ping -c 1 admin.devenv.caasp.suse.net'
PING admin.devenv.caasp.suse.net (10.17.1.0) 56(84) bytes of data.

--- admin.devenv.caasp.suse.net ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

sauer@lightning:~/kubic/automation> sudo unshare -m sh -c 'mount --bind /etc/netns/caasp_dev/resolv.conf /etc/resolv.conf; mount --bind /dev/null /var/run/nscd/socket; sudo -u $SUDO_USER ping -c 1 admin.devenv.caasp.suse.net'
PING admin.devenv.caasp.suse.net (192.168.69.0) 56(84) bytes of data.
64 bytes from admin.devenv.caasp.suse.net (192.168.69.0): icmp_seq=1 ttl=64 time=0.344 ms

--- admin.devenv.caasp.suse.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.344/0.344/0.344/0.000 ms

Bootstrap fails without the altered resolv.conf:

sauer@lightning:~/kubic/automation> ./caasp-devenv --bootstrap
>>> [caasp-devenv] Bootstrap CaaSP Environment
>>> [velum-bootstrap] Running Velum Interactions

Register user and configure cluster
>>> Registering user
  User registers (FAILED - 1)

Failures:

  1) Register user and configure cluster User registers
     Failure/Error: visit "/users/sign_up"

     Capybara::Poltergeist::StatusFailError:
       Request to 'https://admin.devenv.caasp.suse.net/users/sign_up' failed to reach server, check DNS and/or server status
     # ./.bundler/ruby/2.5.0/gems/poltergeist-1.15.0/lib/capybara/poltergeist/browser.rb:377:in `command'
     # ./.bundler/ruby/2.5.0/gems/poltergeist-1.15.0/lib/capybara/poltergeist/browser.rb:36:in `visit'
     # ./.bundler/ruby/2.5.0/gems/poltergeist-1.15.0/lib/capybara/poltergeist/driver.rb:97:in `visit'
     # ./.bundler/ruby/2.5.0/gems/capybara-2.14.4/lib/capybara/session.rb:269:in `visit'
     # ./.bundler/ruby/2.5.0/gems/capybara-2.14.4/lib/capybara/dsl.rb:50:in `block (2 levels) in <module:DSL>'
     # ./spec/support/helpers.rb:32:in `block in register'
     # ./spec/support/helpers.rb:11:in `with_status_ok'
     # ./spec/support/helpers.rb:31:in `register'
     # ./spec/features/01-setup-velum.rb:17:in `block (3 levels) in <top (required)>'
     # ./spec/support/helpers.rb:6:in `with_screenshot'
     # ./spec/features/01-setup-velum.rb:16:in `block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:142:in `block (3 levels) in <top (required)>'
     # ./spec/spec_helper.rb:142:in `block (2 levels) in <top (required)>'

Finished in 4.17 seconds (files took 9.95 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/features/01-setup-velum.rb:15 # Register user and configure cluster User registers

With resolv.conf overridden and nscd effectively disabled:

sauer@lightning:~/kubic/automation> sudo unshare -m sh -c 'mount --bind /etc/netns/caasp_dev/resolv.conf /etc/resolv.conf; mount --bind /dev/null /var/run/nscd/socket; sudo -u $SUDO_USER ./caasp-devenv --bootstrap'
>>> [caasp-devenv] Bootstrap CaaSP Environment
>>> [velum-bootstrap] Running Velum Interactions

Register user and configure cluster
>>> Registering user
<<< User registered
  User registers
>>> User logs in
<<< User logged in
>>> Setting up velum
>>> Disabling Tiller
<<< Velum set up
  User configures the cluster

Boostrap cluster
>>> User logs in
<<< User logged in
>>> Reaching discovery page
<<< Ready to start discovery
  User reaches minion discovery page
>>> User logs in
<<< User logged in
>>> Wait until all 3 minions are pending to be accepted
<<< All minions are pending to be accepted
>>> Wait for accept-all button to be enabled
<<< accept-all button enabled
>>> Click to accept all minion keys
>>> Waiting 120 seconds as a workaround
<<< Waiting 120 seconds as a workaround
>>> Wait until Minion keys are accepted by salt (Timeout: 240)
<<< Minion keys accepted in Velum
>>> Waiting until Minions are accepted in Velum
<<< Minions accepted in Velum
  User accepts all minions
>>> User logs in
<<< User logged in
>>> Waiting for page to settle
<<< Page has settled
>>> Selecting minion roles
<<< Minion roles selected
>>> Confirm roles selection
  User selects minion roles
>>> User logs in
<<< User logged in
>>> Configuring last settings
<<< Last settings configured
>>> Bootstrapping cluster
>>> Wait until orchestration is complete (Timeout: 3600)

Hooray!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment