Docker - Open vSwitch setup

docker-ovs.md

Running Docker on two hosts, bridged with Open vSwitch

This Gist shows how to use Open vSwitch to bridge Docker containers on two hosts. It is based on this blog post http://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/.

A similar Gist using Tinc instead of Open vSwitch is available: https://gist.github.com/noteed/11031504.

Setup

To prepare a host (e.g. a new Digital Ocean Ubuntu 14.04 droplet), simply run the install.sh script:

# wget https://gist.githubusercontent.com/noteed/8656989/raw/install.sh
# sh install.sh

Once done, the two first variables of the shared-docker-network.sh script must be changed, and the script can be run:

# vim shared-docker-network.sh
# sh shared-docker-network.sh

Repeat the steps on a second host.

Route

See the other Gist using Tinc to solve a problem where Docker allocates the same IP addresses on both hosts.

Useful debugging commands

root@node-1:~# ip a s
root@node-1:~# ip r s
root@node-1:~# brctl show
root@node-1:~# ovs-vsctl show
root@node-1:~# ping <other host>
root@node-1:~# ping <other host bridge>

install.sh

# For lxc-docker.
echo deb http://get.docker.io/ubuntu docker main \
  > /etc/apt/sources.list.d/docker.list
apt-key adv \
  --keyserver keyserver.ubuntu.com \
  --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

apt-get update
apt-get install -q -y vim openvswitch-switch bridge-utils lxc-docker
echo 'DOCKER_OPTS="--bridge=docker0"' >> /etc/default/docker
service docker restart

# This Gist.
wget https://gist.githubusercontent.com/noteed/8656989/raw/shared-docker-network.sh

echo Edit shared-docker-network.sh and run it.

shared-docker-network.sh

# From http://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/

# Edit this variable: the 'other' host.
REMOTE_IP=188.226.138.185

# Edit this variable: the bridge address on 'this' host.
BRIDGE_ADDRESS=172.16.42.1/24

# Name of the bridge (should match /etc/default/docker).
BRIDGE_NAME=docker0

# bridges

# Deactivate the docker0 bridge
ip link set $BRIDGE_NAME down
# Remove the docker0 bridge
brctl delbr $BRIDGE_NAME
# Delete the Open vSwitch bridge
ovs-vsctl del-br br0
# Add the docker0 bridge
brctl addbr $BRIDGE_NAME
# Set up the IP for the docker0 bridge
ip a add $BRIDGE_ADDRESS dev $BRIDGE_NAME
# Activate the bridge
ip link set $BRIDGE_NAME up
# Add the br0 Open vSwitch bridge
ovs-vsctl add-br br0
# Create the tunnel to the other host and attach it to the
# br0 bridge
ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre options:remote_ip=$REMOTE_IP
# Add the br0 bridge to docker0 bridge
brctl addif $BRIDGE_NAME br0

# iptables rules

# Enable NAT
iptables -t nat -A POSTROUTING -s 172.16.42.0/24 ! -d 172.16.42.0/24 -j MASQUERADE
# Accept incoming packets for existing connections
iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Accept all non-intercontainer outgoing packets
iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
# By default allow all outgoing traffic
iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT

# Restart Docker daemon to use the new BRIDGE_NAME
service docker restart

I am trying to get a multiple host docker setup and have followed you blog post "http://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/" and also this gist

Based on that I am able to ping the containers from each host [and across the host], so network connectivity between the different containers spread over the 2 hosts seems to work properly. [PS: My Host is Ubuntu 14.04 and my docker version is Docker version 1.0.1, build 990021a

After that I setup some containers and here is what I observe, kindly let me know if I am missing something or is this a limitation of the setup.

When I try ssh from one container from Host A to another container on Host B, or just try to ssh from Host A to any container on Host B, the ssh seems to be stuck. A typical output looks like this

ssh -v zookeeper01
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /home/admin/.ssh/config
debug1: /home/admin/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to zookeeper01 [10.0.5.1] port 22.
debug1: Connection established.
debug1: identity file /home/admin/.ssh/id_rsa type -1
debug1: identity file /home/admin/.ssh/id_rsa-cert type -1
debug1: identity file /home/admin/.ssh/id_dsa type -1
debug1: identity file /home/admin/.ssh/id_dsa-cert type -1
debug1: identity file /home/admin/.ssh/id_ecdsa type -1
debug1: identity file /home/admin/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/admin/.ssh/id_ed25519 type -1
debug1: identity file /home/admin/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent

After that it is stuck at that point.

More importantly I see the same issues on different application similar to SSH.

you response and any feedback/guideline would be really great and appreciated.

Recently got a mail from Caleb Crane who faced the same issue and apparently has solved the issue. Am quoting it here, so that the rest of us can use it if faced with the issue. The solution and all credits go to Caleb Crane.

"
I got it working in my environment. It turned out to be necessary to set the MTU to 1420 in the containers. GRE doesn’t support fragmentation so when ssh tried to send a frame at 1500 bytes the client wasn’t receiving the entire thing.
"

I can confirm that --mtu=1420 is required.

see this great tutorial for setting up a "MULTI-HOST DOCKER NETWORK" : https://wiredcraft.com/blog/multi-host-docker-network/

It is also being specific about MTU.