Skip to content

Instantly share code, notes, and snippets.

@makuk66
Created June 14, 2011 09:25
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save makuk66/1024578 to your computer and use it in GitHub Desktop.
Save makuk66/1024578 to your computer and use it in GitHub Desktop.
Notes about using point-to-point links for routing IP addresses in virtual machines
Some experiments, which worked but were eventually not used.
See http://www.greenhills.co.uk/2011/06/10/lxc.html
Now, the next big milestone is to get networking working.
In the config, we told lxc-create to use br0 as the link,
and inside the container we have eth0:
root@thunder:/# brctl show br0
bridge name bridge id STP enabled interfaces
br0 8000.962e6bb72480 no vethP70LNx
root@thunder:/# ip link list br0
7: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
link/ether 96:2e:6b:b7:24:80 brd ff:ff:ff:ff:ff:ff
root@natty1:/# ip link list eth0
68: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether ee:4d:fd:34:c2:59 brd ff:ff:ff:ff:ff:ff
By way of initial test I did a simple internal routing setup akin to [this description](http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg01969.html),
to a RFC1918 address on the container.
root@thunder:/# ifconfig br0 up
root@thunder:/# ip route add 192.168.0.0/24 dev br0
root@thunder:/# ip addr add dev br0 172.16.0.1/24 brd +
root@thunder:/# ip route add to 192.168.0.99 via 172.16.0.2
root@thunder:/# echo 1 > /proc/sys/net/ipv4/ip_forward
root@natty1:/# ip addr add 192.168.0.99/32 dev eth0
root@natty1:/# ip route add default via 172.16.0.1
root@natty1:/# ping 46.43.35.202
PING 46.43.35.202 (46.43.35.202): 48 data bytes
56 bytes from 46.43.35.202: icmp_seq=0 ttl=64 time=0.047 ms
root@thunder:/# ping 192.168.0.99
PING 192.168.0.99 (192.168.0.99) 56(84) bytes of data.
64 bytes from 192.168.0.99: icmp_req=1 ttl=64 time=9.26 ms
and after copying /etc/resolv.conf from the controlling host to the container and
starting sshd in the container (<code>service sh start</code>) I could <code>ssh 192.168.0.99</code> in.
Now with the actual IP addresses I've been allocated by my ISP:
root@natty1:/# ip addr add 46.43.55.73 dev eth0
root@thunder:/# ip route add to 46.43.55.73 via 172.16.0.2
This allows me to ssh in over the internet.
Next problem: making outbound connections (e.g. ping www.google.com) fails.
tcpdump on the controlling host shows:
14:15:13.182730 IP 172.16.0.2 > ez-in-f105.1e100.net: ICMP echo request, id 49409, seq 512, length 56
Clearly that's not going to work; it needs to have 46.43.55.73.
You can force that with (from the console, not an ssh session):
root@natty1:/# ip route del default dev eth0
root@natty1:/# ip route add default via 172.16.0.1 dev eth0 src 46.43.55.73
or by the looks of it by making the internal address non-global:
root@natty1:/# ip addr del 172.16.0.2/4 dev eth0
root@natty1:/# ip addr add 172.16.0.2/4 brd + scope link dev eth0
root@natty1:/# ip route del default
root@natty1:/# ip route add default via 172.16.0.1 dev eth0
root@natty1:/# ping -q -c 1 www.google.com
PING www.l.google.com (209.85.146.103): 48 data bytes
--- www.l.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 13.763/13.763/13.763/0.000 ms
This is all good, but do I need Proxy-ARP here?
It appears not: if I ping 46.43.55.74 from the internet the hosting provider duly
forwards it to the main IP address, as shown by tcpdump on the controlling host.
So, let's start with a new container, natty2 on 46.43.55.73,
and copy some more in, and optimise some.
NAME=natty2
IP=46.43.55.73
TRANSFER_SOURCE=172.16.73.2
TRANSFER_ROUTER=172.16.73.1
ip addr add dev br0 $TRANSFER_ROUTER/24 brd +
ip route add to $IP via $TRANSFER_SOURCE
echo 1 > /proc/sys/net/ipv4/ip_forward
TRANSFER_MASK=255.255.255.0
LXCDIR=/var/lib/lxc
ROOTFS=$LXCDIR/${NAME}/rootfs
CONFIG=/root/lxc-${NAME}-config.tmp
cat > $CONFIG <<EOM
lxc.network.type = veth
lxc.network.link = br0
lxc.network.name = eth0
EOM
cat > $ROOTFS/etc/network/interfaces <<EOM
auto lo
iface lo inet loopback
# public routable address
auto eth0
iface eth0 inet static
address $IP
netmask 255.255.255.255
# transfer network
auto eth0:0
iface eth0:0 inet static
name transfer network
address $TRANSFER_SOURCE
netmask $TRANSFER_MASK
pointopoint $TRANSFER_ROUTER
post-up ip route add default via $TRANSFER_ROUTER dev eth0 src $IP
pre-down ip route del default dev eth0
EOM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment