calico_libvirt.md

Overview

Non-OpenStack libvirt + kvm plugged into Calico as part of platform control plane. OpenStack is too complex to manage for this workload, and using containers walks the complexity line depending on the solution.

Sources Used

• CentOS 7.3
• libvirt 2.0.0
• OpenStack packages for Calico bits
  • calico-common-2.0.2-1.el7.centos.x86_64
  • calico-felix-2.0.2-1.el7.centos.x86_64
  • dnsmasq-2.72_calico1.0.0-1.el7.x86_64
  • dnsmasq-utils-2.72_calico1.0.0-1.el7.x86_64
• calicoctl v1.0.2

Involved Systems

Node Name	Node Description	Node Networks
router	linux machine playing the role of a router	192.168.100.150/24
compute	linux machine playing the role of a hypervisor	192.168.100.160/24
docker	linux machine playing the role of a docker container machine	192.168.100.170/24

Setup

• Install Cumulus Quagga packages on router device

! /etc/quagga/bgpd.conf
!
router bgp 65001
 bgp router-id 192.168.100.150
 neighbor 192.168.100.160 remote-as 65002
 neighbor 192.168.100.170 remote-as 65003
 !
 address-family ipv4 unicast
  neighbor 192.168.100.160 activate
  neighbor 192.168.100.170 activate
 exit-address-family
!
line vty
!

• Install etcd, calico-felix, calicoctl, dnsmasq, libvirt+qemu on compute node

# /etc/calico/felix.cfg
[global]
EtcdEndpoints = "http://compute.cofront.local:4001"
FelixHostname = compute.cofront.local

    # interface definition for libvirt
    <interface type='ethernet'>
      <mac address='6a:1a:c0:a8:fe:0a'/>
      <target dev='calic0a8fe0a'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <alias name='net0'/>
    </interface>

cat << EOF | ./calicoctl apply -f -
apiVersion: v1
kind: node
metadata:
  name: compute.cofront.local
spec:
  bgp:
    asNumber: 65002
    ipv4Address: 192.168.100.160
EOF

cat << EOF | ./calicoctl apply -f -
- apiVersion: v1
  kind: profile
  metadata:
    name: wide-open
    tags: []
  spec:
    egress:
    - action: allow
      destination:
        net: 0.0.0.0/0
      ipVersion: 4
      source: {}
    - action: allow
      destination:
        net: ::/0
      ipVersion: 6
      source: {}
    ingress:
    - action: allow
      destination: {}
      ipVersion: 4
      source:
        net: 0.0.0.0/0
    - action: allow
      destination: {}
      ipVersion: 6
      source:
        net: ::/0
EOF

cat << EOF | ./calicoctl apply -f -
apiVersion: v1
kind: workloadEndpoint
metadata:
  name: abc123
  workload: vm.cofront.local
  orchestrator: cfnt
  node: compute.cofront.local
  labels: {}
spec:
  interfaceName: calic0a8fe0a
  mac: 6a:1a:c0:a8:fe:0a
  ipNetworks:
  - 192.168.254.10/32
  profiles:
    - wide-open
EOF

ip tuntap add dev calic0a8fe0a mode tap
ip link set dev calic0a8fe0a up

virsh start cirros

tap interface

When bringing up an OpenStack backed VM OpenStack handles the creation of the tap interface which ensures it comes online in an UP state. When letting libvirt do this work it's observed that the interface is down and bring it up puts it in an UNKNOWN state. Using libvirt hooks it should be possible to work around this issue. A rudimentary hook is included below, but would need much more work to be production ready.

#!/usr/bin/python

import sys

from lxml import etree
from pyroute2 import IPRoute

IFPREFIXES = ['cali', 'tap']


def configure_interfaces(interfaces):
    ip = IPRoute()
    try:
        for interface in interfaces:
            ip.link('add', ifname=interface, kind='tuntap', mode='tap')
            ipif = ip.link_lookup(ifname=interface)
            ip.link('set', index=ipif[0], state='up')
    finally:
        ip.close()


def determine_interfaces(domain):
    interfaces = []
    difs = domain.findall('devices/interface')
    for dif in difs:
        dif_target = dif.find('target')
        if dif_target is not None:
            ifname = dif_target.attrib.get('dev', '')
            for ifprefix in IFPREFIXES:
                if ifname.startswith(ifprefix):
                    interfaces.append(ifname)
    return interfaces

if __name__ == '__main__':
    import time
    timestamp = int(time.time())
    with open('/tmp/hook.log', 'a') as log:
        target, oper, sub_oper = sys.argv[1:4]
        log.write('%s %s %s %s\n' % (timestamp, target, oper, sub_oper))

        # only run at prepare begin
        if oper != 'prepare' or sub_oper != 'begin':
            sys.exit(0)

        # read domain configuration
        domain = etree.fromstring(sys.stdin.read())

        # get interface list
        interfaces = determine_interfaces(domain)
        log.write('%s %s\n' % (timestamp, interfaces))

        configure_interfaces(interfaces)

        sys.exit(0)

Problem

None

Useful Links

• http://docs.pyroute2.org/iproute.html
• https://docs.python.org/2/library/xml.etree.elementtree.html
• https://www.libvirt.org/hooks.html
• https://libvirt.org/formatdomain.html#elementsNICS