BGP+EVPN+VXLAN with Apache CloudStack

modifyvxlan.sh

#!/usr/bin/env bash
#
# Use BGP+EVPN for VXLAN with CloudStack instead of Multicast
#
# Place this file on all KVM hypervisors at /usr/share/modifyvxlan.sh
#
# More information about BGP and EVPN with FRR: https://vincent.bernat.ch/en/blog/2017-vxlan-bgp-evpn
#

DSTPORT=4789

# We bind our VXLAN tunnel IP(v4) on Loopback device 'lo'
DEV="lo"

usage() {
    echo "Usage: $0: -o <op>(add | delete) -v <vxlan id> -p <pif> -b <bridge name> (-6)"
}

localAddr() {
    local FAMILY=$1

    if [[ -z "$FAMILY" || $FAMILY == "inet" ]]; then
       ip -4 -o addr show scope global dev ${DEV} | awk 'NR==1 {gsub("/[0-9]+", "") ; print $4}'
    fi

    if [[ "$FAMILY" == "inet6" ]]; then
       ip -6 -o addr show scope global dev ${DEV} | awk 'NR==1 {gsub("/[0-9]+", "") ; print $4}'
    fi
}

addVxlan() {
    local VNI=$1
    local PIF=$2
    local VXLAN_BR=$3
    local FAMILY=$4
    local VXLAN_DEV=vxlan${VNI}
    local ADDR=$(localAddr ${FAMILY})

    echo "local addr for VNI ${VNI} is ${ADDR}"

    if [[ ! -d /sys/class/net/${VXLAN_DEV} ]]; then
        ip -f ${FAMILY} link add ${VXLAN_DEV} type vxlan id ${VNI} local ${ADDR} dstport ${DSTPORT} nolearning
        ip link set ${VXLAN_DEV} up
        sysctl -qw net.ipv6.conf.${VXLAN_DEV}.disable_ipv6=1
    fi

    if [[ ! -d /sys/class/net/$VXLAN_BR ]]; then
        ip link add name ${VXLAN_BR} type bridge
        ip link set ${VXLAN_BR} up
        sysctl -qw net.ipv6.conf.${VXLAN_BR}.disable_ipv6=1
    fi

    bridge link show|grep ${VXLAN_BR}|awk '{print $2}'|grep "^${VXLAN_DEV}\$" > /dev/null
    if [[ $? -gt 0 ]]; then
        ip link set ${VXLAN_DEV} master ${VXLAN_BR}
    fi
}

deleteVxlan() {
    local VNI=$1
    local PIF=$2
    local VXLAN_BR=$3
    local FAMILY=$4
    local VXLAN_DEV=vxlan${VNI}

    ip link set ${VXLAN_DEV} nomaster
    ip link delete ${VXLAN_DEV}

    ip link set ${VXLAN_BR} down
    ip link delete ${VXLAN_BR} type bridge
}

OP=
VNI=
FAMILY=inet
option=$@

while getopts 'o:v:p:b:6' OPTION
do
  case $OPTION in
  o)    oflag=1
        OP="$OPTARG"
        ;;
  v)    vflag=1
        VNI="$OPTARG"
        ;;
  p)    pflag=1
        PIF="$OPTARG"
        ;;
  b)    bflag=1
        BRNAME="$OPTARG"
        ;;
  6)
        FAMILY=inet6
        ;;
  ?)    usage
        exit 2
        ;;
  esac
done

if [[ "$oflag$vflag$pflag$bflag" != "1111" ]]; then
    usage
    exit 2
fi

lsmod|grep ^vxlan >& /dev/null
if [[ $? -gt 0 ]]; then
    modprobe=`modprobe vxlan 2>&1`
    if [[ $? -gt 0 ]]; then
        echo "Failed to load vxlan kernel module: $modprobe"
        exit 1
    fi
fi


#
# Add a lockfile to prevent this script from running twice on the same host
# this can cause a race condition
#

LOCKFILE=/var/run/cloud/vxlan.lock

(
    flock -x -w 10 200 || exit 1
    if [[ "$OP" == "add" ]]; then
        addVxlan ${VNI} ${PIF} ${BRNAME} ${FAMILY}

        if [[ $? -gt 0 ]]; then
            exit 1
        fi
    elif [[ "$OP" == "delete" ]]; then
        deleteVxlan ${VNI} ${PIF} ${BRNAME} ${FAMILY}
    fi
) 200>${LOCKFILE}

Thanks Wido for sharing this, can I ask you 2 specifics things regarding this?.

1. My hypervisors are based on Ubuntu and the file modifyvxlan.sh already exists but in a different path, should I replace it or just copy in the path you suggest.
2. Can I use one of the hyp as the route reflector and as vtep?, do you have a file example of the bgp evpn example I can use?

Many thanks!

Thanks Wido for sharing this, can I ask you 2 specifics things regarding this?.

1. My hypervisors are based on Ubuntu and the file modifyvxlan.sh already exists but in a different path, should I replace it or just copy in the path you suggest.

Just put the file in /usr/share and restart the cloudstack agent. It will detect the file there. (Also see the comments in the header of the file)

2. Can I use one of the hyp as the route reflector and as vtep?, do you have a file example of the bgp evpn example I can use?

Many thanks!

Not sure what you exactly mean. This is the relevant BGP configuration with FRR I use:

router bgp 4200100145
..
 address-family ipv4 unicast
  network 10.255.255.32/32
  neighbor uplinks activate
  neighbor uplinks next-hop-self
  neighbor uplinks soft-reconfiguration inbound
  neighbor uplinks route-map upstream-v4-out out
  neighbor uplinks route-map upstream-v4-in in
 exit-address-family
 !
 address-family ipv6 unicast
  network 2a05:xxxx:xxxx:2::32/128
  neighbor uplinks activate
  neighbor uplinks soft-reconfiguration inbound
  neighbor uplinks route-map upstream-v6-in in
  neighbor uplinks route-map upstream-v6-out out
 exit-address-family
 address-family l2vpn evpn
  neighbor uplinks activate
  advertise-all-vni
 exit-address-family

Thanks Wido, it's working well for me now, just one thing, if I delete a machine and recreate it with same IP, bgp entry
doesn't seem to update until I create another vxlan.

Thanks Wido, it's working well for me now, just one thing, if I delete a machine and recreate it with same IP, bgp entry doesn't seem to update until I create another vxlan.

Hypervisor you mean? Compute node. That is logical. See the script. It takes the IP which is on the loopback interface and is hardcoded to the VXLAN device.

Use unique IPs per host and do not try to change.