Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Provisioning to use Calico
First, tearing down the cluster, by doing this on each node:
kubeadm reset
rm -rf .kube
systemctl start kubelet.service
Did yum -y update on all nodes as well. Downloaded and then ran "kubectl create -f calico.yaml".
Starting up cluster (with just the master node).
kubeadm init
kubeadm join --token=482742.64ff98647d01bf2c
kubectl taint nodes --all dedicated-
kubectl apply -f calico.yaml
kubectl get pods --all-namespaces
Issue: It is unable to download calico/ctl v0.23.1. It does not exist. Options are to use v1.0.0 (and use same for calico/node, to keep them similar?) or drop to v0.23.0 (and either keep calico/node at v0.23.1 or set it to v.23.0 too). Trying v1.0.0. Stopped cluster and then restarting, after updating calico.yaml.
kubeadm join --token=086198.58f8a53b0c2391be
Looks like calico/configure is not starting up (message about calicoctl unknown command pool). Also dns pod not coming up, says CNI config is uninitialized. Will try v0.23.0 for both node and ctl…
kubeadm join --token=b3be10.7356d594b9de79a2
Looks like everything I running…
[root@devstack-77 ~]# kubectl get pods --all-namespaces
kube-system calico-etcd-4zkxh 1/1 Running 0 10m
kube-system calico-node-c8pkm 2/2 Running 0 10m
kube-system calico-policy-controller-2swdt 1/1 Running 0 10m
kube-system dummy-2088944543-c5xlk 1/1 Running 0 12m
kube-system etcd-devstack-77 1/1 Running 0 11m
kube-system kube-apiserver-devstack-77 1/1 Running 1 11m
kube-system kube-controller-manager-devstack-77 1/1 Running 0 11m
kube-system kube-discovery-1769846148-8fp6j 1/1 Running 0 12m
kube-system kube-dns-2924299975-r04zb 4/4 Running 0 12m
kube-system kube-proxy-3lmlh 1/1 Running 0 12m
kube-system kube-scheduler-devstack-77 1/1 Running 0 11m
No IPv6. Need to add IPAM setting to calico.yaml, with "assign_ipv6": "true". Doing that and then restarting the cluster (not sure if I can just reapply).
kubeadm join --token=a68a48.184e49eb7ec5588e
Note: Issue 455 is closed, they pushed a tag v0.23.1 out for calico/ctl.
Looks like DNS pod is not coming up, and is showing this error:
14m 1s 372 {kubelet devstack-77} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-zxr50_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-zxr50_kube-system(6fb8a3ae-df38-11e6-ae00-003a7d69f73c)\" using network plugins \"cni\": No configured Calico ools; Skipping pod"
Issue is that IPv6 I not set up on the nodes. Added this to /etc/sysconfig/network:
For the bridge interface (using br_mgmt), added:
Used 71,77,78 for the three systems.
Next, brought up cloud and started Ubuntu (w/o IPv6) in two containers and checked ping.
kubeadm join --token=228e0d.1b18edb2e1fe7159
Did web.yml and redis-master-deployment.yml and then connected to latter and pinged the former with IPv4 address. Now, enabling IPv6 and trying again.
Note: net-tools for route command, tcpdump installed too. Have "ip -6 addr add <address> dev <dev>" to add an IP address. Have "ip -6 route" or "route -A inet6" with package installed.
kubeadm join --token=3b13ce.433093a70c2e47a0
The DNS pod is not running and we see the error:
5m 3s 28 {kubelet devstack-77} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-1m2c5_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-1m2c5_kube-system(e76de7f4-e192-11e6-86d8-003a7d69f73c)\" using network plugins \"cni\": No configured Calico pools; Skipping pod"
Q: Do I need to use the br_mgmt IP addresses?
Q: Any other config?
Trying the following (can try IPv6 later and maybe other settings, like DNS?):
kubeadm init --api-advertise-addresses=
Control plane never became ready. Instead, trying this:
kubeadm init --pod-network-cidr=2001:1::/64
Seems stuck creating test deployment. Trying adding "subnet" with value "2001:2::/64" to calico.yaml IPAM area and then creating cluster. Still seeing DNS not coming up.
Trying Again 1/26/2017
Seeing notes on reference page will try these steps. First, set global IP on each node, with 2001:1::#/64, where # is 71,77,78. Did this on br_mgmt interface (can try on br_api, if this doesn't work). Verified pings.
Bringing up cluster with calico yaml file, and only the assign_ipv6 flag set (no subnet).
kubeadm init
kubeadm join --token=0f1c24.963cd0640a394e46
kubectl taint nodes --all dedicated-
kubectl apply -f calico.yaml
kubectl get pods --all-namespaces
The DNS pod is showing this:
2m 2s 11 {kubelet devstack-77} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-nqm5n_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-nqm5n_kube-system(cff5cc0e-e3fd-11e6-a9ac-003a7d69f73c)\" using network plugins \"cni\": No configured Calico pools; Skipping pod"
Getting calicoctl using these instructions:
git clone $GOPATH/src/
Install glide:
mkdir $GOPATH/bin
curl | sh
cd /root/go/src/
glide install -strip-vendor
make binary
go build src/
mv calicoctl bin/
~/go/bin/calicoctl node --ip= --ipv6=2001:1::77 --libnetwork
Set the Calico datastore access information in the environment variables or
or supply details in a config file.
Already have Calico node container running. How do I do the required changes? They mention about creating pools, but there is not pool command for calicoctl.
Note: See this page ( has different steps to install calicoctl.
Trying setting assign_ipv6 in addition to ipv6 in config file.
kubeadm join --token=6d8305.5794a9835accd11a
Info from lrobson and epeterson…
Need newer calico.yaml file from
They use BIRD, a software routing daemon that does BGP peering for nodes.
BGP issues: BGP won't accept IPv6 router IDs. Need IPv4 config, even if doing IPv6 BGP.
BIRD caveats: IPv6 ECMP doesn't work right in BIRD. IPv6 loopbacks also don't work correctly.
BIRD6 templates in
epeterson [11:48 AM]
I think that configuring v6 would just work as long as you got the prefixes the node was supposed to have into calico’s etcd. Not 100% sure of the exact logic the template mechanism uses to generate templates but unless there’s something that’s parsing specific strings, it would work
The tl;dr on v6netes is that the large majority of k8s users are doing things on cloud platforms. AWS didn’t support v6 until very recently, azure and GCE still don’t. I’m looking for a PR where a bunch of the issues are being discussed.
One fundamental problem that is being worked for next release is that k8s doesn’t support multiple networks or address families.
Well, not next release, 1.6 is a stability release. But 1.7 multiple networks is on the agenda.
Lrobson says need calico-ipam set to assign IPv6 addresses (
cni_network_config: |-
"name": "k8s-pod-network",
"type": "calico",
"etcd_endpoints": "__ETCD_ENDPOINTS__",
"log_level": "info",
"ipam": {
"type": "calico-ipam",
"assign_ipv6": "true"
"policy": {
"type": "k8s",
"k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/__KUBECONFIG_FILENAME__"
And… add pool here:
# The default IP Pool to be created for the cluster.
# Pod IP addresses will be assigned from this pool.
ippool.yaml: |
apiVersion: v1
kind: ipPool
cidr: 2001:1::/64
enabled: true
nat-outgoing: true
Trying yaml file w/o changes. All pods came up. Trying with changes above:
kubeadm join --token=849d9a.7a5aeb672e07a210
The DNS pod is not coming up. It fails with a message saying:
Failed to get IPv6 addresses for container veth
It keeps restarting. Went into calico-node container (Using docker exec-it <container-name> sh), and then looked at:
cat /etc/calico/confd/config/bird6_ipam.cfg
cat /etc/calico/confd/config/bird6_aggr.cfg
There is a bird.cfg and bird6.cfg in /etc/calico/confd/config/ area. On this setup. There is some interesting things. It shows the router ID as, which is on the t interface and not br_api. I don't have global IPv6 on that I/F. May have just used that IP.
So, DNS container is not getting an IPv6 address. See:
[root@devstack-77 ~]# ~/go/bin/calicoctl node status
Calico process is running.
IPv4 BGP status
No IPv4 peers found.
IPv6 BGP status
No IPv6 peers found.
Eric Peterson, gave me a bird.cfg file to use. Need to create container/VM with BIRD so that have something for cluster to BGP peer with, so that calico nodes receive pod routes from other nodes.
Q: Do I create a VM or container with BIRD running to provide the BGP peer info?
Q: Could we use nexus9k as BGP peer?
Q: Who's versed in BGP?
1/30/2017 BIRD
Created a Dockerfile in ~/bird. Downloaded the Fedora binary for bird ( to Mac and copied to ds-77. Added to Docker file. Issue, install failed as needs ncurses-lib-6.0 and Centos has 5.0 as latest.
Could not add bird repo to yum.repos.d/ as it uses FTP and cannot seem to access the site from within the lab.
Trying Ubuntu (, and have this working, using this Dockerfile:
COPY ./apt.key /
RUN apt-key add /apt.key
RUN apt-get update -y && apt-get upgrade -y
RUN apt-get install -y bird
CMD /bin/bash
Did "docker build ." and then "docker run -it <IMAGE#> bash". Need to label the image, I guess.
Need to configure Bird. Eric says, because inside of a container, may need an external static NAT for docker0 IP to the internal Bird IP. His example had static route, because Bird host was on a secondary subnet and needed a route to primary VLAN IP to peer with it (BGP needs next hop reachability -doesn't use route table to establish peering).
Q: Should I use VM instead? Could I use Bird running on host?
Q: With Bird in container, it is using docker0 I/F. Does it need IPv6 enabled? Does docker need IPv6 enabled? How?
In case need VM, can use virsh:
yum install qemu-kvm qemu-img
yum install virt-manager libvirt libvirt-python libvirt-client
yum groupinstall virtualization-client virtualization-platform virtualization-tools
restart libvirtd
systemctl status libvirtd
Or can use Virtualbox. Info? Decide on method, if do this.
Issue with auto detect IP for calico with multiple nodes (
May be able to remove other networks from host, and calico may autodetect, on the remaining network. Instead of picking a network, and getting, as it did in my case with the t interface.
ozdanborne [2:45 PM]
@pcm I think a few options are:
- setup your interfaces so calico autodetects the right one
- build a custom calicoctl.go / calico/node which autodetects the right one
- update the `BGPPeer`s after they come up.
a less "quick" solution would be for us to start putting together that `--can-reach` support (edited)
Going to try the first option tomorrow… setting up interfaces so that auto-detect works.
Took IP/Mask off of t and br_mgmt interfaces(how to shutdown too?), and added IPv6 address 2001:2::77/64 to brapi interface.Did the same on other nodes. Starting up kubeadm to see what happens.
kubeadm join --token=10f438.23104a16fdf4dd69
Did taint and applied calico.yaml (after changed pool to 2001:2::/64. The calico-node container still has IP and 2001:1::/64 for filter. Will tear down and then remove container images from docker and then retry. Had to delete stopped containers to get rid of the image.
kubeadm join --token=cdee1f.03c5873344ab69c1
Same issue with IP used. Copied some config files in ~ to ~/save, in case they are being used. Removed containers:
docker rm `docker ps --format "table {{.ID}}" -a | tail -n +2`
Removed images and retrying kubeadm init.
kubeadm join --token=125b8a.5858c1c63bd9f5a4
It still is using Trying Ipv6 for calico.yaml and rechecking.
kubeadm join --token=ff89b8.fa557582bb0ff77e
Looks like maybe info is in etcd (my theory). Deleting all containers, deleting all images, and deleting files in /var/lib/etcd/. Rebooted system and restarting up kubeadm. Instead of rebooting, may be able to do "ip route flush proto bird". Instead of deleting dir, may be able to do "curl -sL http://<IP_OF_ETCD>:2379/v2/keys/calico?recursive=true -XDELETE". Try some time.
Still see for ID, and 2001:1:: in bird6.cfg files.
Lrobson/caseydavenport indicate that IP can be set manually. Can either delete DaemonSet that starts calico-node, and manually run calico-node with IP (, or use calicoctl to edit the IP on the node ( Suggested that I do:
calicoctl get nodes -o yaml > nodes.txt
calicoctl apply -f nodes.txt
Tried. No node info listed. Need to set ETCD_AUTHORITY env variable to and then run it. Or can set this in /etc/calico/calicoctl.cfg:
apiVersion: v1
kind: calicoApiConfig
datastoreType: "etcdv2"
etcdEndpoints: ""
Applied change, but container still shows Should be in datastore, so will restart the calico-node. Did docker restart and the node has the correct IP now.
Cleaning containers, reset kubeadm, clear etcd, delete .kube dir, flushed route on all nodes, and will restart with IPv6 configuration. Doing init:
kubeadm join --token=b29e05.9d010309f5113284
Copied calico.yaml.v6 config file to calico.yaml and doing rest:
kubectl taint nodes --all dedicated-
kubectl apply -f calico.yaml
kubectl get pods --all-namespaces
Started up. See IPPools, 2001:1::/64, and 2001:2::/64. In calico-node, the router IP is (correct). In bird6_aggr.cfg, it has blackhole, accept, and reject for 2001:1:…. In bird6_ipam.cfg, it has entries for both 2001:1::/64 and 2001:2::/64.
Did "calicoctl delete ippool 2001:1::/64" and now that is gone. The bird6_aggr.cfg has the 2001:2::… addresses now, and bird6_ipam.cfg now has the 2001:2::/64 network. So, it seems OK now.
Q: Is it OK on the other nodes?
Q: Why is DNS pod not able to get IPv6 address?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment