Spin up Kubernetes without a network, then... install weave per the installation guide
In theory before CNI plugin install nodes should be not ready & the CNI net.d folder should be empty...
[centos@kube-nonetwork-master ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-nonetwork-master NotReady master 58s v1.13.3
kube-nonetwork-node-1 NotReady <none> 34s v1.13.3
[centos@kube-nonetwork-master ~]$ ls /etc/cni/net.d/ -1
[centos@kube-nonetwork-master ~]$ ls /etc/cni/net.d/ -1 | wc -l
0
Install weave...
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
Watch it come up...
[centos@kube-nonetwork-master ~]$ watch -n1 kubectl get pods --all-namespaces
Now your nodes should be ready...
[centos@kube-nonetwork-master ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-nonetwork-master Ready master 10m v1.13.3
kube-nonetwork-node-1 Ready <none> 9m50s v1.13.3
Create a YAML file with these contents (I called mine multus.yaml)...
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: network-attachment-definitions.k8s.cni.cncf.io
spec:
group: k8s.cni.cncf.io
version: v1
scope: Namespaced
names:
plural: network-attachment-definitions
singular: network-attachment-definition
kind: NetworkAttachmentDefinition
shortNames:
- net-attach-def
validation:
openAPIV3Schema:
properties:
spec:
properties:
config:
type: string
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: multus
rules:
- apiGroups: ["k8s.cni.cncf.io"]
resources:
- '*'
verbs:
- '*'
- apiGroups:
- ""
resources:
- pods
- pods/status
verbs:
- get
- update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: multus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: multus
subjects:
- kind: ServiceAccount
name: multus
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: multus
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-multus-ds-amd64
namespace: kube-system
labels:
tier: node
app: multus
spec:
template:
metadata:
labels:
tier: node
app: multus
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: multus
containers:
- name: kube-multus
image: nfvpe/multus:snapshot
command: ["/entrypoint.sh"]
args: ["--multus-conf-file=auto"]
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: true
volumeMounts:
- name: cni
mountPath: /host/etc/cni/net.d
- name: cnibin
mountPath: /host/opt/cni/bin
volumes:
- name: cni
hostPath:
path: /etc/cni/net.d
- name: cnibin
hostPath:
path: /opt/cni/bin
Then apply that file...
kubectl create -f multus.yaml
Watch the pods come up...
[centos@kube-nonetwork-master ~]$ watch -n1 kubectl get pods --all-namespaces
Then you can see there's a new Multus config file created in addition to weave's config...
[centos@kube-nonetwork-master ~]$ ls /etc/cni/net.d/ -lathr
total 8.0K
drwxr-xr-x. 3 root root 19 Feb 5 20:30 ..
-rw-r--r--. 1 root root 318 Feb 5 20:36 10-weave.conflist
drwxr-xr-x. 2 root root 31 Feb 5 20:44 multus.d
-rw-r--r--. 1 root root 326 Feb 5 20:44 00-multus.conf
drwxr-xr-x. 3 root root 69 Feb 5 20:44 .
Cat that file to see that it contains a weave config packed in it...
cat /etc/cni/net.d/00-multus.conf
Now, let's create a macvlan configuration NOTE: My master device eth0 -- map this to the master device of the interface on your machines. I also picked a gateway and IP address range that matches my network.
cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "192.168.122.0/24",
"rangeStart": "192.168.122.200",
"rangeEnd": "192.168.122.216",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "192.168.122.1"
}
}'
EOF
You can list that...
kubectl get network-attachment-definitions
Now let's create a pod that uses that...
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
containers:
- name: samplepod
command: ["/bin/bash", "-c", "sleep 2000000000000"]
image: dougbtv/centos-network
EOF
Wait until that pod comes up...
watch -n1 kubectl get pods -o wide
Now you can see it has multiple interfaces...
[centos@kube-nonetwork-master ~]$ kubectl exec -it samplepod -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 2a:86:a0:1f:c2:0a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.122.200/24 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::2886:a0ff:fe1f:c20a/64 scope link
valid_lft forever preferred_lft forever
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP
link/ether d6:40:08:96:74:7e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.40.0.2/12 brd 10.47.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::d440:8ff:fe96:747e/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever
I can also ping on both IPs...
[centos@kube-nonetwork-master ~]$ ping -c 5 10.40.0.2
PING 10.40.0.2 (10.40.0.2) 56(84) bytes of data.
64 bytes from 10.40.0.2: icmp_seq=1 ttl=64 time=1.68 ms
64 bytes from 10.40.0.2: icmp_seq=2 ttl=64 time=0.565 ms
64 bytes from 10.40.0.2: icmp_seq=3 ttl=64 time=0.618 ms
64 bytes from 10.40.0.2: icmp_seq=4 ttl=64 time=0.474 ms
64 bytes from 10.40.0.2: icmp_seq=5 ttl=64 time=0.680 ms
--- 10.40.0.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 0.474/0.804/1.686/0.446 ms
[centos@kube-nonetwork-master ~]$ ping -c 5 192.168.122.200
PING 192.168.122.200 (192.168.122.200) 56(84) bytes of data.
64 bytes from 192.168.122.200: icmp_seq=1 ttl=64 time=0.593 ms
64 bytes from 192.168.122.200: icmp_seq=2 ttl=64 time=0.288 ms
64 bytes from 192.168.122.200: icmp_seq=3 ttl=64 time=0.427 ms
64 bytes from 192.168.122.200: icmp_seq=4 ttl=64 time=0.369 ms
64 bytes from 192.168.122.200: icmp_seq=5 ttl=64 time=0.368 ms
You can also get the IPs from the status annotation...
[centos@kube-nonetwork-master ~]$ kubectl describe pod samplepod
Look for the annotations section...
Annotations: k8s.v1.cni.cncf.io/networks: macvlan-conf
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "weave",
"ips": [
"10.40.0.2"
],
"default": true,
"dns": {}
},{
"name": "macvlan-conf",
"interface": "net1",
"ips": [
"192.168.122.200"
],
"mac": "2a:86:a0:1f:c2:0a",
"dns": {}
}]