Skip to content

Instantly share code, notes, and snippets.

@dougbtv

dougbtv/README.md Secret

Created February 20, 2019 21:15
Show Gist options
  • Save dougbtv/31b53730afc11eeffee30f30907d1060 to your computer and use it in GitHub Desktop.
Save dougbtv/31b53730afc11eeffee30f30907d1060 to your computer and use it in GitHub Desktop.
Download ZIP
OpenShift SDN IP files not deleted when failing pod deleted with Multus
Raw
README.md

bugzilla link

Reproduce

Label each node.

oc label nodes {your-worker-node-1} debugger=true
oc get nodes --show-labels

Create a failing pod per Meng Bo's suggestion...

cat <<EOF | oc create -f -
apiVersion: v1
kind: Pod
metadata:
  name: testpod
spec:
  containers:
  - name: testpod
    command: ["/bin/bash", "-c", "sleep 5; false"]
    image: centos/tools
  nodeSelector:
    debugger: "true"
EOF

Create & delete, and then get a count of ip files...

ls -1 /var/lib/cni/networks/openshift-sdn | wc -l

You can see them growing as you create/delete/create/delete/etc.

Try to fix by using newly built from Tomo's caching PR?

Locate Multus binary on disk:

[root@ip-10-0-135-168 /]# ls /var/lib/cni/bin/multus

scp local copy up and replace... ummm... yeah. Doesn't look like it did it.

Enable debugging and analyze.

Enable logging, used this config:

[root@ip-10-0-135-168 core]# cat /etc/kubernetes/cni/net.d/00-multus.conf 
{ "name": "multus-cni-network", "type": "multus", "logFile": "/var/log/multus.log", "logLevel": "debug", "namespaceIsolation": true, "kubeconfig": "/etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig", "delegates": [ { "cniVersion": "0.2.0", "name": "openshift-sdn", "type": "openshift-sdn" } ] }

No logs on delete? That's.... not good.

Logs on create.

2019-02-20T20:56:05Z [debug] LoadDelegateNetConf: {"cniVersion":"0.2.0","name":"openshift-sdn","type":"openshift-sdn"}, <nil>, 
2019-02-20T20:56:05Z [debug] GetK8sArgs: &{1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e /proc/16057/ns/net eth0 IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=testpod;K8S_POD_INFRA_CONTAINER_ID=1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e /var/lib/cni/bin:/opt/multus/bin [123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 50 46 48 34 44 34 110 97 109 101 34 58 34 111 112 101 110 115 104 105 102 116 45 115 100 110 34 44 34 116 121 112 101 34 58 34 111 112 101 110 115 104 105 102 116 45 115 100 110 34 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 107 117 98 101 114 110 101 116 101 115 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 108 111 103 70 105 108 101 34 58 34 47 118 97 114 47 108 111 103 47 109 117 108 116 117 115 46 108 111 103 34 44 34 108 111 103 76 101 118 101 108 34 58 34 100 101 98 117 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 110 97 109 101 115 112 97 99 101 73 115 111 108 97 116 105 111 110 34 58 116 114 117 101 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]}
2019-02-20T20:56:11Z [debug] TryLoadPodDelegates: &{{true} <nil> testpod default 1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e}, &{{ multus-cni-network multus map[] {} {[]  [] []}} <nil> <nil> /etc/cni/multus/net.d /var/lib/cni/multus /opt/cni/bin [] [0xc4202b3680] [] /etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig  [] /var/log/multus.log debug <nil>  true}, <nil>
2019-02-20T20:56:11Z [debug] GetK8sClient: /etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig, <nil>
2019-02-20T20:56:11Z [debug] setKubeClientInfo: &{<nil>  }, &{0xc420334690}, &{{true} <nil> testpod default 1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e}
2019-02-20T20:56:11Z [debug] tryLoadK8sPodDefaultNetwork: &{0xc420334690}, &{{true} <nil> testpod default 1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e}
2019-02-20T20:56:11Z [debug] getPodDefaultNetworkAnnotation: &{0xc420334690}, &{{true} <nil> testpod default 1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e}
2019-02-20T20:56:11Z [debug] tryLoadK8sPodDefaultNetwork: Pod default network annotation is not defined
2019-02-20T20:56:11Z [debug] GetPodNetwork: &{0xc420334690}, &{{true} <nil> testpod default 1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e}, /etc/cni/multus/net.d
2019-02-20T20:56:11Z [debug] getPodNetworkAnnotation: &{0xc420334690}, &{{true} <nil> testpod default 1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e}
2019-02-20T20:56:11Z [debug] saveDelegates: 1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e, /var/lib/cni/multus, [0xc4202b3680]
2019-02-20T20:56:11Z [debug] saveScratchNetConf: 1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e, /var/lib/cni/multus, [{"Conf":{"cniVersion":"0.2.0","name":"openshift-sdn","type":"openshift-sdn","ipam":{},"dns":{}},"ConfList":{},"Bytes":"eyJjbmlWZXJzaW9uIjoiMC4yLjAiLCJuYW1lIjoib3BlbnNoaWZ0LXNkbiIsInR5cGUiOiJvcGVuc2hpZnQtc2RuIn0="}]
2019-02-20T20:56:11Z [debug] getIfname: &{{0.2.0 openshift-sdn openshift-sdn map[] {} {[]  [] []}} {  []}    true false [123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 50 46 48 34 44 34 110 97 109 101 34 58 34 111 112 101 110 115 104 105 102 116 45 115 100 110 34 44 34 116 121 112 101 34 58 34 111 112 101 110 115 104 105 102 116 45 115 100 110 34 125]}, eth0, 0
2019-02-20T20:56:11Z [debug] LoadCNIRuntimeConf: &{1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e /proc/16057/ns/net eth0 IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=testpod;K8S_POD_INFRA_CONTAINER_ID=1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e /var/lib/cni/bin:/opt/multus/bin [123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 50 46 48 34 44 34 110 97 109 101 34 58 34 111 112 101 110 115 104 105 102 116 45 115 100 110 34 44 34 116 121 112 101 34 58 34 111 112 101 110 115 104 105 102 116 45 115 100 110 34 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 107 117 98 101 114 110 101 116 101 115 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 108 111 103 70 105 108 101 34 58 34 47 118 97 114 47 108 111 103 47 109 117 108 116 117 115 46 108 111 103 34 44 34 108 111 103 76 101 118 101 108 34 58 34 100 101 98 117 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 110 97 109 101 115 112 97 99 101 73 115 111 108 97 116 105 111 110 34 58 116 114 117 101 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]}, &{{true} <nil> testpod default 1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e}, eth0, <nil>
2019-02-20T20:56:11Z [debug] delegateAdd: <nil>, eth0, &{{0.2.0 openshift-sdn openshift-sdn map[] {} {[]  [] []}} {  []}    true false [123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 50 46 48 34 44 34 110 97 109 101 34 58 34 111 112 101 110 115 104 105 102 116 45 115 100 110 34 44 34 116 121 112 101 34 58 34 111 112 101 110 115 104 105 102 116 45 115 100 110 34 125]}, &{1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e /proc/16057/ns/net eth0 [[IgnoreUnknown 1] [K8S_POD_NAMESPACE default] [K8S_POD_NAME testpod] [K8S_POD_INFRA_CONTAINER_ID 1ec86af232233f97c3e56deae9cc33d7c587f44160bf2aaecd0768038eba222e]] map[] }, /opt/cni/bin
2019-02-20T20:56:11Z [debug] validateIfName: /proc/16057/ns/net, eth0
2019-02-20T20:56:12Z [debug] LoadNetworkStatus: IP4:{IP:{IP:10.128.2.32 Mask:fffffe00} Gateway:<nil> Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:10.128.2.1} {Dst:{IP:224.0.0.0 Mask:f0000000} GW:<nil>} {Dst:{IP:10.128.0.0 Mask:fffc0000} GW:<nil>}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}, openshift-sdn, true
2019-02-20T20:56:12Z [debug] SetNetworkStatus: &{0xc420359de0 default testpod}, [0xc420488370]
2019-02-20T20:56:12Z [debug] setPodNetworkAnnotation: &{0xc420334690}, default, &Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:testpod,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/pods/testpod,UID:f06fdd89-3551-11e9-9afa-0a55b95f7356,ResourceVersion:123300,Generation:0,CreationTimestamp:2019-02-20 20:56:05 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{openshift.io/scc: anyuid,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PodSpec{Volumes:[{default-token-28zcn {nil nil nil nil nil SecretVolumeSource{SecretName:default-token-28zcn,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{testpod centos/tools [/bin/bash -c sleep 5; false] []  [] [] [] {map[] map[]} [{default-token-28zcn true /var/run/secrets/kubernetes.io/serviceaccount  <nil>}] [] nil nil nil /dev/termination-log File Always SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[MKNOD],},Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,} false false false}],RestartPolicy:Always,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{debugger: true,},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:ip-10-0-135-168.us-west-2.compute.internal,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c13,c12,},RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[],},ImagePullSecrets:[{default-dockercfg-pg26z}],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[{node.kubernetes.io/not-ready Exists  NoExecute 0xc42031a910} {node.kubernetes.io/unreachable Exists  NoExecute 0xc42031a970}],HostAliases:[],PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],},Status:PodStatus{Phase:Pending,Conditions:[{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-02-20 20:56:05 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-02-20 20:56:05 +0000 UTC ContainersNotReady containers with unready status: [testpod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-02-20 20:56:05 +0000 UTC ContainersNotReady containers with unready status: [testpod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-02-20 20:56:05 +0000 UTC  }],Message:,Reason:,HostIP:10.0.135.168,PodIP:,StartTime:2019-02-20 20:56:05 +0000 UTC,ContainerStatuses:[{testpod {ContainerStateWaiting{Reason:ContainerCreating,Message:,} nil nil} {nil nil nil} false 0 centos/tools  }],QOSClass:BestEffort,InitContainerStatuses:[],NominatedNodeName:,},}, [{
    "name": "openshift-sdn",
    "ips": [
        "10.128.2.32"
    ],
    "default": true,
    "dns": {}
}]

Logs on delete, if these are even related...

2019-02-20T21:00:55Z [debug] LoadDelegateNetConf: {"cniVersion":"0.2.0","name":"openshift-sdn","type":"openshift-sdn"}, <nil>,
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment