Skip to content

Instantly share code, notes, and snippets.

@anfernee
Last active June 22, 2023 04:49
Show Gist options
  • Save anfernee/31db0f4a9a72f12f353aa0c0158e637e to your computer and use it in GitHub Desktop.
Save anfernee/31db0f4a9a72f12f353aa0c0158e637e to your computer and use it in GitHub Desktop.
2023-06-22T03:48:44Z [debug] Used defaults from parsed flat file config @ /etc/cni/net.d/whereabouts.d/whereabouts.conf
2023-06-22T03:48:44Z [debug] ADD - IPAM configuration successfully read: {Name:whereaboutsexample Type:whereabouts Routes:[] Addresses:[] IPRanges:[{OmitRanges:[] Range:192.168.2.224/28 RangeStart:192.168.2.224 RangeEnd:<nil>}] OmitRanges:[] DNS:{Nameservers:[] Domain: Search:[] Options:[]} Range: RangeStart:<nil> RangeEnd:<nil> GatewayStr: LeaderLeaseDuration:1500 LeaderRenewDeadline:1000 LeaderRetryPeriod:500 LogFile:/var/log/whereabouts.log LogLevel:debug ReconcilerCronExpression:30 4 * * * OverlappingRanges:true SleepForRace:0 Gateway:<nil> Kubernetes:{KubeConfigPath:/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig K8sAPIRoot:} ConfigurationPath: PodName:netshoot-deployment-77bc5477c4-dc7mw PodNamespace:default NetworkName:}
2023-06-22T03:48:44Z [debug] Beginning IPAM for ContainerID: ceed7f96e1b31566dec70bdde1d6ea7b8f20ce1f8c3599b4415ba9e7acf3db00
2023-06-22T03:48:44Z [debug] Started leader election
2023-06-22T03:48:44Z [debug] OnStartedLeading() called
2023-06-22T03:48:44Z [debug] Elected as leader, do processing
2023-06-22T03:48:44Z [debug] IPManagement -- mode: 0 / containerID: ceed7f96e1b31566dec70bdde1d6ea7b8f20ce1f8c3599b4415ba9e7acf3db00 / podRef: default/netshoot-deployment-77bc5477c4-dc7mw
2023-06-22T03:48:44Z [error] IPAM error reading pool allocations (attempt: 0): k8s pool initialized
2023-06-22T03:48:44Z [debug] IterateForAssignment input >> range_start: 192.168.2.224 | range_end: <nil> | ipnet: {192.168.2.224 fffffff0} | first IP: 192.168.2.225 | last IP: 192.168.2.238
2023-06-22T03:48:44Z [debug] Reserving IP: |192.168.2.225 ceed7f96e1b31566dec70bdde1d6ea7b8f20ce1f8c3599b4415ba9e7acf3db00|
2023-06-22T03:48:44Z [debug] OverlappingRangewide allocation check for IP: 192.168.2.225
2023-06-22T03:48:44Z [debug] K8s UpdateOverlappingRangeAllocation success on allocate: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:192.168.2.225 GenerateName: Namespace:kube-system SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[] ZZZ_DeprecatedClusterName: ManagedFields:[]} Spec:{ContainerID:ceed7f96e1b31566dec70bdde1d6ea7b8f20ce1f8c3599b4415ba9e7acf3db00 PodRef:default/netshoot-deployment-77bc5477c4-dc7mw}}
2023-06-22T03:48:44Z [debug] OnStoppedLeading() called
2023-06-22T03:48:44Z [debug] Finished leader election
2023-06-22T03:48:44Z [debug] IPManagement: [{192.168.2.225 fffffff0}], <nil>
kubectl get ippool -A -o yaml 
apiVersion: v1
items:
- apiVersion: whereabouts.cni.cncf.io/v1alpha1
  kind: IPPool
  metadata:
    creationTimestamp: "2023-06-22T03:48:44Z"
    generation: 2
    name: 192.168.2.224-28
    namespace: kube-system
    resourceVersion: "932"
    uid: 1d883459-7d47-4519-afde-599a74337b22
  spec:
    allocations:
      "1":
        id: ceed7f96e1b31566dec70bdde1d6ea7b8f20ce1f8c3599b4415ba9e7acf3db00
        podref: default/netshoot-deployment-77bc5477c4-dc7mw
    range: 192.168.2.224/28
kind: List
metadata:
  resourceVersion: ""
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"whereabouts-cni"},"rules":[{"apiGroups":["whereabouts.cni.cncf.io"],"resources":["ippools","overlappingrangeipreservations"],"verbs":["get","list","watch","create","update","patch","delete"]},{"apiGroups":["coordination.k8s.io"],"resources":["leases"],"verbs":["*"]},{"apiGroups":[""],"resources":["pods"],"verbs":["list","watch"]},{"apiGroups":[""],"resources":["nodes"],"verbs":["get"]},{"apiGroups":["k8s.cni.cncf.io"],"resources":["network-attachment-definitions"],"verbs":["get","list","watch"]},{"apiGroups":["","events.k8s.io"],"resources":["events"],"verbs":["create","patch","update"]}]}
creationTimestamp: "2023-06-22T03:48:28Z"
name: whereabouts-cni
resourceVersion: "849"
uid: dc062347-1887-4017-865e-b92d0e5e24ef
rules:
- apiGroups:
- whereabouts.cni.cncf.io
resources:
- ippools
- overlappingrangeipreservations
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- '*'
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- k8s.cni.cncf.io
resources:
- network-attachment-definitions
verbs:
- get
- list
- watch
- apiGroups:
- ""
- events.k8s.io
resources:
- events
verbs:
- create
- patch
- update
➜  whereabouts git:(master) ✗ kubectl get lease -A
NAMESPACE         NAME                        HOLDER                                                           AGE
kube-node-lease   whereabouts-control-plane   whereabouts-control-plane                                        32m
kube-node-lease   whereabouts-worker          whereabouts-worker                                               31m
kube-node-lease   whereabouts-worker2         whereabouts-worker2                                              31m
kube-system       kube-controller-manager     whereabouts-control-plane_eac89ec9-5ee0-4336-b86a-a59ad5f2f79d   32m
kube-system       kube-scheduler              whereabouts-control-plane_bdc4d4e4-3c14-427a-b538-ec016bf71f94   32m
kube-system       whereabouts                                                                                  29m

kube-system/whereabouts doesn't have a holder, because it is for whereabouts-cni to sync with each other. whereabouts-cni runs on all the nodes.

whereabouts logs:

➜  doc git:(master) ✗ kubectl -n kube-system logs whereabouts-jlqtj 
Done configuring CNI.  Sleep=false
2023-05-17T05:14:45Z [debug] Filtering pods with filter key 'spec.nodeName' and filter value 'kind-control-plane'
2023-05-17T05:14:45Z [verbose] pod controller created
2023-05-17T05:14:45Z [verbose] Starting informer factories ...
2023-05-17T05:14:45Z [verbose] Informer factories started
2023-05-17T05:14:45Z [verbose] starting network controller
W0517 05:14:45.704391      30 reflector.go:324] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
E0517 05:14:45.704565      30 reflector.go:138] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.NetworkAttachmentDefinition: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
W0517 05:14:46.992809      30 reflector.go:324] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
E0517 05:14:46.992854      30 reflector.go:138] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.NetworkAttachmentDefinition: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
W0517 05:14:49.298524      30 reflector.go:324] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
E0517 05:14:49.298553      30 reflector.go:138] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.NetworkAttachmentDefinition: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
W0517 05:14:53.864023      30 reflector.go:324] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
E0517 05:14:53.864050      30 reflector.go:138] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.NetworkAttachmentDefinition: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
W0517 05:15:04.670321      30 reflector.go:324] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
E0517 05:15:04.670370      30 reflector.go:138] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.NetworkAttachmentDefinition: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
W0517 05:15:18.322701      30 reflector.go:324] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
E0517 05:15:18.322719      30 reflector.go:138] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.NetworkAttachmentDefinition: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
W0517 05:15:47.935805      30 reflector.go:324] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
E0517 05:15:47.935843      30 reflector.go:138] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.NetworkAttachmentDefinition: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
W0517 05:16:20.857929      30 reflector.go:324] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
E0517 05:16:20.857950      30 reflector.go:138] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.NetworkAttachmentDefinition: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
W0517 05:16:59.897190      30 reflector.go:324] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
E0517 05:16:59.897240      30 reflector.go:138] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.NetworkAttachmentDefinition: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
W0517 05:17:45.361383      30 reflector.go:324] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
E0517 05:17:45.361474      30 reflector.go:138] github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.NetworkAttachmentDefinition: failed to list *v1.NetworkAttachmentDefinition: the server could not find the requested resource (get network-attachment-definitions.k8s.cni.cncf.io)
2023-05-17T05:31:56Z [verbose] deleted pod [default/netshoot-deployment-77bc5477c4-25jg5]
2023-05-17T05:31:56Z [verbose] skipped net-attach-def for default network
2023-05-17T05:31:56Z [debug] pod's network status: {Name:default/whereabouts-conf Interface:lo IPs:[127.0.0.1 ::1] Mac:00:00:00:00:00:00 Default:false DNS:{Nameservers:[] Domain: Search:[] Options:[]} DeviceInfo:<nil>}
2023-05-17T05:31:56Z [verbose] the NAD's config: {{ "cniVersion": "0.3.0", "name": "whereaboutsexample", "type": "loopback", "ipam": { "type": "whereabouts", "range": "192.168.2.225/28" } }}
2023-05-17T05:31:56Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
2023-05-17T05:31:56Z [verbose] result of garbage collecting pods: failed to get the IPPool data: ippool.whereabouts.cni.cncf.io "192.168.2.224-28" not found
2023-05-17T05:31:56Z [verbose] re-queuing IP address reconciliation request for pod default/netshoot-deployment-77bc5477c4-25jg5; retry #: 0
2023-05-17T05:31:56Z [verbose] skipped net-attach-def for default network
2023-05-17T05:31:56Z [debug] pod's network status: {Name:default/whereabouts-conf Interface:lo IPs:[127.0.0.1 ::1] Mac:00:00:00:00:00:00 Default:false DNS:{Nameservers:[] Domain: Search:[] Options:[]} DeviceInfo:<nil>}
2023-05-17T05:31:56Z [verbose] the NAD's config: {{ "cniVersion": "0.3.0", "name": "whereaboutsexample", "type": "loopback", "ipam": { "type": "whereabouts", "range": "192.168.2.225/28" } }}
2023-05-17T05:31:56Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
2023-05-17T05:31:56Z [verbose] result of garbage collecting pods: failed to get the IPPool data: ippool.whereabouts.cni.cncf.io "192.168.2.224-28" not found
2023-05-17T05:31:56Z [verbose] re-queuing IP address reconciliation request for pod default/netshoot-deployment-77bc5477c4-25jg5; retry #: 1
2023-05-17T05:31:56Z [verbose] skipped net-attach-def for default network
2023-05-17T05:31:56Z [debug] pod's network status: {Name:default/whereabouts-conf Interface:lo IPs:[127.0.0.1 ::1] Mac:00:00:00:00:00:00 Default:false DNS:{Nameservers:[] Domain: Search:[] Options:[]} DeviceInfo:<nil>}
2023-05-17T05:31:56Z [verbose] the NAD's config: {{ "cniVersion": "0.3.0", "name": "whereaboutsexample", "type": "loopback", "ipam": { "type": "whereabouts", "range": "192.168.2.225/28" } }}
2023-05-17T05:31:56Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
2023-05-17T05:31:56Z [verbose] result of garbage collecting pods: failed to get the IPPool data: ippool.whereabouts.cni.cncf.io "192.168.2.224-28" not found
2023-05-17T05:31:56Z [verbose] re-queuing IP address reconciliation request for pod default/netshoot-deployment-77bc5477c4-25jg5; retry #: 2
2023-05-17T05:31:56Z [verbose] skipped net-attach-def for default network
2023-05-17T05:31:56Z [debug] pod's network status: {Name:default/whereabouts-conf Interface:lo IPs:[127.0.0.1 ::1] Mac:00:00:00:00:00:00 Default:false DNS:{Nameservers:[] Domain: Search:[] Options:[]} DeviceInfo:<nil>}
2023-05-17T05:31:56Z [verbose] the NAD's config: {{ "cniVersion": "0.3.0", "name": "whereaboutsexample", "type": "loopback", "ipam": { "type": "whereabouts", "range": "192.168.2.225/28" } }}
2023-05-17T05:31:56Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
2023-05-17T05:31:56Z [verbose] result of garbage collecting pods: failed to get the IPPool data: ippool.whereabouts.cni.cncf.io "192.168.2.224-28" not found
2023-05-17T05:31:56Z [error] dropping pod [default/netshoot-deployment-77bc5477c4-25jg5] deletion out of the queue - could not reconcile IP: failed to get the IPPool data: ippool.whereabouts.cni.cncf.io "192.168.2.224-28" not found
2023-05-17T05:31:56Z [verbose] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"netshoot-deployment-77bc5477c4-25jg5", UID:"6618914d-126f-4e1e-919e-2e9cf664899f", APIVersion:"v1", ResourceVersion:"1084002", FieldPath:""}): type: 'Warning' reason: 'IPAddressGarbageCollectionFailed' failed to garbage collect addresses for pod default/netshoot-deployment-77bc5477c4-25jg5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment