service loadbalancer deleteion

README.md

Service LoadBalancer deletion

Services of Type: LoadBalancer are handled by external controllers.

These controllers need to handle the infrastructure necessary to provisiong the external Loadbalancer, and use finalizers to avoid deleting the Service before all the necessary steps are done.

Kubernetes provide a reference implementation of a controller for Loadbalancers in https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/cloud-provider/controllers/service/controller.go

Example

kubectl apply -f https://gist.githubusercontent.com/aojea/3ac345b81aae3d3bd18bd57cb56450e3/raw/2a3776659360ba233ac1f01f671446db323fa118/loadbalancer.yaml
deployment.apps/test-deployment configured
service/lb-service created

In another terminal start watching the Service

kubectl get service lb-service --watch
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
lb-service   LoadBalancer   10.92.14.171   <pending>     8080:32515/TCP   12s
lb-service   LoadBalancer   10.92.14.171   192.158.28.237   8080:32515/TCP   41s

Once the ExternalIP is acquired, delete the Service without closing the watch, add the -v7 flag to kubectl so we can observe the calls dony by kubectl

 kubectl delete service lb-service -v7
I0511 14:58:26.383669  921501 loader.go:373] Config loaded from file:  /usr/local/google/home/aojea/.kube/config
I0511 14:58:26.389971  921501 round_trippers.go:463] DELETE https://34.22.135.47/api/v1/namespaces/default/services/lb-service
I0511 14:58:26.389996  921501 round_trippers.go:469] Request Headers:
I0511 14:58:26.390004  921501 round_trippers.go:473]     Accept: application/json
I0511 14:58:26.390010  921501 round_trippers.go:473]     Content-Type: application/json
I0511 14:58:26.390015  921501 round_trippers.go:473]     User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112
I0511 14:58:26.461063  921501 round_trippers.go:574] Response Status: 200 OK in 71 milliseconds
service "lb-service" deleted
I0511 14:58:26.461312  921501 round_trippers.go:463] GET https://34.22.135.47/api/v1/namespaces/default/services/lb-service
I0511 14:58:26.461334  921501 round_trippers.go:469] Request Headers:
I0511 14:58:26.461343  921501 round_trippers.go:473]     Accept: application/json
I0511 14:58:26.461350  921501 round_trippers.go:473]     User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112
I0511 14:58:26.471591  921501 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0511 14:58:26.472491  921501 reflector.go:287] Starting reflector *unstructured.Unstructured (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I0511 14:58:26.472514  921501 reflector.go:323] Listing and watching *unstructured.Unstructured from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I0511 14:58:26.472595  921501 round_trippers.go:463] GET https://34.22.135.47/api/v1/namespaces/default/services?fieldSelector=metadata.name%3Dlb-service&limit=500&resourceVersion=0
I0511 14:58:26.472609  921501 round_trippers.go:469] Request Headers:
I0511 14:58:26.472616  921501 round_trippers.go:473]     Accept: application/json
I0511 14:58:26.472623  921501 round_trippers.go:473]     User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112
I0511 14:58:26.517380  921501 round_trippers.go:574] Response Status: 200 OK in 44 milliseconds
I0511 14:58:26.517782  921501 round_trippers.go:463] GET https://34.22.135.47/api/v1/namespaces/default/services?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dlb-service&resourceVersion=55105085&timeoutSeconds=404&watch=true
I0511 14:58:26.517800  921501 round_trippers.go:469] Request Headers:
I0511 14:58:26.517807  921501 round_trippers.go:473]     Accept: application/json
I0511 14:58:26.517814  921501 round_trippers.go:473]     User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112
I0511 14:58:26.526200  921501 round_trippers.go:574] Response Status: 200 OK in 8 milliseconds
I0511 14:58:26.573419  921501 shared_informer.go:341] caches populated
I0511 14:58:57.500045  921501 reflector.go:293] Stopping reflector *unstructured.Unstructured (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146

We can see that kubectl establish a watch and holds until the Service disappears

GET https://34.22.135.47/api/v1/namespaces/default/services?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dlb-service&resourceVersion=55105085&timeoutSeconds=404&watch=true
I0511 14:58:26.517800  921501 round_trippers.go:469] Request Headers:

If we check the controller-manager logs we can see that the controller use finalizers to avoid the Service object to be deleted before it is able to clean up the associated resources

https://github.com/kubernetes/kubernetes/blob/e1ad9bee5bba8fbe85a6bf6201379ce8b1a611b1/staging/src/k8s.io/cloud-provider/controllers/service/controller.go#L388-L403

https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/

loadbalancer.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
  labels:
    app: MyApp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: MyApp
  template:
    metadata:
      labels:
        app: MyApp
    spec:
      containers:
      - name: agnhost8080
        image: k8s.gcr.io/e2e-test-images/agnhost:2.41
        args:
          - netexec
          - --http-port=8080
          - --udp-port=8080
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: lb-service
spec:
  type: LoadBalancer
  selector:
    app: MyApp
  ports:
    - name: port8080
      protocol: TCP
      port: 8080
      targetPort: 8080