Working of kubernetes cluster IP

k8s-cluster-ip.md

Working of kubernetes cluster IP

The by default created Service named kubernetes can be used to connect to the Kubernetes API. Behind this Service, there are endpoints for each control plane/master node.

Kubernetes docs on Service: https://kubernetes.io/docs/concepts/services-networking/service/

Commands to lookup Service and associated Endpoints:

> kubectl get svc kubernetes
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   54m
> kubectl get ep kubernetes
NAME         ENDPOINTS                                                 AGE
kubernetes   172.26.17.102:6443,172.26.18.77:6443,172.26.26.247:6443   54m

Showing what endpoint/host is hit when querying kube-apiserver

From https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/

Image used: rancherlabs/swiss-army-knife

# Point to the internal API server hostname
APISERVER=https://kubernetes.default.svc

# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt

# Explore the API with TOKEN
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api
172.26.17.102:6443

# Loop to see each endpoint
while true; do curl -s --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api | jq -r '.serverAddressByClientCIDRs[].serverAddress'; sleep 1; done
172.26.17.102:6443
172.26.18.77:6443
172.26.17.102:6443
172.26.26.247:6443

Rancher

Rancher can proxy calls to the managed clusters via /k8s endpoint.

RANCHER_URL=rancher.yourdomain.com
TOKEN=token-xxxxxx:xxxxxx
CLUSTERID=c-xxxxxx
curl -k \
     -H "Authorization: Bearer $TOKEN" \
     https://$RANCHER_URL/k8s/clusters/$CLUSTERID/api

Loop to see endpoint (this will only show the IP address of the apiEndpoint for the cluster)

RANCHER_URL=rancher.yourdomain.com
TOKEN=token-xxxxxx:xxxxxx
CLUSTERID=c-xxxxxx
while true; do curl -s -k -H "Authorization: Bearer $TOKEN" https://$RANCHER_URL/k8s/clusters/$CLUSTERID/api | jq -r '.serverAddressByClientCIDRs[].serverAddress'; sleep 1; done
172.26.26.247:6443
172.26.26.247:6443
172.26.26.247:6443
172.26.26.247:6443
172.26.26.247:6443
172.26.26.247:6443
172.26.26.247:6443
172.26.26.247:6443
172.26.26.247:6443
172.26.26.247:6443

The reason for this seems to be the clusterrouter taking the apiEndpoint for the cluster to send the request to:

https://github.com/rancher/rancher/blob/v2.5.9/pkg/clusterrouter/proxy/proxy_server.go#L116

func NewRemote(cluster *v3.Cluster, clusterLister v3.ClusterLister, factory dialer.Factory) (*RemoteService, error) {
	if !v32.ClusterConditionProvisioned.IsTrue(cluster) {
		return nil, httperror.NewAPIError(httperror.ClusterUnavailable, "cluster not provisioned")
	}

	urlGetter := func() (url.URL, error) {
		newCluster, err := clusterLister.Get("", cluster.Name)
		if err != nil {
			return url.URL{}, err
		}

		u, err := url.Parse(newCluster.Status.APIEndpoint)