Get `kubectl` access to your private cluster from anywhere

expose-apiserver-kubernetes-inlets-pro.md

Update: This gist has graduated to a full blog post over on my site -> https://blog.alexellis.io/get-private-kubectl-access-anywhere/

Get kubectl access to your private cluster from anywhere

This tutorial shows you how to punch your private Kubernetes API server out to the Internet, so that you can manage your cluster from anywhere, just like you would with a cloud offering.

These steps have been tested with kubeadm, k3s and OpenShift.

You'll need:

• A host with Kubernetes running - installed via kubeadm or k3s, this will be on your private network
• An access key / API token for public cloud, where a host will be provisioned
• A laptop that will connect to your Kubernetes cluster over the public IP
• inlets-pro and a license, get a free 14-day trial here

On the private host behind a firewall/NAT where Kubernetes is running, you'll see two things:

• TCP traffic served on port 6443
• A KUBECONFIG file at $HOME/.kube/config

Let's get started with the guide. We'll start by creating an exit-node which will run the inlets-pro server. Our laptop will connect to this address to access our private cluster.

Get the inlets binaries

On the Kubernetes host install inlets-pro and inletsctl

curl -sLSf https://inletsctl.inlets.dev | sudo sh

sudo inletsctl download --pro

Now provision an exit node to a cloud provider such as DigitalOcean.

• --remote-tcp - find the IP of your primary Ethernet adapter, for a home network this may be something like 192.168.0.10
• --access-token-file - get this from your cloud provider's dashboard
• --provider - run inletsctl create --help for a list

inletsctl create \
  --remote-tcp 192.168.0.10 \
  --access-token-file ~/Downloads/do-access-token \
  --provider digitalocean

You'll see output like this when your exit-server and public IP are ready:

inlets-pro exit-node summary:
  IP: 159.65.82.66
  Auth-token: fKuyjI4QY12zDyU6Kjwyox6sPgIf65wY1eVGTGDVK9nRWsRbBBI1pACNMRTLnJKk

Command:
  export TCP_PORTS="8000"
  export LICENSE=""
  inlets-pro client --connect "wss://159.65.82.66:8123/connect" \
	--token "fKuyjI4QY12zDyU6Kjwyox6sPgIf65wY1eVGTGDVK9nRWsRbBBI1pACNMRTLnJKk" \
	--license "$LICENSE" \
	--tcp-ports $TCP_PORTS

To Delete:
	  inletsctl delete --provider digitalocean --id "175227193"

Run the tunnel

Now edit the parameters:

• Set export TCP_PORTS=6443" - the port of the apiserver in Kubernetes
• Set export LICENSE="" to your inlets-pro key

Now run the command:

export TCP_PORTS="6443"

# Valid for 3 days as of 11 Jan
export LICENSE="eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYW1lIjoiQWxleCBFbGxpcyIsImVtYWlsX2FkZHJlc3MiOiJhbGV4QG9wZW5mYWFzLmNvbSIsImF1ZCI6ImlubGV0cy1wcm8iLCJleHAiOjE1NzkwNDE4ODcsImp0aSI6IjgwODEiLCJpYXQiOjE1Nzg3ODI2ODcsImlzcyI6ImlubGV0cy1wcm8iLCJzdWIiOiJBbGV4IEVsbGlzIn0.cd5763OEDdwcujD5zzA3CMemL08qCEbcdYsmXIybfWU3jgAKQSO12ZZ_oZlkrnjNNjIhbXe2NMNZYrHOPTrgbA"

inlets-pro client --connect "wss://142.93.41.125:8123/connect" \
--token "1L8CjqlDt0DACfNwcjwDMzEbwfZRttlbV80UhBIsCpkqaS5nM4Vlk6l3rZLoINX0" \
--license "$LICENSE" \
--tcp-ports $TCP_PORTS

The tunnel is now established and you can use curl to test it.

curl -i -k https://159.65.82.66:6443

curl -k https://142.93.41.125:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

You'll see an error saying access denied, that's fine. It just shows that we need to get a valid kubeconfig file.

Get the KUBECONFIG file

Copy the ~/.kube/config file from your Kubernetes host to your laptop.

Now edit and replace its IP address with the IP of the public node.

scp 192.168.0.10:~/.kube/config config

sed -ie s/192.168.0.10/142.93.41.125/g config

Access your cluster

export KUBECONFIG=`pwd`/config

kubectl get pods -A

You may see an error about the public IP not matching the TLS certificate, there are two ways you can resolve this.

1. Use the --insecure-skip-tls-verify flag

   kubectl get pods -A --insecure-skip-tls-verify
   

2. If you're using k3s instead of kubeadm, you can edit the TLS SAN value in the k3s system unit file and restart k3s, see systemctl cat k3s

   ExecStart=/usr/local/bin/k3s \
   server \
       '--tls-san' \
       '142.93.41.125' \
   

   Now you'll no-longer need to use --insecure-skip-tls-verify

3. Run kubeadm init again and supply the public IP via --apiserver-cert-extra-sans

   See kubeadm init --help for more information.

4. Update your kubeadm config without reinstalling - advanced Adding a Name to the Kubernetes API Server Certificate

To test that you can access your cluster from anywhere in the world try connecting from a coffee shop, a different WiFi network, or your mobile hotspot.

Wrapping up

In much the same way as AWS or GKE provide a public endpoint for your Kubernetes API Server, we've been able to achieve the same using inlets-pro for your Kubernetes cluster running on your private network.

You can also use inlets-pro to punch out from one private network to another, where incoming connections are not allowed.

Would you like to know more? Contact me on Twitter

Appendix

For k3d run:

k3d create --api-port 0.0.0.0:6443

Get the config via:

cat $(k3d get-kubeconfig --name='k3s-default')

Replace 0.0.0.0 with the public IP of the exit-server.