How to connect to Azure AKS Kubernetes node VM by SSH

README.md

How to connect to Azure AKS Kubernetes worker node by SSH

Nodes are not assigned public IP. If you have accessible VM in the same VNET as worker nodes, then you can use that VM as jump host and connect the worker via private IP.

Alternatively public IP can be assigned to a worker node. This readme shows how to do that.

Steps how to attach public IP to a worker node

find out the resource group that AKS created for the node VMs

az group list -o table

list resources in the group and find the VM you want to access

az resource list -g MC_kubernetes_kubernetes-cluster_ukwest -o table

show parameters of that VM, see for example: "adminUsername": "azureuser"

az vm show -g MC_kubernetes_kubernetes-cluster_ukwest -n aks-agentpool1-18549766-0

create the public IP

az network public-ip create -g MC_kubernetes_kubernetes-cluster_ukwest -n test-ip

find out correct NIC where to add the public IP

az network nic list -g MC_kubernetes_kubernetes-cluster_ukwest -o table

find out the name of the ipconfig within that NIC

az network nic ip-config list --nic-name aks-agentpool1-18549766-nic-0 -g MC_kubernetes_kubernetes-cluster_ukwest

modify the ipconfig by adding the public IP address

az network nic ip-config update -g MC_kubernetes_kubernetes-cluster_ukwest --nic-name aks-agentpool1-18549766-nic-0 --name ipconfig1 --public-ip-address test-ip

find out what the allocated public IP address is

az network public-ip show -g MC_kubernetes_kubernetes-cluster_ukwest -n test-ip

then finally connect with SSH

ssh azureuser@<public ip address>

to later remove the public-ip, just pass an empty string:

az network nic ip-config update -g MC_kubernetes_kubernetes-cluster_ukwest --nic-name aks-agentpool1-18549766-nic-0 --name ipconfig1 --public-ip-address ""

I found my node name by

kubectl get nodes

and replaced the node name but still cannot get the vm info by command:

az vm show -g MC_kubernetes_kubernetes-cluster_ukwest -n aks-agentpool1-xxxxxx-0

[updated]
I finally found that there is another Resource Group its name starts with MC_!!!!!!!

didnt work for me :(

If you performed the default AKS creation via the CLI, you will have two more steps. A Network Security Group rule needs to be added in order to connect via SSH. By default, the nodes are closed to the outside world.

Addtl Steps:

1. az network nsg list -g [MC resource group name] -o table
2. az network nsg rule create -g [MC resource group name] --nsg-name [nsg name]-n SshRule --priority 100 --source-address-prefixes Internet --destination-port-ranges 22 --access Allow --protocol Tcp --direction Inbound

Everything worked for me except for the main part ssh'ing. When I try to ssh, I am getting "Connection timed out" message. Anyone has any idea what went wrong?

[Edit]
I followed the Additional steps from @stevekmicrosoft, and now I get Permission denied(publickey) error.

Here is another approach to SSH into AKS node
https://github.com/yokawasa/kubectl-plugin-ssh-jump

Everything worked for me except for the main part ssh'ing. When I try to ssh, I am getting "Connection timed out" message. Anyone has any idea what went wrong?

[Edit]
I followed the Additional steps from @stevekmicrosoft, and now I get Permission denied(publickey) error.

I had the same problem (got to the public key error). To solve I just followed the steps detailed here: MicrosoftDocs/azure-docs#7463

-regenerate SSH keys
-In the portal, select an AKS node VM and reset password
-Select Reset SSH public key, enter the username azureuser by default, and paste in the new public key id_rsa.pub
-Create SSH connection as described in the doc

@TSAANRNI,

I was able to get the ssh connection to work. To "SteveKMicrosoft's" point, you have to add a rule to the NSG to allow the connection through and what that means is identifying the NSG (you can do that through the gui) or cmdline
► rg="resource-group"
► az network nsg list -g $rg -o table
Location Name ProvisioningState ResourceGroup ResourceGuid

---

eastus aks-agentpool-12113221-nsg Succeeded MC_itots_ITCluster_eastus a72b5ba9-78da-4ad3-b49b-d4d84a919c76

Now we can create the rule to allow the system to connect to port 22

az network nsg rule create -g MC_itots_ITCluster_eastus --nsg-name aks-agentpool-12113221-nsg -n SshRule --priority 499 --source-address-prefixes Internet --destination-port-ranges 22 --access Allow --protocol Tcp --direction Inbound -o table

There is another piece that was left off, go to /home//.ssh, be sure to copy the file to a file on your desktop or copy the file contents, save it to a file, call it id_rsa.key, then open the file with puttygen, save the file to id_rsa.ppk, load the file to putty under "SSH | Auth | Private Key file for authentication"

Save it and with azureuser@ with the private key, then you are good to go.

I want to thank the team for their insight, this was a good learning exercise.

Tasks: 253 total, 1 running, 190 sleeping, 1 stopped, 0 zombie
%Cpu(s): 5.9 us, 2.7 sy, 0.0 ni, 90.9 id, 0.3 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 7113160 total, 505744 free, 1610612 used, 4996804 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 5086612 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
83305 root 20 0 32232 3428 2888 R 6.7 0.0 0:00.01 top
1 root 20 0 38604 6536 4032 S 0.0 0.1 4:56.32 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.13 kthreadd
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
7 root 20 0 0 0 0 S 0.0 0.0 2:37.20 ksoftirqd/0
8 root 20 0 0 0 0 I 0.0 0.0 25:51.34 rcu_sched
9 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_bh
10 root rt 0 0 0 0 S 0.0 0.0 0:21.84 migration/0
11 root rt 0 0 0 0 S 0.0 0.0 0:03.39 watchdog/0
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1
14 root rt 0 0 0 0 S 0.0 0.0 0:03.42 watchdog/1
15 root rt 0 0 0 0 S 0.0 0.0 0:22.38 migration/1
16 root 20 0 0 0 0 S 0.0 0.0 4:26.16 ksoftirqd/1
18 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/1:0H
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
20 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 netns
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_tasks_kthre
22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kauditd
25 root 20 0 0 0 0 S 0.0 0.0 0:01.38 khungtaskd
26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
27 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 writeback
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
29 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
30 root 39 19 0 0 0 S 0.0 0.0 0:03.51 khugepaged
31 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 crypto
32 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kintegrityd
T

You can also access nodes via privileged pods

You can try this approach. This is easy and quick. This creates a daemonset attached to the node. You will be able to login to the AKS nodes without going through the steps of ssh key creation and all.

Does this work with VMSS created by AKS nodepools? I have no network NIC in the ressource group of my VMSS.

Does this work with VMSS created by AKS nodepools? I have no network NIC in the ressource group of my VMSS.

Try this approach. It will work with AKS nodes. You do not need to attach any additional IP Address or ssh key to the worker nodes. This uses daemon-set approach. https://amalgjose.com/2021/05/26/how-to-ssh-into-azure-kubernetes-cluster-aks-worker-node/

Does this work with VMSS created by AKS nodepools? I have no network NIC in the ressource group of my VMSS.

Try this approach. It will work with AKS nodes. You do not need to attach any additional IP Address or ssh key to the worker nodes. This uses daemon-set approach. https://amalgjose.com/2021/05/26/how-to-ssh-into-azure-kubernetes-cluster-aks-worker-node/

Ah sorry I haven't noticed your latest message! I'll try it out asap and let you know how it went thank you