Deploy a Production Ready Kubernetes Cluster using Kubespray with Ansible

kubernetes-cluster-using-kubespray.md

Deployment Architecture

System Configuration

1. All kubernetes nodes: set SELINUX to permissive mode

$ vi /etc/selinux/config
SELINUX=permissive

$ setenforce 0

2. All kubernetes nodes: set iptables parameters

$ cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

$ sysctl --system

3. All kubernetes nodes: disable swap

$ swapoff -a

# disable swap mount point in /etc/fstab
$ vi /etc/fstab
#/dev/mapper/centos-swap swap                    swap    defaults        0 0

# check swap is disabled
$ cat /proc/swaps
Filename                Type        Size    Used    Priority

4. Stop and Disable UFW Service

$systemctl stop ufw.service
$systemctl disable ufw.service

5. All kubernetes nodes: reboot hosts

# reboot hosts
$ reboot

firewalld and iptables settings

1. All kubernetes nodes: enable firewalld

# restart firewalld service
$ systemctl enable firewalld
$ systemctl restart firewalld
$ systemctl status firewalld

2. Master ports list

Protocol	Direction	Port	Comment
TCP	Inbound	16443*	Load balancer Kubernetes API server port
TCP	Inbound	6443*	Kubernetes API server
TCP	Inbound	4001	etcd listen client port
TCP	Inbound	2379-2380	etcd server client API
TCP	Inbound	10250	Kubelet API
TCP	Inbound	10251	kube-scheduler
TCP	Inbound	10252	kube-controller-manager
TCP	Inbound	10255	Read-only Kubelet API (Deprecated)
TCP	Inbound	30000-32767	NodePort Services

• All master nodes: set firewalld policy

$ firewall-cmd --zone=public --add-port=16443/tcp --permanent
$ firewall-cmd --zone=public --add-port=6443/tcp --permanent
$ firewall-cmd --zone=public --add-port=4001/tcp --permanent
$ firewall-cmd --zone=public --add-port=2379-2380/tcp --permanent
$ firewall-cmd --zone=public --add-port=10250/tcp --permanent
$ firewall-cmd --zone=public --add-port=10251/tcp --permanent
$ firewall-cmd --zone=public --add-port=10252/tcp --permanent
$ firewall-cmd --zone=public --add-port=30000-32767/tcp --permanent

$ firewall-cmd --reload

$ firewall-cmd --list-all --zone=public
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens2f1 ens1f0 nm-bond
  sources:
  services: ssh dhcpv6-client
  ports: 4001/tcp 6443/tcp 2379-2380/tcp 10250/tcp 10251/tcp 10252/tcp 30000-32767/tcp

3. Worker ports list

Protocol	Direction	Port	Comment
TCP	Inbound	10250	Kubelet API
TCP	Inbound	30000-32767	NodePort Services

• All worker nodes: set firewalld policy

$ firewall-cmd --zone=public --add-port=10250/tcp --permanent
$ firewall-cmd --zone=public --add-port=30000-32767/tcp --permanent

$ firewall-cmd --reload

$ firewall-cmd --list-all --zone=public
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens2f1 ens1f0 nm-bond
  sources:
  services: ssh dhcpv6-client
  ports: 10250/tcp 30000-32767/tcp

4. All kubernetes nodes: set firewalld to enable kube-proxy port forward

$ firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 1 -i docker0 -j ACCEPT -m comment --comment "kube-proxy redirects"
$ firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 1 -o docker0 -j ACCEPT -m comment --comment "docker subnet"
$ firewall-cmd --reload

$ firewall-cmd --direct --get-all-rules
ipv4 filter INPUT 1 -i docker0 -j ACCEPT -m comment --comment 'kube-proxy redirects'
ipv4 filter FORWARD 1 -o docker0 -j ACCEPT -m comment --comment 'docker subnet'

# restart firewalld service
$ systemctl restart firewalld

Tools to be installed on Ansible Server

• Ansible v2.7

$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo apt-add-repository --yes --update ppa:ansible/ansible
$ sudo apt-get install ansible
$ ansible — version

• python-netaddr

$ apt-get install python-netaddr

• Jinja 2.9

$ sudo apt-get install python-pip
$ pip2 install jinja2 --upgrade

Setup passwordless SSH between Ansible server and k8s nodes

On Ansible server

1. Create Authentication SSH-Kegen Keys

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa): [Press enter key]
Created directory '/home/ubuntu/.ssh'.
Enter passphrase (empty for no passphrase): [Press enter key]
Enter same passphrase again: [Press enter key]
Your identification has been saved in /home/ubuntu/.ssh/id_rsa.
Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub.
The key fingerprint is:
5f:ad:40:00:8a:d1:9b:99:b3:b0:f8:08:99:c3:ed:d3 ubuntu
The key's randomart image is:
+--[ RSA 2048]----+
|        ..oooE.++|
|         o. o.o  |
|          ..   . |
|         o  . . o|
|        S .  . + |
|       . .    . o|
|      . o o    ..|
|       + +       |
|        +.       |
+-----------------+

2. Create .ssh Directory on k8s nodes

$ ssh ubuntu@10.0.0.4 mkdir -p .ssh
$ ssh ubuntu@10.0.0.5 mkdir -p .ssh

3. Copy our newly generated public key to the k8s nodes

# After running this command you will be able to SSH into the machine directly without using a password
$ cat ~/.ssh/id_rsa.pub | ssh ubuntu@10.0.0.4 'cat >> ~/.ssh/authorized_keys'

# Replace 10.0.0.4 with other k8s nodes IP.

4. Set permissions on .ssh directory and authorized_keys file.

$ ssh ubuntu@10.0.0.4 "chmod 700 .ssh; chmod 640 .ssh/authorized_keys"
$ ssh ubuntu@10.0.0.5 "chmod 700 .ssh; chmod 640 .ssh/authorized_keys"

Now we can log into k8s nodes from Ansible server without password.