간단한 cloud init을 활용한 쿠버네티스 클러스터링 방법

README.md

Simple kubernetes clustering by cloud-init

kubernetes 로 클러스터를 만드는 과정이 복잡하다고 생각하실 수 있지만 사실 스크립트로 만들면 이정도로 간단해 집니다. kubernetes, docker package가 준비되었다고 한다면 남는건 3가지 종류의 프로세스가 있습니다.

1. 첫번째 마스터

$./init_master.sh $MASTER_IP

• kubeadm init 으로 셋업
• (옵션) CNI등 네트워크 셋업 (사실 클러스터링과 관계 없음)
• kubeadm init 으로 생성된 certification 등을 다른 이후 서버들에게 제공

2. 두번째 이후 마스터

$./join_master.sh $MASTER_IP master

• 첫번째 마스터가 파일을 서빙 할때까지 대기
• 파일 서빙되면 certification 을 다운로드
• 이후 kubeadm join

3. 워커

$./join_master.sh $MASTER_IP

• 첫번째 마스터가 파일을 서빙 할때까지 대기
• 이후 kubeadm join

위의 3가지 프로세스는 모두 동시에 시작합니다. 그래서 테라폼으로 모든 vm node가 각자 필요한 cloud-init 스크립트를 가진 상태로 시작하면 이후 모두 클러스터링이 끝난 상태로 정리가 됩니다.

만약 100년 짜리 certification을 갖는 클러스터를 만들려면 아래와 같은 방법으로 미리 certification을 만들어서 모든 마스터가 나눠 가진 상태에서 시작하면 됩니다.

https://github.com/leoh0/k8s-cluster-api-cert-gen

init_master.sh

#!/bin/bash -eux

K8S_API=$1
if [ -z "${K8S_API}" ]; then
    echo "K8S_API : \"${K8S_API}\" is not exit. Quit..."
    exit 0
fi

######################## INSTALL K8S FIRST MASTER START ########################

cat << EOF > /tmp/init-config.yaml
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer: {}
clusterName: ring0
controlPlaneEndpoint: ${K8S_API}:6443
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
kubernetesVersion: v1.19.2
networking:
  dnsDomain: cluster.local
  podSubnet: 192.168.0.0/18
  serviceSubnet: 10.96.0.0/12
scheduler: {}

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: ""
  bindPort: 0
nodeRegistration:
  name: $(hostname -s)
  kubeletExtraArgs:
    node-ip: "${INTERNALIP}"
EOF
kubeadm init --config /tmp/init-config.yaml

# cloud init 에서는 HOME 변수가 없어서 / 를 home으로 동작 할 수 있다.
export HOME=/root
mkdir -p $HOME/.kube
cp /etc/kubernetes/admin.conf $HOME/.kube/config

kubectl taint nodes "$(hostname -s)" node-role.kubernetes.io/master-

######################## INSTALL K8S FIRST MASTER END ########################

######################## INSTALL CNI START ########################

curl -Ls -o /tmp/helm-v3.2.1-linux-amd64.tar.gz https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
cd /tmp && tar zxvf /tmp/helm-v3.2.1-linux-amd64.tar.gz && cd -
mv /tmp/linux-amd64/helm /tmp/helm

# install cilium
/tmp/helm repo add cilium https://helm.cilium.io/
/tmp/helm repo update

cd /tmp && /tmp/helm pull cilium/cilium --untar

/tmp/helm install cilium /tmp/cilium --version 1.7.4 \
  --namespace kube-system

kubectl wait --for=condition=Available --timeout=300s -n kube-system deployment cilium-operator
kubectl wait --for=condition=Available --timeout=300s -n kube-system deployment coredns

######################## INSTALL CNI END ########################

######################## SERVING CERTS FOR OTHER MASTER START ########################

# send to other master, nodes
mkdir -p /tmp/metadata
cd /tmp/metadata

CAHASH=$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')
TOKEN=$(kubeadm token list | awk '/authentication/{print $1}')

cp /etc/kubernetes/admin.conf admin.conf
cp /etc/kubernetes/pki/etcd/ca.crt etcd-ca.crt
cp /etc/kubernetes/pki/etcd/ca.key etcd-ca.key
cp /etc/kubernetes/pki/ca.crt ca.crt
cp /etc/kubernetes/pki/ca.key ca.key
cp /etc/kubernetes/pki/front-proxy-ca.crt front-proxy-ca.crt
cp /etc/kubernetes/pki/front-proxy-ca.key front-proxy-ca.key
cp /etc/kubernetes/pki/sa.key sa.key
cp /etc/kubernetes/pki/sa.pub sa.pub

echo "${K8S_API} ${CAHASH} ${TOKEN}" > k8s

nohup python3 -m http.server 8880 &

######################## SERVING CERTS FOR OTHER MASTER END ########################

join_master.sh

#!/bin/bash -eux

K8S_API=$1
if [ -z "${K8S_API}" ]; then
    echo "K8S_API : \"${K8S_API}\" is not exit. Quit..."
    exit 0
fi

######################## WAITING FOR FIRST MASTER START ########################
DEFAULT_PORT=${DEFAULT_PORT:-8880}

count=0
while true ; do
    if curl -s "$K8S_API:$DEFAULT_PORT/k8s" ; then
        DATA=$(curl -s "$K8S_API:$DEFAULT_PORT/k8s")
        K8S_API=$(echo "$DATA" | cut -d' ' -f1)
        CAHASH=$(echo "$DATA" | cut -d' ' -f2)
        TOKEN=$(echo "$DATA" | cut -d' ' -f3)

        break
    fi

    count=$((count+1))

    if [[ ${count} == "3600" ]]; then
        break
    fi

    sleep 1
done

if [ -z "$K8S_API" ] || [ -z "$CAHASH" ] || [ -z "$TOKEN" ] ; then
    echo "Some value is empty. k8s_api : $K8S_API, cahash : $CAHASH, token :$TOKEN, Quit..."
    exit 0
fi

######################## WAITING FOR FIRST MASTER END ########################

######################## INSTALL CERTS FOR MASTERS START ########################

ISMASTER=$2
EXTRACONFIG=""
if [[ ${ISMASTER} == "master" ]]; then
    EXTRACONFIG='controlPlane:
  localAPIEndpoint:
    advertiseAddress: ""
    bindPort: 0
'
    mkdir -p /etc/kubernetes/pki/etcd/
    curl -o /etc/kubernetes/pki/etcd/ca.crt "$K8S_API:$DEFAULT_PORT/etcd-ca.crt"
    curl -o /etc/kubernetes/pki/etcd/ca.key "$K8S_API:$DEFAULT_PORT/etcd-ca.key"
    curl -o /etc/kubernetes/pki/ca.crt "$K8S_API:$DEFAULT_PORT/ca.crt"
    curl -o /etc/kubernetes/pki/ca.key "$K8S_API:$DEFAULT_PORT/ca.key"
    curl -o /etc/kubernetes/pki/front-proxy-ca.crt "$K8S_API:$DEFAULT_PORT/front-proxy-ca.crt"
    curl -o /etc/kubernetes/pki/front-proxy-ca.key "$K8S_API:$DEFAULT_PORT/front-proxy-ca.key"
    curl -o /etc/kubernetes/pki/sa.key "$K8S_API:$DEFAULT_PORT/sa.key"
    curl -o /etc/kubernetes/pki/sa.pub "$K8S_API:$DEFAULT_PORT/sa.pub"
fi

######################## INSTALL CERTS FOR MASTERS END ########################

######################## JOIN TO CLUSTER START ########################

INTERNALIP=$(ip -f inet -o addr show eth0 | cut -d\  -f 7 | cut -d/ -f 1)
cat << EOF > /tmp/join-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
${EXTRACONFIG}
discovery:
  bootstrapToken:
    apiServerEndpoint: ${K8S_API}:6443
    caCertHashes:
    - sha256:${CAHASH}
    token: ${TOKEN}
    unsafeSkipCAVerification: false
kind: JoinConfiguration
nodeRegistration:
  name: $(hostname -s)
  kubeletExtraArgs:
    node-ip: "${INTERNALIP}"
EOF
kubeadm join --config /tmp/join-config.yaml

######################## JOIN TO CLUSTER END ########################