Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Generate Rancher 2 cluster/node agent definitions

Generate Rancher 2 cluster/node agents definitions

This is not official documentation/tooling, use with caution

This generate the Kubernetes definitions of the cattle-cluster-agent Deployment and cattle-node-agent DaemonSet, in case it's accidentally removed/server-url was changed/certficates were changed. It is supposed to run on every cluster Rancher manages. If you have custom clusters created in Rancher, see Kubeconfig for Custom clusters created in Rancher how to obtain the kubeconfig to directly talk to the Kubernetes API (as usually it doesn't work via Rancher anymore). For other clusters, use the tools provided by the provider to get the kubeconfig.

IMPORTANT: You get the cluster/node agents definitions from Rancher, and you apply them to the cluster that is created/managed so you need to switch kubeconfig to point to that cluster before applying them.

Running it

  • Supported OS: Linux and MacOS

Single install (or locally on a host added to the RKE cluster running a/the Rancher pod)

Note: as the definitions get out of date, please see https://gist.github.com/superseb/076f20146e012f1d4e289f5bd1bd4971 how to dynamically generate them instead

This uses docker ps to find the rancher/rancher container, so it should be run on the node running the rancher/rancher container.

Use the file content from get_agents_yaml_single.sh and append the cluster name to it, i.e. custom. The definitions will be created in rancher-agents-custom.yml.

bash get_agents_yaml_single.sh custom
Container ID running Rancher is 41a636e9bdc2
'kubectl get cluster' returns clusters available
Clusters found:
custom      rancherKubernetesEngine   c-9tc6c
yyy   azureKubernetesService    c-ps4c4
local     imported                  local
You can now configure kubeconfig to your created/managed cluster and run kubectl apply -f rancher-agents-custom.yml to restore cluster/node agents

HA

Note: as the definitions get out of date, please see https://gist.github.com/superseb/076f20146e012f1d4e289f5bd1bd4971 how to dynamically generate them instead

To use this approach, you need kubectl configured pointing the the RKE cluster created to run Rancher. If it's correct, you should get output from kubectl get clusters.

Use the file content from get_agents_yaml_ha.sh and append the cluster name to it, i.e. custom. The definitions will be created in rancher-agents-custom.yml.

$ bash get_agents_yaml_ha.sh custom
'kubectl get cluster' returns clusters available
Clusters found:
custom      rancherKubernetesEngine   c-9tc6c
yyy   azureKubernetesService    c-ps4c4
local     imported                  local
You can now configure kubeconfig to your created/managed cluster and run kubectl apply -f rancher-agents-custom.yml to restore cluster/node agents

Kubeconfig for Custom clusters created in Rancher

Note: as the definitions get out of date, please see https://gist.github.com/superseb/076f20146e012f1d4e289f5bd1bd4971 how to dynamically generate them instead

To launch the cluster/node agent definitions on Custom clusters created in Rancher, you have to obtain the kubeconfig to talk directly to the Kubernetes API for that cluster using this gist and the name of the cluster (in this case, custom)

curl https://gist.githubusercontent.com/superseb/f6cd637a7ad556124132ca39961789a4/raw/c2bb60a51fa4dafef2964589ce0bc9d923653aa1/get_kubeconfig_custom_cluster_rancher2.sh | bash -s -- custom

Then apply the definitions:

kubectl --kubeconfig kubeconfig apply -f rancher-agents-custom.yml
#!/bin/bash
# Usage: ./get_agents_yaml_ha.sh cluster_name
# Needs to have KUBECONFIG environment variable set or ~/.kube/config pointing to the RKE Rancher cluster
# Check if jq exists
command -v jq >/dev/null 2>&1 || { echo "jq is not installed. Exiting." >&2; exit 1; }
command -v kubectl >/dev/null 2>&1 || { echo "kubectl is not installed. Exiting." >&2; exit 1; }
# Check if clustername is given
if [ -z "$1" ]; then
echo "Usage: $0 [clustername]"
exit 1
fi
case "$OSTYPE" in
darwin*) MD5=md5; SHA256SUM="shasum -a 256" ;;
linux*) MD5=md5sum; SHA256SUM=sha256sum ;;
*) echo "unsupported: $OSTYPE"; exit 1 ;;
esac
# Provide clustername as first argument
CLUSTERNAME=$1
# Validate that we are querying the correct etcd
if kubectl get cluster > /dev/null; then
echo "'kubectl get cluster' returns clusters available"
else
echo "'kubectl get cluster' returns error, this should be run using a kubeconfig pointing to the RKE cluster running Rancher"
exit 1
fi
# Get clusters
CLUSTERINFO=$(kubectl get cluster --no-headers --output=custom-columns=Name:.spec.displayName,Driver:.status.driver,ClusterID:.metadata.name)
echo "Clusters found:"
echo "${CLUSTERINFO}"
# Get clusterid from clustername
CLUSTERID=$(echo "${CLUSTERINFO}" | awk -v CLUSTERNAME=$CLUSTERNAME '$1==CLUSTERNAME { print $3 }')
if [[ -z $CLUSTERID ]]; then
echo "No CLUSTERID could be retrieved for $CLUSTERNAME, make sure you entered the correct clustername"
exit 1
fi
# Get all needed settings
CACERTS=$(kubectl get setting cacerts -o json | jq -r .value)
if [ "x${CACERTS}" == "x" ]; then
CACHECKSUM=""
else
CACHECKSUM=$(echo "$CACERTS" | $SHA256SUM | awk '{ print $1 }')
fi
SERVERURL=$(kubectl get setting server-url -o json | jq -r .value)
AGENTIMAGE=$(kubectl get settings agent-image -o json | jq -r .value)
TOKEN=$(kubectl get clusterregistrationtoken system -n $CLUSTERID -o json | jq -r .status.token)
# Getting correct values
B64TOKEN=$(echo -n $TOKEN | base64)
B64URL=$(echo -n $SERVERURL | base64)
TOKENKEY=$(echo -n $TOKEN | $MD5 | cut -c1-7)
#echo $B64TOKEN
#echo $B64URL
#echo $TOKENKEY
cat >./rancher-agents-$CLUSTERNAME.yml <<EOL
---
apiVersion: v1
kind: Namespace
metadata:
name: cattle-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cattle
namespace: cattle-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: cattle-admin-binding
namespace: cattle-system
labels:
cattle.io/creator: "norman"
subjects:
- kind: ServiceAccount
name: cattle
namespace: cattle-system
roleRef:
kind: ClusterRole
name: cattle-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
metadata:
name: cattle-credentials-${TOKENKEY}
namespace: cattle-system
type: Opaque
data:
url: "${B64URL}"
token: "${B64TOKEN}"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cattle-admin
labels:
cattle.io/creator: "norman"
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: cattle-cluster-agent
namespace: cattle-system
spec:
selector:
matchLabels:
app: cattle-cluster-agent
template:
metadata:
labels:
app: cattle-cluster-agent
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: NotIn
values:
- windows
serviceAccountName: cattle
containers:
- name: cluster-register
imagePullPolicy: IfNotPresent
env:
- name: CATTLE_SERVER
value: "${SERVERURL}"
- name: CATTLE_CA_CHECKSUM
value: "${CACHECKSUM}"
- name: CATTLE_CLUSTER
value: "true"
- name: CATTLE_K8S_MANAGED
value: "true"
image: ${AGENTIMAGE}
volumeMounts:
- name: cattle-credentials
mountPath: /cattle-credentials
readOnly: true
volumes:
- name: cattle-credentials
secret:
secretName: cattle-credentials-${TOKENKEY}
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: cattle-node-agent
namespace: cattle-system
spec:
selector:
matchLabels:
app: cattle-agent
template:
metadata:
labels:
app: cattle-agent
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: NotIn
values:
- windows
hostNetwork: true
serviceAccountName: cattle
tolerations:
- effect: NoExecute
key: "node-role.kubernetes.io/etcd"
value: "true"
- effect: NoSchedule
key: "node-role.kubernetes.io/controlplane"
value: "true"
containers:
- name: agent
image: ${AGENTIMAGE}
imagePullPolicy: IfNotPresent
env:
- name: CATTLE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CATTLE_SERVER
value: "${SERVERURL}"
- name: CATTLE_CA_CHECKSUM
value: "${CACHECKSUM}"
- name: CATTLE_CLUSTER
value: "false"
- name: CATTLE_K8S_MANAGED
value: "true"
- name: CATTLE_AGENT_CONNECT
value: "true"
volumeMounts:
- name: cattle-credentials
mountPath: /cattle-credentials
readOnly: true
- name: k8s-ssl
mountPath: /etc/kubernetes
- name: var-run
mountPath: /var/run
- name: run
mountPath: /run
securityContext:
privileged: true
volumes:
- name: k8s-ssl
hostPath:
path: /etc/kubernetes
type: DirectoryOrCreate
- name: var-run
hostPath:
path: /var/run
type: DirectoryOrCreate
- name: run
hostPath:
path: /run
type: DirectoryOrCreate
- name: cattle-credentials
secret:
secretName: cattle-credentials-${TOKENKEY}
updateStrategy:
type: RollingUpdate
EOL
echo "You can now configure kubeconfig to your created/managed cluster and run kubectl apply -f rancher-agents-${CLUSTERNAME}.yml to restore cluster/node agents"
#!/bin/bash
# Usage: ./get_agents_yaml_single.sh cluster_name
# Needs to be run on the server running `rancher/rancher` container
# Check if jq exists
command -v jq >/dev/null 2>&1 || { echo "jq is not installed. Exiting." >&2; exit 1; }
command -v sha256sum >/dev/null 2>&1 || { echo "jq is not installed. Exiting." >&2; exit 1; }
command -v base64 >/dev/null 2>&1 || { echo "jq is not installed. Exiting." >&2; exit 1; }
command -v md5sum >/dev/null 2>&1 || { echo "jq is not installed. Exiting." >&2; exit 1; }
# Check if clustername is given
if [ -z "$1" ]; then
echo "Usage: $0 [clustername]"
exit 1
fi
# Provide clustername as first argument
CLUSTERNAME=$1
# Retrieve Docker container ID of container running `rancher/rancher` image
CONTID=$(docker ps | grep -E "rancher/rancher:|rancher/rancher@|rancher_rancher" | awk '{ print $1 }')
echo "Container ID running Rancher is ${CONTID}"
# Validate that we are querying the correct etcd
if docker exec $CONTID kubectl get cluster > /dev/null; then
echo "'kubectl get cluster' returns clusters available"
else
echo "'kubectl get cluster' returns error, this should be run on the host running the 'rancher/rancher' container or embedded kubectl of the 'local' imported cluster"
exit 1
fi
# Get clusters
CLUSTERINFO=$(docker exec $CONTID kubectl get cluster --no-headers --output=custom-columns=Name:.spec.displayName,Driver:.status.driver,ClusterID:.metadata.name)
echo "Clusters found:"
echo "${CLUSTERINFO}"
# Get clusterid from clustername
CLUSTERID=$(echo "${CLUSTERINFO}" | awk -v CLUSTERNAME=$CLUSTERNAME '$1==CLUSTERNAME { print $3 }')
if [[ -z $CLUSTERID ]]; then
echo "No CLUSTERID could be retrieved for $CLUSTERNAME, make sure you entered the correct clustername"
exit 1
fi
# Get all needed settings
CACERTS=$(docker exec $CONTID kubectl get setting cacerts -o json | jq -r .value)
if [ "x${CACERTS}" == "x" ]; then
CACHECKSUM=""
else
CACHECKSUM=$(echo "$CACERTS" | sha256sum | awk '{ print $1 }')
fi
SERVERURL=$(docker exec $CONTID kubectl get setting server-url -o json | jq -r .value)
AGENTIMAGE=$(docker exec $CONTID kubectl get settings agent-image -o json | jq -r .value)
TOKEN=$(docker exec $CONTID kubectl get clusterregistrationtoken system -n $CLUSTERID -o json | jq -r .status.token)
# Getting correct values
B64TOKEN=$(echo -n $TOKEN | base64)
B64URL=$(echo -n $SERVERURL | base64)
TOKENKEY=$(echo -n $TOKEN | md5sum | cut -c1-7)
#echo $B64TOKEN
#echo $B64URL
#echo $TOKENKEY
cat >./rancher-agents-$CLUSTERNAME.yml <<EOL
---
apiVersion: v1
kind: Namespace
metadata:
name: cattle-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cattle
namespace: cattle-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: cattle-admin-binding
namespace: cattle-system
labels:
cattle.io/creator: "norman"
subjects:
- kind: ServiceAccount
name: cattle
namespace: cattle-system
roleRef:
kind: ClusterRole
name: cattle-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
metadata:
name: cattle-credentials-${TOKENKEY}
namespace: cattle-system
type: Opaque
data:
url: "${B64URL}"
token: "${B64TOKEN}"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cattle-admin
labels:
cattle.io/creator: "norman"
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: cattle-cluster-agent
namespace: cattle-system
spec:
selector:
matchLabels:
app: cattle-cluster-agent
template:
metadata:
labels:
app: cattle-cluster-agent
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: NotIn
values:
- windows
serviceAccountName: cattle
containers:
- name: cluster-register
imagePullPolicy: IfNotPresent
env:
- name: CATTLE_SERVER
value: "${SERVERURL}"
- name: CATTLE_CA_CHECKSUM
value: "${CACHECKSUM}"
- name: CATTLE_CLUSTER
value: "true"
- name: CATTLE_K8S_MANAGED
value: "true"
image: ${AGENTIMAGE}
volumeMounts:
- name: cattle-credentials
mountPath: /cattle-credentials
readOnly: true
volumes:
- name: cattle-credentials
secret:
secretName: cattle-credentials-${TOKENKEY}
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: cattle-node-agent
namespace: cattle-system
spec:
selector:
matchLabels:
app: cattle-agent
template:
metadata:
labels:
app: cattle-agent
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: NotIn
values:
- windows
hostNetwork: true
serviceAccountName: cattle
tolerations:
- effect: NoExecute
key: "node-role.kubernetes.io/etcd"
value: "true"
- effect: NoSchedule
key: "node-role.kubernetes.io/controlplane"
value: "true"
containers:
- name: agent
image: ${AGENTIMAGE}
imagePullPolicy: IfNotPresent
env:
- name: CATTLE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CATTLE_SERVER
value: "${SERVERURL}"
- name: CATTLE_CA_CHECKSUM
value: "${CACHECKSUM}"
- name: CATTLE_CLUSTER
value: "false"
- name: CATTLE_K8S_MANAGED
value: "true"
- name: CATTLE_AGENT_CONNECT
value: "true"
volumeMounts:
- name: cattle-credentials
mountPath: /cattle-credentials
readOnly: true
- name: k8s-ssl
mountPath: /etc/kubernetes
- name: var-run
mountPath: /var/run
- name: run
mountPath: /run
securityContext:
privileged: true
volumes:
- name: k8s-ssl
hostPath:
path: /etc/kubernetes
type: DirectoryOrCreate
- name: var-run
hostPath:
path: /var/run
type: DirectoryOrCreate
- name: run
hostPath:
path: /run
type: DirectoryOrCreate
- name: cattle-credentials
secret:
secretName: cattle-credentials-${TOKENKEY}
updateStrategy:
type: RollingUpdate
EOL
echo "You can now configure kubeconfig to your created/managed cluster and run kubectl apply -f rancher-agents-${CLUSTERNAME}.yml to restore cluster/node agents"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.