Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Our project aims to build a Multi-platform kubernetes cluster, we use the VM on x86 as master node and two nvidia tx2 development kits as node. So we can schedule both CPU and GPU resources.

Kubernetes cluster setup based on amd64 && NVIDIA JETSON TX2

Thanks to @luxas create the Kubernetes on ARM project. But my project has some different, i have a VM on x86 as master node and two nvidia tx2 development kits as work node. So my kubernetes cluster are multi-platform. I use kubeadm as a deployment method.Some basic information of my platform is as follows.

Versions

kubeadm version (use kubeadm version):1.10.0

Environment:

  • Kubernetes version (use kubectl version):1.10.0
  • Cloud provider or hardware configuration:an amd64 master and 1 arm nodes (nvidia tagra TX2)
  • OS (e.g. from /etc/os-release): ubuntu 16.04
  • Kernel (e.g. uname -a): Linux ubuntu 4.13.0-39-generic # 44~16.04.1-Ubuntu SMP Thu Apr 5 16:43:10 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux for amd64 && Linux tegra-ubuntu 4.4.38 # 1 SMP PREEMPT Sun Apr 22 02:51:59 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux for arm
  • Others: iptables version: 1.6.0

Steps

Follow the instructions at https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ for setting up the master node. Before bring up the other nodes by running the kubeadm join ..., there are two important things we should do. first, we should customize the kube-proxy on the arm node. Without this step, the kube-proxy pod on the arm node will be crashloopback.This is caused by image mismatch between master node and the worker node.Kube-proxy is run as a daemonset. This means,the daemonsets will be schedued to run on all nodes.If you inspect the kube-proxy, kubectl describe kube-proxy -n kube-system you will notice the image tag referencing to amd64. Fortunately @squidpickles proposed a solution ,Multiplatform (amd64 and arm) Kubernetes cluster setup. Add nodeSelector to kube-proxy manifest. You can get the node labes by running

kubectl describe node <node-name>

By running

KUBE_EDITOR="nano" kubectl edit daemonset kube-proxy --namespace=kube-system

you can modify the manifest of kube-proxy.

       dnsPolicy: ClusterFirst
       hostNetwork: true
+      nodeSelector:
+        beta.kubernetes.io/arch: amd64
       restartPolicy: Always
       schedulerName: default-scheduler
       securityContext: {}

Make a new manifest for arm node, name kube-proxy-arm.

 apiVersion: extensions/v1beta1
 kind: DaemonSet
 metadata:
-  creationTimestamp: 2018-01-27T07:27:28Z
-  generation: 2
   labels:
     k8s-app: kube-proxy
-  name: kube-proxy
+  name: kube-proxy-arm
   namespace: kube-system
-  resourceVersion: "1662"
-  selfLink: /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/kube-proxy
-  uid: 8769e0b3-0333-11e8-8cb9-40a8f02df8cb
 spec:
   revisionHistoryLimit: 10
   selector:
       - command:
         - /usr/local/bin/kube-proxy
         - --config=/var/lib/kube-proxy/config.conf
-        image: gcr.io/google_containers/kube-proxy-amd64:v1.10.1
+        image: gcr.io/google_containers/kube-proxy-arm:v1.10.1
         imagePullPolicy: IfNotPresent
         name: kube-proxy
         resources: {}
       dnsPolicy: ClusterFirst
       hostNetwork: true
       nodeSelector:
-        beta.kubernetes.io/arch: amd64
+        beta.kubernetes.io/arch: arm64
       restartPolicy: Always
       schedulerName: default-scheduler
       securityContext: {}
     rollingUpdate:
       maxUnavailable: 1
     type: RollingUpdate
-status:
-  currentNumberScheduled: 1
-  desiredNumberScheduled: 1
-  numberAvailable: 1
-  numberMisscheduled: 0
-  numberReady: 1
-  observedGeneration: 2
-  updatedNumberScheduled: 1

you can create the new DaemonSet by running

kubectl create -f daemonset-arm.yaml

When I completed this, I found that the status of the kube-proxy pod on the arm node is running. But when adding a network plug-in(weave), it was found that the arm node has been in a not-ready state, and the weave pod is crashloopbackoff. The logs of weave:

Get https://10.96.0.1:443/api/v1/nodes: dial tcp 10.96.0.1:443: i/o timeout;
Failed to get peers

I think the problem is in kube-proxy-arm. So i check the logs:

kubectl logs kube-proxy-arm-bzrvg -n=kube-system
I0502 13:19:00.679884       1 feature_gate.go:226] feature gates: &{{} map[]}
W0502 13:19:00.699625       1 server_others.go:290] Can't use ipvs proxier, trying iptables proxier
I0502 13:19:00.702634       1 server_others.go:140] Using iptables Proxier.
I0502 13:19:00.740414       1 server_others.go:174] Tearing down inactive rules.
I0502 13:19:01.034561       1 server.go:444] Version: v1.10.1
I0502 13:19:01.099829       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0502 13:19:01.100324       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0502 13:19:01.101273       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0502 13:19:01.101430       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0502 13:19:01.102335       1 config.go:202] Starting service config controller
I0502 13:19:01.102946       1 controller_utils.go:1019] Waiting for caches to sync for service config controller
I0502 13:19:01.103395       1 config.go:102] Starting endpoints config controller
I0502 13:19:01.104678       1 controller_utils.go:1019] Waiting for caches to sync for endpoints config controller
I0502 13:19:01.206422       1 controller_utils.go:1026] Caches are synced for service config controller
I0502 13:19:01.206422       1 controller_utils.go:1026] Caches are synced for endpoints config controller
E0502 13:19:01.415172       1 proxier.go:1285] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 27 failed
)
E0502 13:19:31.288315       1 proxier.go:1285] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 27 failed
)
E0502 13:20:01.540382       1 proxier.go:1285] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 27 failed
)
E0502 13:20:31.758797       1 proxier.go:1285] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 27 failed

See more details in Failed to execute iptables-restore: exit status 1 in kube-proxy #784. It looks like kube-proxy encountered an error when generating iptables on the arm node. After some inspection,I re-compiled the kernel of NVIDIA TX2 and loaded the kernel modules needed by netfliter and weave. Such as nf_conntrakxt_set,vxlan,openvswitch. My workflow looks like:

$ git clone https://github.com/jetsonhacks/buildJetsonTX2Kernel.git
$ cd buildJetsonTX2Kernel
# For L4T 28.2, do the following:
$ git checkout vL4T28.2

There are three main scripts. The first script, getKernelSources.sh gets the kernel sources from the NVIDIA developer website, then unpacks the sources into /usr/src/kernel.

$ ./getKernelSources.sh

After the sources are installed, the script opens an editor on the kernel configuration file. PS: Need to install QT4. You can choose install the module you want. The second script, makeKernel.sh, fixes up the makefiles so that the source can be compiled on the Jetson, and then builds the kernel and modules specified.

$ ./makeKernel.sh

The modules are then installed in /lib/modules/ The third script, copyImage.sh, copies over the newly built Image and zImage files into the /boot directory.

$ ./copyImage.sh

Once the images have been copied over to the /boot directory, the machine must be restarted for the new kernel to take effect.

When this is done, the error disappears. Everything is fine.

master@ubuntu:~$ kubectl describe node tegra-ubuntu
Name:               tegra-ubuntu
Roles:              <none>
Labels:             beta.kubernetes.io/arch=arm64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=tegra-ubuntu
Annotations:        node.alpha.kubernetes.io/ttl=0
                    volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp:  Sat, 05 May 2018 11:56:45 +0800
Taints:             <none>
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Mon, 07 May 2018 20:51:32 +0800   Mon, 07 May 2018 20:40:48 +0800   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Mon, 07 May 2018 20:51:32 +0800   Mon, 07 May 2018 20:40:48 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Mon, 07 May 2018 20:51:32 +0800   Mon, 07 May 2018 20:40:48 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Mon, 07 May 2018 20:51:32 +0800   Sat, 05 May 2018 11:56:45 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Mon, 07 May 2018 20:51:32 +0800   Mon, 07 May 2018 20:40:48 +0800   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  10.108.71.185
  Hostname:    tegra-ubuntu
Capacity:
 cpu:                4
 ephemeral-storage:  28768380Ki
 memory:             8034624Ki
 pods:               110
Allocatable:
 cpu:                4
 ephemeral-storage:  26512938965
 memory:             7932224Ki
 pods:               110
System Info:
 Machine ID:                 ae15719763c84b35196c20a95728b806
 System UUID:                ae15719763c84b35196c20a95728b806
 Boot ID:                    b4dfd2df-dd05-45b9-8251-51e871c474a3
 Kernel Version:             4.4.384.4.38-test
 OS Image:                   Ubuntu 16.04 LTS
 Operating System:           linux
 Architecture:               arm64
 Container Runtime Version:  docker://1.13.1
 Kubelet Version:            v1.10.0
 Kube-Proxy Version:         v1.10.0
PodCIDR:                     10.32.4.0/24
ExternalID:                  tegra-ubuntu
Non-terminated Pods:         (2 in total)
  Namespace                  Name                    CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                    ------------  ----------  ---------------  -------------
  kube-system                kube-proxy-arm-2fpgd    0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                weave-net-jwrv5         20m (0%)      0 (0%)      0 (0%)           0 (0%)

master@ubuntu:~$ kubectl get pod --all-namespaces
NAMESPACE     NAME                             READY     STATUS    RESTARTS   AGE
kube-system   etcd-ubuntu                      1/1       Running   0          2d
kube-system   kube-apiserver-ubuntu            1/1       Running   0          2d
kube-system   kube-controller-manager-ubuntu   1/1       Running   0          2d
kube-system   kube-dns-86f4d74b45-pgwb8        3/3       Running   0          11d
kube-system   kube-proxy-arm-2fpgd             1/1       Running   5          2d
kube-system   kube-proxy-pjnwn                 1/1       Running   0          11d
kube-system   kube-proxy-wmd2p                 1/1       Running   0          10d
kube-system   kube-scheduler-ubuntu            1/1       Running   0          2d
kube-system   weave-net-jwrv5                  2/2       Running   3          1d
kube-system   weave-net-p282j                  2/2       Running   0          1d
kube-system   weave-net-vq8fz                  2/2       Running   0          1d

master@ubuntu:~$ kubectl get node
NAME           STATUS    ROLES     AGE       VERSION
nodex86        Ready     <none>    10d       v1.10.1
tegra-ubuntu   Ready     <none>    2d        v1.10.0
ubuntu         Ready     master    11d       v1.10.0
@yeliuang

This comment has been minimized.

Copy link

commented May 7, 2018

厉害!

@buptliuwei

This comment has been minimized.

Copy link
Owner Author

commented May 7, 2018

@yeliuang 不行,不行

@martwetzels

This comment has been minimized.

Copy link

commented Jun 9, 2018

Could you create a gist with your config file that you used to make a new kernel? With the added modules, Weave still throws the same error. The Kube-proxy-arm is throwing: 1 proxier.go:1285] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 46 failed).

@buptliuwei

This comment has been minimized.

Copy link
Owner Author

commented Jun 10, 2018

@martwetzels you can read thisBuild Kernel and Modules – NVIDIA Jetson TX2. I think the error is caused by the kernel.

@StupidYe

This comment has been minimized.

Copy link

commented Jul 3, 2018

有几点疑问:1.那个proxy的manifest在哪里的呀? 2.可以自己这样子重新生成一个proxy?3.不知道您是否遇到过,在arm64上使用kubeadm部署,flannel始终失败,log为:Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-fqw9w': Get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-fqw9w: dial tcp 10.96.0.1:443: getsockopt: no route to `host```
希望指教一二~

@buptliuwei

This comment has been minimized.

Copy link
Owner Author

commented Jul 4, 2018

@StupidYe 你好,gist没有邮件提醒,现在才看到,不好意思啊。首先proxy的manifest,我是这样做的,通过编辑器打开原来的kube-proxy文件,然后复制到自己新建的文本中,改成kube-proxy-arm,然后做一些修改,主要是nodeselector。关于flannel部署失败的问题,从报的error上看也是不能路由到service ip。很大关系和kube-proxy有关,我不知道你arm上装的系统是什么,ubuntu的话,可能是有些内核模块没打开,flannel也是overlay的解决方案,需要openvswitch等内核模块的支持,你可以lsmod下,看看模块是否都加载了。最后,kubeadm的文档里面建议在arm上部署网络插件的话,weave的兼容性最好。所以你也可以考虑下weave.

@nithya-krish

This comment has been minimized.

Copy link

commented Aug 17, 2018

Hi,
I faced the same crashloopbackoff state while I tried to use calico as my network plugin. Is it solvable like what you did for weavenet. Do I want to recompile the kernel? Also you mentioned that, I re-compiled the kernel of NVIDIA TX2 and loaded the kernel modules needed by netfliter and weave. Do you have any idea how we can do this for calico. If so can you please help me to solve the issue.

@buptliuwei

This comment has been minimized.

Copy link
Owner Author

commented Mar 17, 2019

Hi,
I faced the same crashloopbackoff state while I tried to use calico as my network plugin. Is it solvable like what you did for weavenet. Do I want to recompile the kernel? Also you mentioned that, I re-compiled the kernel of NVIDIA TX2 and loaded the kernel modules needed by netfliter and weave. Do you have any idea how we can do this for calico. If so can you please help me to solve the issue.

Firstly,i have to apologize to you. The Github gist has no email notifications. I missed your message in :https://gist.github.com/buptliuwei/8a340cc151507cb48a071cda04e1f882.
Maybe you have solved your problem long ago. According to this document:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
Calico can work on arm64,so if you have problem when calico pod crashloopbackoff ,i think most likely caused by the tx2 kernel, You can check the error log to determine which modules need to be compiled into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.