The steps below requires that you have followed the installation steps for installing K3s on RPIs.
Start registry container on master node, we will use insecure mode as configured in my K3s installation guide:
docker run -d -p 5000:5000 --restart=always --name registry registry:2
Login to docker on master node (requires that you have an account on Docker Hub):
docker login
Create a secret named regcred on master node that will be used later on in K3s pod or deployment specs (NOTE: If k3s-master-1 is not present on /etc/hosts then change to master node´s IP, also use your credentials on Docker Hub in the command below):
kubectl create secret docker-registry regcred --docker-server=k3s-master-1 --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
Inspect the created secret (NOTE: Command below requires alias being configured, if not exchange kto kubectl:
k get secret regcred --output=yaml
The sensitive information is base64 encoded, to decode use:
k get secret regcred --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode
You can also base64 decode the auth parameter:
echo "c3R...zE2" | base64 --decode
Example of use in a deployment spec, note the use of imagePull:Sectrets below:
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: regcred
K3s uses default K3d (containerd) as container runtime, you can configure K3s to use docker by passing --docker to the update command. See appendix below if you want to use docker!
Configure K3d (containerd) to be able to pull images from local repository. Create a file called /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl on all nodes with the contents below. NOTE: The file uses k3s-master-1 as hostname, if you haven't configured hostnames in /etc/hosts you must use the IP address of the master node:
sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
# Original section: no changes
[plugins.opt]
path = "{{ .NodeConfig.Containerd.Opt }}"
[plugins.cri]
stream_server_address = "{{ .NodeConfig.AgentConfig.NodeName }}"
stream_server_port = "10010"
{{- if .IsRunningInUserNS }}
disable_cgroup = true
disable_apparmor = true
restrict_oom_score_adj = true
{{ end -}}
{{- if .NodeConfig.AgentConfig.PauseImage }}
sandbox_image = "{{ .NodeConfig.AgentConfig.PauseImage }}"
{{ end -}}
{{- if not .NodeConfig.NoFlannel }}
[plugins.cri.cni]
bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"
{{ end -}}
# Added section: additional registries and the endpoints
[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."k3s-master-1:5000"]
endpoint = ["http://k3s-master-1:5000"]
[plugins.cri.containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
Restart k3s on all nodes.
sudo systemctl restart k3s-agent
Test that runc v2 is actually used by containerd:
sudo crictl info |grep runc
"defaultRuntimeName": "runc",
"runc": {
"runtimeType": "io.containerd.runc.v2",
To push a container image to your local registry you will need an image :-).
Lets say you have built a Docker image named openhab with tag 2.5.0.M3, then you need to tag it with the local registry name and repo name and tag, in the example below k3s-master-1:5000/openhab:latest Note: This requires k3s-master-1to be present in /etc/hosts, use master node's IP instead:
docker tag openhab:2.5.0.M3 k3s-master-1:5000/openhab:latest
Then it is time to push it:
docker push k3s-master-1:5000/openhab:latest
You can try to pull the image from a worker node using cri which containerd uses:
sudo crictl --debug pull k3s-master-1:5000/openhab:latest
DEBU[2019-09-28T20:38:49.027935548+01:00] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:k3s-master-1:5000/openhab:latest,},Auth:nil,SandboxConfig:nil,}
DEBU[2019-09-28T20:38:49.206211341+01:00] PullImageResponse: &PullImageResponse{ImageRef:sha256:cbc9490d8062bb74776d05e9e9e78ec16ba36a0f668b6bd175727e1a682bb022,}
Image is up to date for sha256:cbc9490d8062bb74776d05e9e9e78ec16ba36a0f668b6bd175727e1a682bb022
In this case the image was already pulled.
If this works, then it will work when deploying pods using container images present in your local docker registry.
To debug it is possible to get logs from the registry using the docker logscommand:
docker ps|grep registry
01c29fcf39dc registry:2 "/entrypoint.sh /etc…" 22 hours ago Up 4 hours 0.0.0.0:5000->5000/tcp registry
docker logs 01c29fcf39dc
192.168.1.161 - - [28/Sep/2019:12:39:25 +0000] "GET /v2/openhab/blobs/sha256:2410c39b407e3f574d3a009892ef3a78ab9d7d516eefe1afec631afc2b8e2242 HTTP/1.1" 200 76332610 "" "containerd/v1.2.8-k3s.1"
time="2019-09-28T12:40:02.296101103Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="k3s-master-1:5000" http.request.id=eaf26473-126c-489a-aa3a-05b6bd5a5d8a http.request.method=GET http.request.remoteaddr="192.168.1.161:44614" http.request.uri="/v2/openhab/blobs/sha256:52daf50280e3702d60fe6f1ecada50ecd5c00c32885330415142f6cd235528f3" http.request.useragent="containerd/v1.2.8-k3s.1" http.response.contenttype="application/octet-stream" http.response.duration=35.862984057s http.response.status=200 http.response.written=125131517
192.168.1.161 - - [28/Sep/2019:12:39:26 +0000] "GET /v2/openhab/blobs/sha256:52daf50280e3702d60fe6f1ecada50ecd5c00c32885330415142f6cd235528f3 HTTP/1.1" 200 125131517 "" "containerd/v1.2.8-k3s.1"
time="2019-09-28T15:49:31.912848532Z" level=info msg="Starting upload purge in 32m0s" go.version=go1.11.2 instance.id=78d841d3-5b19-4476-a6a8-55f005d09721 service=registry version=v2.7.1
time="2019-09-28T15:49:31.912619157Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.11.2 instance.id=78d841d3-5b19-4476-a6a8-55f005d09721 service=registry version=v2.7.1
time="2019-09-28T15:49:31.91311723Z" level=info msg="redis not configured" go.version=go1.11.2 instance.id=78d841d3-5b19-4476-a6a8-55f005d09721 service=registry version=v2.7.1
time="2019-09-28T15:49:31.91832973Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.11.2 instance.id=78d841d3-5b19-4476-a6a8-55f005d09721 service=registry version=v2.7.1
time="2019-09-28T15:49:31.919082959Z" level=info msg="listening on [::]:5000" go.version=go1.11.2 instance.id=78d841d3-5b19-4476-a6a8-55f005d09721 service=registry version=v2.7.1
time="2019-09-28T16:21:41.179893674Z" level=info msg="PurgeUploads starting: olderThan=2019-09-21 16:21:41.158772809 +0000 UTC m=-602879.146802630, actuallyDelete=true"
time="2019-09-28T16:21:41.194807998Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"
time="2019-09-28T16:21:41.194975445Z" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.11.2 instance.id=78d841d3-5b19-4476-a6a8-55f005d09721 service=registry version=v2.7.1
time="2019-09-28T19:38:49.074777812Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="k3s-master-1:5000" http.request.id=d6c03b8a-1811-4e61-abd0-8106c3f5101f http.request.method=HEAD http.request.remoteaddr="192.168.1.161:46880" http.request.uri="/v2/openhab/manifests/latest" http.request.useragent="containerd/v1.2.8-k3s.1" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=38.553762ms http.response.status=200 http.response.written=2210
192.168.1.161 - - [28/Sep/2019:19:38:49 +0000] "HEAD /v2/openhab/manifests/latest HTTP/1.1" 200 2210 "" "containerd/v1.2.8-k3s.1"
time="2019-09-28T19:38:49.088313855Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="k3s-master-1:5000" http.request.id=7d44fe0c-6951-4b6c-9a92-fd71dd5a382d http.request.method=HEAD http.request.remoteaddr="192.168.1.161:46880" http.request.uri="/v2/openhab/manifests/latest" http.request.useragent="containerd/v1.2.8-k3s.1" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=7.51643ms http.response.status=200 http.response.written=2210
192.168.1.161 - - [28/Sep/2019:19:38:49 +0000] "HEAD /v2/openhab/manifests/latest HTTP/1.1" 200 2210 "" "containerd/v1.2.8-k3s.1"
NOTE: It is not necessary and not recommended to configure K3s to use docker, later steps will show how to get image pull to work from K3D. If you want to use docker then follow steps below:
# If you want K3s to user docker, add --docker to update command
# On master node
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 --docker
# On worker nodes
curl -sfL https://get.k3s.io | sh -s - --docker
It is also possible to configure --docker directly in the file etc/systemd/system/k3s-agent.service;
pi@k3s-worker-3:~ $ cat /etc/systemd/system/k3s-agent.service
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network-online.target
[Service]
Type=exec
EnvironmentFile=/etc/systemd/system/k3s-agent.service.env
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
agent \
'--docker' \
KillMode=process
Delegate=yes
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target