k3s etcd commands

k3s-etcd-commands.md

k3s etcd commands

etcd

Setup etcdctl using the instructions at https://github.com/etcd-io/etcd/releases/tag/v3.4.13 (changed path to /usr/local/bin):

Note: if you want to match th etcdctl binaries with the embedded k3s etcd version, please run the curl command for getting the version first and adjust ETCD_VER below accordingly:

curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/version

ETCD_VER=v3.4.13

# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}

rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local/bin --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

etcd --version
etcdctl version

• etcdctl check perf

ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl check perf

• etcdctl endpoint status

ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table

• etcdctl endpoint health

ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint health --cluster --write-out=table

• etcdctl alarm list

ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl alarm list

• etcdctl compact

rev=$(ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --write-out fields | grep Revision | cut -d: -f2)
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl compact $rev

• etcdctl defrag

ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl defrag --cluster

• etcdctl get

ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl get / --prefix --keys-only

• curl metrics

NOTE Since the following k3s versions, the HTTP port moved to 2382 (the example below uses port 2379):

• v1.25.15+k3s1
• v1.26.10+k3s1
• v1.27.7+k3s1
• v1.28.3+k3s1
• v1.29.0+k3s1

curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/metrics

• curl version

NOTE Since the following k3s versions, the HTTP port moved to 2382 (the example below uses port 2379):

• v1.25.15+k3s1
• v1.26.10+k3s1
• v1.27.7+k3s1
• v1.28.3+k3s1
• v1.29.0+k3s1

curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/version

• export all environment variables (thanks to @clementnuss)

export ETCDCTL_ENDPOINTS='https://127.0.0.1:2379'
export ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt'
export ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt'
export ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key'
export ETCDCTL_API=3

curl -kL --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/version
curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)

@631068264 is this is on default settings? What does curl --version and head -1 /var/lib/rancher/k3s/server/tls/etcd/server-client.key output? (make sure you only share the first line, not everything in that file)

k3s version: v1.22.10+k3s1 @superseb

curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.36 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets 

head -1 /var/lib/rancher/k3s/server/tls/etcd/server-client.key
-----BEGIN EC PRIVATE KEY-----

@631068264 I think you are running into https://access.redhat.com/solutions/3390021 / https://superuser.com/questions/1482345/how-to-use-an-ec-private-key-with-curllibnss

@superseb k3s don't have curl , apk or apt-get , can you guide on how to run etcd client.

@superseb k3s don't have curl , apk or apt-get , can you guide on how to run etcd client.

I am also running into this problem. while wget is available, this also does not work because it is compiled without --ca-cert support...

I ideally would like to extract the etcd version from this docker run:

docker run -it --rm --privileged --name k3s-server-1 --hostname k3s-server-1 rancher/k3s:v1.29.1-k3s1 server

I was not able to get it running in any way...

But searching for the matching etcd version showed that actually every k3s release on github mentions versions of subcomponents explicitly

For example the most recent k3s https://github.com/k3s-io/k3s/releases/tag/v1.29.1%2Bk3s2 lists etcd v3.5.9-k3s1

I assume you want to retrieve it from the binary before running it? Or is that not a hard requirement?

It's a bit hacky but here are some ways.

From a running container/instance, the logs will contain the version:

{"level":"info","ts":"2024-02-26T14:50:47.999296Z","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.9","git-sha"...

You can trigger this by ad-hoc by letting the startup crash:

docker run -it --rm --privileged --name dummy --hostname k3s-server-1 --read-only rancher/k3s:v1.29.1-k3s1 server --cluster-init 2>&1 | grep "starting an etcd server"

If you have a running instance, you can query etcd using curl by using a different container:

docker run --volumes-from=k3s-server-1 --network container:k3s-server-1 -u 0  curlimages/curl -sL --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://localhost:2379/version

In newer versions (I will update the commands in the gist as well with the version numbers), the HTTP port moved to 2382:

docker run --volumes-from=k3s-server-1 --network container:k3s-server-1 -u 0  curlimages/curl -sL --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://localhost:2382/version

Thanks, this helped a lot.

Thanks!

When I run the initial curl command, I get no response, and when I tack on a -I flag, I get:

$ sudo curl -LI --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
    --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
    --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key \
    https://127.0.0.1:2379/version
HTTP/2 415
content-type: application/grpc
grpc-status: 3
grpc-message: invalid gRPC request content-type ""

If I just use the latest binaries, I get:

# NOTE: Same result whether I use port 2382 or 2379 or anything in between
$ ETCDCTL_API=3 ETCDCTL_ENDPOINTS='https://127.0.0.1:2382' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' sudo ./etcdctl member list
{"level":"warn","ts":"2024-07-23T21:06:25.723722-0500","logger":"etcd-client","caller":"v3@v3.5.15/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x400001a000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"error reading server preface: EOF\""}
Error: context deadline exceeded

@weberc2 Can you share what OS you are using, how you installed k3s and what k3s version you are using? I tried with stable and latest from yesterday and had no issues.

@weberc2 I had the same problem, and the answer is at the bottom of the gist where it talks about later k3s versions moving the http port - use port 2382 and not 2379:

sudo curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
  --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
  --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key \
  https://127.0.0.1:2382/version