Skip to content

Instantly share code, notes, and snippets.

@superseb
Last active February 26, 2024 19:47
  • Star 80 You must be signed in to star a gist
  • Fork 12 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save superseb/0c06164eef5a097c66e810fe91a9d408 to your computer and use it in GitHub Desktop.
k3s etcd commands

k3s etcd commands

etcd

Setup etcdctl using the instructions at https://github.com/etcd-io/etcd/releases/tag/v3.4.13 (changed path to /usr/local/bin):

Note: if you want to match th etcdctl binaries with the embedded k3s etcd version, please run the curl command for getting the version first and adjust ETCD_VER below accordingly:

curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/version
ETCD_VER=v3.4.13

# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}

rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local/bin --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

etcd --version
etcdctl version
  • etcdctl check perf
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl check perf
  • etcdctl endpoint status
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table
  • etcdctl endpoint health
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint health --cluster --write-out=table
  • etcdctl alarm list
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl alarm list
  • etcdctl compact
rev=$(ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --write-out fields | grep Revision | cut -d: -f2)
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl compact $rev
  • etcdctl defrag
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl defrag --cluster
  • etcdctl get
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl get / --prefix --keys-only
  • curl metrics

NOTE Since the following k3s versions, the HTTP port moved to 2382 (the example below uses port 2379):

  • v1.25.15+k3s1
  • v1.26.10+k3s1
  • v1.27.7+k3s1
  • v1.28.3+k3s1
  • v1.29.0+k3s1
curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/metrics
  • curl version

NOTE Since the following k3s versions, the HTTP port moved to 2382 (the example below uses port 2379):

  • v1.25.15+k3s1
  • v1.26.10+k3s1
  • v1.27.7+k3s1
  • v1.28.3+k3s1
  • v1.29.0+k3s1
curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/version
  • export all environment variables (thanks to @clementnuss)
export ETCDCTL_ENDPOINTS='https://127.0.0.1:2379'
export ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt'
export ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt'
export ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key'
export ETCDCTL_API=3
@ambis
Copy link

ambis commented Jan 31, 2021

Thank you! <3

@ntx-ben
Copy link

ntx-ben commented Feb 2, 2021

This is gold! Thanks

@garygan89
Copy link

This is indeed gold! Thanks!

@drscat
Copy link

drscat commented Apr 2, 2021

Thanks!

@samcday
Copy link

samcday commented Aug 8, 2021

Thanks for providing this. I raised k3s-io/k3s#3796 in the hopes that connecting to a k3s embedded etcd might be easier in future.

@shuxue051
Copy link

Thanks for providing this.

@clementnuss
Copy link

thanks ! just posting an export version as well

export ETCDCTL_ENDPOINTS='https://127.0.0.1:2379'
export ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt'
export ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt'
export ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key'
export ETCDCTL_API=3

@mritd
Copy link

mritd commented Jan 26, 2022

Thanks!

@AlexanderBabel
Copy link

Hey there!

I created a customized install script for etcdctl:

#!/bin/sh

ETCD_VER=v3.5.4
DOWNLOAD_URL=https://github.com/etcd-io/etcd/releases/download

case "$(uname -m)" in
    aarch64) ETCD_ARCH="arm64" ;;
    x86_64) ETCD_ARCH="amd64" ;;
esac;

ETCD_NAME=etcd-${ETCD_VER}-linux-${ETCD_ARCH}
ETCD_TAR=${ETCD_NAME}.tar.gz

rm -f /tmp/${ETCD_TAR}

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/${ETCD_TAR} -o /tmp/${ETCD_TAR}
tar xzvf /tmp/${ETCD_TAR} -C /usr/local/bin --strip-components=1 ${ETCD_NAME}/etcdctl
rm -f /tmp/${ETCD_TAR}

etcdctl version

It improves the original script in the following ways:

  • Support for arm64
  • Usage of variables to make modifications easier
  • install only etcdcli
  • It does not extract markdown or other files to /usr/local/bin
  • It does not create /tmp/etcd-download-test

@onedr0p
Copy link

onedr0p commented Aug 5, 2022

Thanks @AlexanderBabel

I took your script and made some adjustments.

#!/usr/bin/env bash

etcd_version=v3.5.3

case "$(uname -m)" in
    aarch64) arch="arm64" ;;
    x86_64) arch="amd64" ;;
esac;

etcd_name="etcd-${etcd_version}-linux-${arch}"

curl -sSfL "https://github.com/etcd-io/etcd/releases/download/${etcd_version}/${etcd_name}.tar.gz" \
    | tar xzvf - -C /usr/local/bin --strip-components=1 "${etcd_name}/etcdctl"

etcdctl version

It improves the above script in the following ways:

  • Less code
  • No tmp file
  • Pipe curl to tar

@631068264
Copy link

curl -kL --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/version
curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)

@superseb
Copy link
Author

@631068264 is this is on default settings? What does curl --version and head -1 /var/lib/rancher/k3s/server/tls/etcd/server-client.key output? (make sure you only share the first line, not everything in that file)

@631068264
Copy link

k3s version: v1.22.10+k3s1 @superseb

curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.36 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets 
head -1 /var/lib/rancher/k3s/server/tls/etcd/server-client.key
-----BEGIN EC PRIVATE KEY-----

@VivekSheshadari
Copy link

VivekSheshadari commented Feb 24, 2024

@superseb k3s don't have curl , apk or apt-get , can you guide on how to run etcd client.

@schlichtanders
Copy link

@superseb k3s don't have curl , apk or apt-get , can you guide on how to run etcd client.

I am also running into this problem. while wget is available, this also does not work because it is compiled without --ca-cert support...

I ideally would like to extract the etcd version from this docker run:

docker run -it --rm --privileged --name k3s-server-1 --hostname k3s-server-1 rancher/k3s:v1.29.1-k3s1 server

@schlichtanders
Copy link

I was not able to get it running in any way...

But searching for the matching etcd version showed that actually every k3s release on github mentions versions of subcomponents explicitly

For example the most recent k3s https://github.com/k3s-io/k3s/releases/tag/v1.29.1%2Bk3s2 lists etcd v3.5.9-k3s1

@superseb
Copy link
Author

I assume you want to retrieve it from the binary before running it? Or is that not a hard requirement?

It's a bit hacky but here are some ways.

From a running container/instance, the logs will contain the version:

{"level":"info","ts":"2024-02-26T14:50:47.999296Z","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.9","git-sha"...

You can trigger this by ad-hoc by letting the startup crash:

docker run -it --rm --privileged --name dummy --hostname k3s-server-1 --read-only rancher/k3s:v1.29.1-k3s1 server --cluster-init 2>&1 | grep "starting an etcd server"

If you have a running instance, you can query etcd using curl by using a different container:

docker run --volumes-from=k3s-server-1 --network container:k3s-server-1 -u 0  curlimages/curl -sL --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://localhost:2379/version

In newer versions (I will update the commands in the gist as well with the version numbers), the HTTP port moved to 2382:

docker run --volumes-from=k3s-server-1 --network container:k3s-server-1 -u 0  curlimages/curl -sL --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://localhost:2382/version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment