CoreOS ETCD Cluster RUN book

ETCD Run Book - etcdctl

There are two basic ways of interacting with etcd. Through the HTTP/JSON API and through a client, like the included etcdctl utility. We will go over etcdctl first.

1. Viewing Keys and Directories:
--------------------------------
To get started, let's look a what etcdctl is currently storing. We can see the top-level keys by typing:

core@core-01 ~ $ etcdctl ls /
/message
/test -d value=tested
/coreos.com
core@core-01 ~ $ 

At this point, it is unclear whether this is output directory or a key. We can attempt to get the node to see either the key's value or to see that it is a directory:

core@core-01 ~ $ etcdctl get /coreos.com
/coreos.com: is a directory       #It failed because it's a directory not a key vaalue pair
core@core-01 ~ $ etcdctl get /message   #It's a key so we got value of it.
Hello
core@core-01 ~ $ 

In order to avoid this manual recursive process, we can tell etcdctl to list its entire hierarchy of visible information by typing:

core@core-01 ~ $ etcdctl ls / --recursive
/coreos.com
/coreos.com/updateengine
/coreos.com/updateengine/rebootlock
/coreos.com/updateengine/rebootlock/semaphore
/coreos.com/network
/coreos.com/network/config
/coreos.com/network/subnets
/coreos.com/network/subnets/10.1.57.0-24
/coreos.com/network/subnets/10.1.64.0-24
/test -d value=tested
/test -d value=tested/00000000000000059444
/message
core@core-01 ~ $ 

There can be directories and keys within a given directory. Distinct both types using the -p flag.

core@core-01 ~ $ etcdctl ls -p  
/coreos.com/
/test -d value=tested/
/example/
/bar/
/message
/here/
/x/
core@core-01 ~ $ 

Distinguishable by the trailing slash, everything without one is a key. Here, /foo is just a key and everything else a directory.


As you can see, there were quite a few directories under the initial /coreos.com node. We can see what it looks like to get actual data out of a node by asking for the information at the final endpoint:

core@core-01 ~ $ etcdctl get /coreos.com/network/subnets/10.1.64.0-24
{"PublicIP":"172.17.8.103"}
core@core-01 ~ $ 

We can get some additional metadata about this entry by passing in the -o extended option. This is a global option, so it must come before the get command:

core@core-01 ~ $ etcdctl -o extended get /coreos.com/network/subnets/10.1.64.0-24
Key: /coreos.com/network/subnets/10.1.64.0-24
Created-Index: 1903971
Modified-Index: 1903971
TTL: 46977
Index: 1928975

{"PublicIP":"172.17.8.103"}


2. Setting Keys and Creating Nodes:-
-------------------------------------

To create a new directory, you can use the mkdir command like so:

core@core-01 ~ $ etcdctl mkdir /example
core@core-01 ~ $ 

To make a key, you can use the mk command:

core@core-01 ~ $ etcdctl mk /example/key data
data
core@core-01 ~ $ 

This will only work if the key does not already exist. If we ask for the value of the key we created, we can retrieve the data we set:

core@core-01 ~ $ etcdctl get /example/key
data
core@core-01 ~ $ 
If you ry to create it again then it will fail as shown below.
core@core-01 ~ $ etcdctl mk /example/key data
Error:  105: Key already exists (/example/key) [1929358]
core@core-01 ~ $ 

To update an existing key, use the update command:

core@core-01 ~ $ etcdctl update /example/key turtles
turtles
core@core-01 ~ $ 

There is  updatedir command for directories and it's is  only useful if you have set a TTL, or time-to-live on a directory. 
This will update the TTL time with the one passed. You can set TTLs for directories or keys by passing the --ttl # argument, where "#" is the number of seconds to keep:

core@core-01 ~ $ etcdctl mkdir /here/you/go --ttl 120
You can then update the TTL with updatedir:

core@core-01 ~ $ etcdctl updatedir /here/you/go --ttl 500

To change the value of an existing key, or to create a key if it does not exist, use the set command. 
Think of this as a combination of the mk and update command:

core@core-01 ~ $ etcdctl set /example/key new
new
core@core-01 ~ $ 
This can include non-existent paths. The path components will be created dynamically:

core@core-01 ~ $ etcdctl set /a/b/c here
here
core@core-01 ~ $ 

Removing Entries

To remove existing keys, you can use the rm or rmdir command.

The rm command can be used to remove a key:
core@core-01 ~ $ etcdctl rm /a/b/c
PrevNode.Value: here

It can also be used recursively to remove a directory and every subdirectory:

core@core-01 ~ $ etcdctl rm /a --recursive
core@core-01 ~ $ 

To remove only an empty directory or a key, use the rmdir command:
core@core-01 ~ $ etcdctl rmdir /x/y/z
Note:- This can be used to make sure you are only removing the endpoints of the hierarchies.

3.Watching for Changes:
-------------------------
You can watch either a specific key or an entire directory for changes. 
Watching these with etcdctl will cause the operation to hang until some event happens to whatever is being watched.

To watch a key, use it without any flags:

core@core-01 ~ $ etcdctl watch /example/hello
^Ccore@core-01 ~ $ 
To stop watching, you can press CTRL-C. If a change is detected during the watch, the new value will be returned.

To watch an entire directory structure, use the --recursive flag:

core@core-01 ~ $ etcdctl watch --recursive /example
^Ccore@core-01 ~ $ 

If you would like to execute a command whenever a change is detected, use the exec-watch command:
etcdctl exec-watch --recursive  /example -- echo "hello"
This will echo "hello" to the screen whenever a value in that directory changes.
I executed tow commands and We gor hello two times on screen. Commands were.
  core@core-01 ~ $ etcdctl set /example/b/c/ hello
  hello
  core@core-01 ~ $ etcdctl set /example/b/c/ test 
  test
We got hello two times on screen.

core@core-01 ~ $ 
core@core-01 ~ $ etcdctl exec-watch --recursive  /example -- echo "hello"
hello
hello


4.Hidden information:
---------------------
core@core-01 ~ $ etcdctl ls --recursive /_coreos.com 
/_coreos.com/fleet
/_coreos.com/fleet/engine
/_coreos.com/fleet/engine/version
/_coreos.com/fleet/lease
/_coreos.com/fleet/lease/engine-leader
/_coreos.com/fleet/machines
/_coreos.com/fleet/machines/219d42232433483c8ad19163ba1c6020
/_coreos.com/fleet/machines/219d42232433483c8ad19163ba1c6020/object
/_coreos.com/fleet/machines/5b74677278ea4b6ca5dcc43262d2b0e5
/_coreos.com/fleet/machines/5b74677278ea4b6ca5dcc43262d2b0e5/object
/_coreos.com/fleet/machines/edbfd19500b0496485e286801bdfa04b
/_coreos.com/fleet/machines/edbfd19500b0496485e286801bdfa04b/object
core@core-01 ~ $ 

ETCD Run Book - REST API

Other way of intracting with ETCD is HTTP/JSON API.

Etcd HTTP/JSON API Usage
The other way to interacting with etcd is with the simple HTTP/JSON API.

To access the API, you can use a simple HTTP program like curl. You must supply the -L flag to follow any redirects that are passed back. From within your cluster, you can use the local 127.0.0.1 interface and port 4001/2379 for most queries.


The normal keyspace can be reached by going to http://127.0.0.1:4001/v2/keys/ on any of the host machines. For instance, to get a listing of the top-level keys/directories, type:

core@core-01 ~ $ curl -L http://127.0.0.1:4001/v2/keys/
{"action":"get","node":{"dir":true,"nodes":[{"key":"/test -d value=tested","dir":true,"modifiedIndex":59444,"createdIndex":59444},{"key":"/example","dir":true,"modifiedIndex":1929174,"createdIndex":1929174},{"key":"/here","dir":true,"modifiedIndex":1929576,"createdIndex":1929576},{"key":"/x","dir":true,"modifiedIndex":1929885,"createdIndex":1929885},{"key":"/coreos.com","dir":true,"modifiedIndex":5,"createdIndex":5},{"key":"/message","value":"Hello","modifiedIndex":1879285,"createdIndex":1879285}]}}
core@core-01 ~ $ 


The trailing slash in the request is mandatory. It will not resolve correctly without it.

You can set or retrieve values using normal HTTP verbs.

To modify the behavior of these operations, you can pass in flags at the end of your request using the ?flag=value syntax. Multiple flags can be separated by a & character.

For instance, to recursively list all of the keys, we could type:

core@core-01 ~ $ curl -L http://127.0.0.1:4001/v2/keys/?recursive=true
{"action":"get","node":{"dir":true,"nodes":[{"key":"/test -d value=tested","dir":true,"nodes":[{"key":"/test -d value=tested/00000000000000059444","value":"","modifiedIndex":59444,"createdIndex":59444}],"modifiedIndex":59444,"createdIndex":59444},{"key":"/example","dir":true,"nodes":[{"key":"/example/key","value":"new","modifiedIndex":1929829,"createdIndex":1929829},{"key":"/example/b","dir":true,"nodes":[{"key":"/example/b/c","value":"test","modifiedIndex":1930464,"createdIndex":1930464}],"modifiedIndex":1930454,"createdIndex":1930454}],"modifiedIndex":1929174,"createdIndex":1929174},{"key":"/here","dir":true,"nodes":[{"key":"/here/you","dir":true,"modifiedIndex":1929576,"createdIndex":1929576}],"modifiedIndex":1929576,"createdIndex":1929576},{"key":"/x","dir":true,"nodes":[{"key":"/x/y","dir":true,"modifiedIndex":1929885,"createdIndex":1929885}],"modifiedIndex":1929885,"createdIndex":1929885},{"key":"/coreos.com","dir":true,"nodes":[{"key":"/coreos.com/updateengine","dir":true,"nodes":[{"key":"/coreos.com/updateengine/rebootlock","dir":true,"nodes":[{"key":"/coreos.com/updateengine/rebootlock/semaphore","value":"{\"semaphore\":1,\"max\":1,\"holders\":[]}","modifiedIndex":1503516,"createdIndex":5}],"modifiedIndex":5,"createdIndex":5}],"modifiedIndex":5,"createdIndex":5},{"key":"/coreos.com/network","dir":true,"nodes":[{"key":"/coreos.com/network/config","value":"{ \"Network\": \"10.1.0.0/16\" }","modifiedIndex":1863903,"createdIndex":1863903},{"key":"/coreos.com/network/subnets","dir":true,"nodes":[{"key":"/coreos.com/network/subnets/10.1.57.0-24","value":"{\"PublicIP\":\"172.17.8.102\"}","expiration":"2017-07-15T01:21:24.611307194Z","ttl":63928,"modifiedIndex":1917313,"createdIndex":1917313},{"key":"/coreos.com/network/subnets/10.1.64.0-24","value":"{\"PublicIP\":\"172.17.8.103\"}","expiration":"2017-07-14T19:30:19.101675433Z","ttl":42862,"modifiedIndex":1903971,"createdIndex":1903971}],"modifiedIndex":32,"createdIndex":32}],"modifiedIndex":14,"createdIndex":14}],"modifiedIndex":5,"createdIndex":5},{"key":"/message","value":"Hello","modifiedIndex":1879285,"createdIndex":1879285}]}}
core@core-01 ~ $ 

Another useful piece of information that is accessible outside of the normal keyspace is version info, accessible here:

core@core-01 ~ $ curl -L http://127.0.0.1:4001/version
{"etcdserver":"2.3.7","etcdcluster":"2.3.0"}core@core-01 ~ $ 
core@core-01 ~ $ 

You can view stats about each of the cluster leader's relationship with each follower by visiting this endpoint:

core@core-01 ~ $ curl -L http://172.17.8.102:4001/v2/stats/leader
{"leader":"348dd9a63bc9c9d3","followers":{"7d26e3d2ee11a98e":{"latency":{"current":0.001677,"average":0.018121494622911873,"standardDeviation":0.15678433061232888,"minimum":3.2e-05,"maximum":13.506931},"counts":{"fail":0,"success":723161}},"95d2e7af71fc961d":{"latency":{"current":0.001256,"average":0.019510112290459205,"standardDeviation":0.3501678220133334,"minimum":3.2e-05,"maximum":87.37375},"counts":{"fail":6232,"success":76596}}}}core@core-01 ~ $ 
core@core-01 ~ $ 


A similar operation can be used to detect stats about the machine you are currently on:

core@core-01 ~ $ curl -L http://127.0.0.1:4001/v2/stats/self
{"name":"5b74677278ea4b6ca5dcc43262d2b0e5","id":"95d2e7af71fc961d","state":"StateFollower","startTime":"2017-07-07T01:42:15.705504408Z","leaderInfo":{"leader":"348dd9a63bc9c9d3","uptime":"29h14m31.452569558s","startTime":"2017-07-13T02:32:02.488757881Z"},"recvAppendRequestCnt":112257,"recvPkgRate":6.451458356459762,"recvBandwidthRate":586.179506267934,"sendAppendRequestCnt":6464807}core@core-01 ~ $ 
core@core-01 ~ $ 

To see stats about operations that have been preformed, type:

ore@core-01 ~ $ curl -L http://127.0.0.1:4001/v2/stats/store
{"getsSuccess":2759657,"getsFail":8145754,"setsSuccess":259,"setsFail":1,"deleteSuccess":56,"deleteFail":11,"updateSuccess":547962,"updateFail":25,"createSuccess":48,"createFail":31,"compareAndSwapSuccess":1385650,"compareAndSwapFail":2,"compareAndDeleteSuccess":5,"compareAndDeleteFail":0,"expireCount":37,"watchers":4}core@core-01 ~ $ 
core@core-01 ~ $ 

ETCD Run Book- Examples

core@core-01 ~ $ 
core@core-01 ~ $ cat /run/systemd/system/etcd2.service.d/20-cloudinit.conf  | grep ETCD_DISCOVERY
Environment="ETCD_DISCOVERY=https://discovery.etcd.io/6fa563e0095f92a0db43bc75ccd4e60b"
core@core-01 ~ $ 
core@core-01 ~ $ 
core@core-01 ~ $ 
core@core-01 ~ $ etcdctl ls /_etcd/machines --recursive
Error:  100: Key not found (/_etcd) [1934345]
core@core-01 ~ $ 
core@core-01 ~ $ 
core@core-01 ~ $ 
core@core-01 ~ $ etcdctl set /foo bar  
bar
core@core-01 ~ $ etcdctl set /bar/foo "This is CoreOS"  
This is CoreOS
core@core-01 ~ $ etcdctl update /bar/foo "This is updated CoreOS"
This is updated CoreOS
core@core-01 ~ $ etcdctl mkdir /fooDir 
core@core-01 ~ $ etcdctl updatedir /fooDir --ttl 10 
core@core-01 ~ $ etcdctl set /fooDir/foo bar  
bar
core@core-01 ~ $ etcdctl mk /fooDir/bar foo  
foo
core@core-01 ~ $ etcdctl set /foo bar --ttl 5 
bar
core@core-01 ~ $ etcdctl get /foo 
Error:  100: Key not found (/foo) [1934475]
core@core-01 ~ $ 
core@core-01 ~ $ etcdctl set /foo bar --ttl 5 
bar
core@core-01 ~ $ etcdctl get /foo 
bar
core@core-01 ~ $ etcdctl -o extended get /foo  
Error:  100: Key not found (/foo) [1934486]
core@core-01 ~ $ etcdctl set /foo bar --ttl 5 
bar
core@core-01 ~ $ etcdctl -o extended get /foo  
Key: /foo
Created-Index: 1934488
Modified-Index: 1934488
TTL: 4
Index: 1934489

bar
core@core-01 ~ $ etcdctl ls  
/coreos.com
/test -d value=tested
/example
/bar
/message
/here
/x
core@core-01 ~ $ etcdctl ls -p  
/coreos.com/
/test -d value=tested/
/example/
/bar/
/message
/here/
/x/
core@core-01 ~ $ etcdctl ls -p  
/coreos.com/
/test -d value=tested/
/example/
/bar/
/message
/here/
/x/
core@core-01 ~ $ 
core@core-01 ~ $ etcdctl ls -recursive  
/bar
/bar/foo
/coreos.com
/coreos.com/network
/coreos.com/network/config
/coreos.com/network/subnets
/coreos.com/network/subnets/10.1.64.0-24
/coreos.com/network/subnets/10.1.57.0-24
/coreos.com/updateengine
/coreos.com/updateengine/rebootlock
/coreos.com/updateengine/rebootlock/semaphore
/test -d value=tested
/test -d value=tested/00000000000000059444
/example
/example/key
/example/b
/example/b/c
/message
/here
/here/you
/x
/x/y
core@core-01 ~ $ 
core@core-01 ~ $ 
core@core-01 ~ $ etcdctl rm /bar/foo --with-value "This is CoreOS" 
Error:  101: Compare failed ([This is CoreOS != This is updated CoreOS]) [1934641]
core@core-01 ~ $ etcdctl rm /foo --with-index 42  
Error:  100: Key not found (/foo) [1934645]
core@core-01 ~ $ 
core@core-01 ~ $ 
core@core-01 ~ $ curl -L -X PUT http://127.0.0.1:2379/v2/keys/foo -d value="bar"  
{"action":"set","node":{"key":"/foo","value":"bar","modifiedIndex":1934662,"createdIndex":1934662}}
core@core-01 ~ $ curl -L http://127.0.0.1:2379/v2/keys/foo  
{"action":"get","node":{"key":"/foo","value":"bar","modifiedIndex":1934662,"createdIndex":1934662}}
core@core-01 ~ $ curl -L http://127.0.0.1:2379/v2/keys/  
{"action":"get","node":{"dir":true,"nodes":[{"key":"/coreos.com","dir":true,"modifiedIndex":5,"createdIndex":5},{"key":"/test -d value=tested","dir":true,"modifiedIndex":59444,"createdIndex":59444},{"key":"/foo","value":"bar","modifiedIndex":1934662,"createdIndex":1934662},{"key":"/bar","dir":true,"modifiedIndex":1934436,"createdIndex":1934436},{"key":"/message","value":"Hello","modifiedIndex":1879285,"createdIndex":1879285},{"key":"/example","dir":true,"modifiedIndex":1929174,"createdIndex":1929174},{"key":"/here","dir":true,"modifiedIndex":1929576,"createdIndex":1929576},{"key":"/x","dir":true,"modifiedIndex":1929885,"createdIndex":1929885}]}}
core@core-01 ~ $ curl -L http://127.0.0.1:2379/v2/keys/  
{"action":"get","node":{"dir":true,"nodes":[{"key":"/here","dir":true,"modifiedIndex":1929576,"createdIndex":1929576},{"key":"/x","dir":true,"modifiedIndex":1929885,"createdIndex":1929885},{"key":"/bar","dir":true,"modifiedIndex":1934436,"createdIndex":1934436},{"key":"/message","value":"Hello","modifiedIndex":1879285,"createdIndex":1879285},{"key":"/example","dir":true,"modifiedIndex":1929174,"createdIndex":1929174},{"key":"/test -d value=tested","dir":true,"modifiedIndex":59444,"createdIndex":59444},{"key":"/foo","value":"bar","modifiedIndex":1934662,"createdIndex":1934662},{"key":"/coreos.com","dir":true,"modifiedIndex":5,"createdIndex":5}]}}
core@core-01 ~ $ curl -L http://127.0.0.1:2379/v2/keys/foo?wait=true 
^C
core@core-01 ~ $ curl http://127.0.0.1:2379/v2/keys/foo -XDELETE  
{"action":"delete","node":{"key":"/foo","modifiedIndex":1934694,"createdIndex":1934662},"prevNode":{"key":"/foo","value":"bar","modifiedIndex":1934662,"createdIndex":1934662}}
core@core-01 ~ $