Running a demo of a 3 node Scylla cluster, Scylla Manager and Monitoring

Demo Scylla Manager with Docker

# Start a Scylla cluster
########################

docker run --name some-scylla -d scylladb/scylla --smp 1 --memory 750M --overprovisioned 1 --api-address 0.0.0.0
# Do not use the above --api-address for production. Exposing API IP is not secure.
SEED=$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' some-scylla)
echo $SEED
docker run --name some-scylla2 -d scylladb/scylla --smp 1 --memory 750M --overprovisioned 1 --api-address 0.0.0.0 --seeds="$SEED"
docker run --name some-scylla3 -d scylladb/scylla --smp 1 --memory 750M --overprovisioned 1 --api-address 0.0.0.0 --seeds="$SEED"

# Valdiate cluster is up and running
> docker exec -it some-scylla nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns    Host ID                               Rack
UN  172.18.0.2  110.77 KB  256          ?       63ef531d-13a4-4ffa-9d28-3985aeb5dc18  rack1
UN  172.18.0.3  104.88 KB  256          ?       7e985ea1-66d8-4c47-a517-2ef22f70d29c  rack1
UN  172.18.0.4  110.58 KB  256          ?       c5c0bd6f-c7b4-4dee-8fe9-0ca495b02a2b  rack1

# Start A Scylla Manager
########################

# Best is *not* to use a production cluster as a backend for Manager
docker run --name scylla-manager-db -d scylladb/scylla --smp 1 --memory 750M --overprovision 1
docker run -d --name scylla-manager --link scylla-manager-db scylladb/scylla-manager

# Validate Scylla Manager is up and running (may take 1-2 min)

> docker exec -it scylla-manager sctool version
Client version: 1.3.1-0.20190124.08fc477
Server version: 1.3.1-0.20190124.08fc477

# Add the cluster to the Manager

> docker exec -it scylla-manager sctool cluster add --host="$SEED" --name=my-cluster 
30ab8b15-3515-4e37-b3d0-fa1d92c84870
 __
/  \     Cluster added, to set it as a default run:
@  @     export SCYLLA_MANAGER_CLUSTER=30ab8b15-3515-4e37-b3d0-fa1d92c84870
|  |
|| |/    Repair will run on 27 Feb 19 00:00:00 UTC and will be repeated every 7d.
|| ||    To see the currently scheduled tasks: sctool task list -c 30ab8b15-3515-4e37-b3d0-fa1d92c84870
|\_/|
\___/

> docker exec -it scylla-manager bash
#> sctool cluster list
╭──────────────────────────────────────┬────────────────┬──────────╮
│ cluster id                           │ name           │ ssh user │
├──────────────────────────────────────┼────────────────┼──────────┤
│ 30ab8b15-3515-4e37-b3d0-fa1d92c84870 │ my-cluster     │          │
╰──────────────────────────────────────┴────────────────┴──────────╯

#> exit

# Extract the IP address of Scylla and Manager for the next step
> docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' some-scylla some-scylla2 some-scylla3
> docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' scylla-manager


# Setup Monitoring
##################
git clone https://github.com/scylladb/scylla-grafana-monitoring.git
cd scylla-grafana-monitoring

# edit prometheus/scylla_servers.yml with scylla nodes:

- targets:
    - 172.17.0.2:9180
    - 172.17.0.3:9180
    - 172.17.0.4:9180

  labels:
       cluster: my-cluster

# edit prometheus/scylla_manager_servers.yml with manager IP:

- targets:
  - 172.17.0.5:56090

./start-all.sh -v 3.0 -M 1.3

Grafana IP: http://localhost:3000
Prometheus IP: http://localhost:9090

More:

Take a node down

> docker stop some-scylla3

Use monitoring tools to see the effect:

> docker exec -it scylla-manager sctool status -c my-cluster
> docker exec -it some-scylla nodetool status

and monitoring dashboard (manager, cluster)

Start it agian with

docker start some-scylla3