Skip to content

Instantly share code, notes, and snippets.

@jpapazian2000

jpapazian2000/Vault_as_a_KMS_for_Ceph.md

Forked from jeromebaude/Vault_as_a_KMS_for_Ceph.md
Created July 22, 2020 15:33
Show Gist options
  • Save jpapazian2000/0f79ece4356ff24fd317257fc6cb5aff to your computer and use it in GitHub Desktop.
Save jpapazian2000/0f79ece4356ff24fd317257fc6cb5aff to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Vault_as_a_KMS_for_Ceph.md

HashiCorp Vault as a KMS for Ceph

The following readme guides you thru the steps to set up a minimal demo using a local Vault and a 1 node Rook/Ceph cluster hosted in Minikube

Install and start minikube

$ minikube start (ou minikube start --driver=virtualbox)

When installing rook for the first time, make sure we have a raw device on the minikube host (https://rook.io/docs/rook/v1.3/ceph-quickstart.html)

$ minikube ssh
$ lsblk -f
$ exit

Install and configure Rook

$ git clone --single-branch --branch release-1.3 https://github.com/rook/rook.git

Deploy the rook operator

$ cd ./rook/cluster/examples/kubernetes/ceph
$ kubectl create -f common.yaml
$ kubectl create -f operator.yaml

Verify the rook-ceph-operator is in the Running state before proceeding

$ kubectl -n rook-ceph get pod

Update the Ceph config (ceph.conf)

First, update the cluster-test.yaml file with the floowing ceph configuration:

debug rgw = 20/5
rgw crypt s3 kms backend = vault
rgw crypt vault auth = token
rgw crypt vault token file = /ceph/vault.token
rgw crypt vault addr = http://192.168.0.12:8200
rgw crypt vault secret engine = transit
rgw crypt vault prefix = /v1/transit/export/encryption-key
rgw crypt require ssl = false

Deploy the rook cluster

$ kubectl create -f cluster-test.yaml

(To be added: config: encryptedDevice: "true")

Check pods are running

$ kubectl -n rook-ceph get pod

Check logs

$ kubectl logs -n rook-ceph --all-containers rook-ceph-osd-prepare-minikube-dztj7

(you should see that Ceph is using a raw device to create an OSD. If it can't, you will see an error message at the bottom)

Provision Object storage

$ kubectl create -f object-test.yaml

Check that the rgw pod is running (it takes a few minutes before being deployed)

$ kubectl -n rook-ceph get pod -l app=rook-ceph-rgw

Create a StorageClass

$ kubectl create -f storageclass-bucket-delete.yaml

Claim a bucket

$ kubectl create -f object-bucket-claim-delete.yaml

Set a Vault token for RGW to AuthN to Vault

$ kubectl -n rook-ceph exec -it rook-ceph-rgw-my-store-a-688c79f8ff-5xtpv /bin/sh
sh-4.4# mkdir /ceph
sh-4.4# echo "s.WK1AULVFXXXXXXXXXX" > /ceph/vault.token
sh-4.4# chmod 600 /ceph/vault.token
sh-4.4# chown ceph /ceph/vault.token

Configure Vault

Enable the transit engine

$ vault secrets enable transit

Create an exportable key

$ vault write -f transit/keys/mybucketkey exportable=true

Create a policy and a token to access the previous key

$ TOKEN=xxx vault read transit/export/encryption-key/mybucketkey/1

Communicate as a client to the RadosGateway

Get you S3 AKSK

$ kubectl -n default get secret ceph-delete-bucket -o yaml | grep AWS_ACCESS_KEY_ID | awk '{print $2}' | base64 --decode
$ kubectl -n default get secret ceph-delete-bucket -o yaml | grep AWS_SECRET_ACCESS_KEY | awk '{print $2}' | base64 --decode

Install and run the toolbox

$ kubectl apply -f toolbox.yaml
$ kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') /bin/sh

Look at the ceph and osd status

# ceph status
# ceph osd status

Check that the S3 bucket has been created

# radosgw-admin bucket list
[
    "rookbucket-6ae7d67e-0e27-42a5-ba00-bb869739b163"
]

Upload a file to the newly created bucket

# echo "Hello Rook" > /tmp/rookObj
# yum --assumeyes install s3cmd
# s3cmd put /tmp/rookObj --access_key=6GYOI27BO9GWNN87EU72 --secret_key=HslhLOXztccsxEpB6VEdbGYkbDGQ3Vf7wD9Tuyao  --no-ssl --host=rook-ceph-rgw-my-store.rook-ceph --server-side-encryption --server-side-encryption-kms-id=mybucketkey/1 --host-bucket=  s3://rookbucket-6ae7d67e-0e27-42a5-ba00-bb869739b163

Download and verify the file content from the bucket

# s3cmd get s3://rookbucket-6ae7d67e-0e27-42a5-ba00-bb869739b163/rookObj /tmp/rookObj-download --access_key=6GYOI27BO9GWNN87EU72 --secret_key=HslhLOXztccsxEpB6VEdbGYkbDGQ3Vf7wD9Tuyao  --no-ssl --host=rook-ceph-rgw-my-store.rook-ceph --host-bucket=
# cat /tmp/rookObj-download

Check that my data is encrypted

exec in my OSD prod

$ kubectl -n rook-ceph exec -it rook-ceph-osd-0-74c969c6f6-7zlnt /bin/sh
sh-4.4# cd /var/lib/ceph/osd/ceph-0
sh-4.4# cat block

Debugging

Check rgw logs

$ kubectl -n rook-ceph logs -f rook-ceph-rgw-my-store-a-688c79f8ff-5xtpv

Check that your Vault key is accessible

$ curl --header "X-Vault-Token: s.WK1AUXXXXXXXX" http://192.168.0.12:8200/v1/transit2/export/encryption-key/mybucketkey/1 | jq

Enable the Dashboard (optional)

Create a User

$ kubectl create -f object-user.yaml

Check that the user was created

$ kubectl -n rook-ceph describe secret rook-ceph-object-user-my-store-my-user

Retrieve AccessKey for this user

$ kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml | grep AccessKey | awk '{print $2}' | base64 --decode

Retrieve SecretKey for this user

$ kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml | grep SecretKey | awk '{print $2}' | base64 --decode

Access the Ceph Dashboard

$ kubectl port-forward -n rook-ceph svc/rook-ceph-mgr-dashboard 7000:7000

Enabling Dashboard Object Gateway management (commands to be entered in the toolbox)

Enable system flag on the user:

radosgw-admin user modify --uid=my-user --system

Provide the user credentials:

ceph dashboard set-rgw-api-user-id my-user
ceph dashboard set-rgw-api-access-key <access-key>
ceph dashboard set-rgw-api-secret-key <secret-key>

Clean up things

$ kubectl delete -f object-user.yaml (if created)
$ kubectl delete -f toolbox.yaml
$ kubectl delete -f object-bucket-claim-delete.yaml
$ kubectl delete -f storageclass-bucket-delete.yaml
$ kubectl delete -f object-test.yaml
$ kubectl delete -f cluster-test.yaml
$ kubectl delete -f operator.yaml
$ kubectl delete -f common.yaml
$ minikube ssh "sudo rm -rf /var/lib/rook"
$ minikube stop
(delete the inital raw device from virtualbox)

References:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment