Skip to content

Instantly share code, notes, and snippets.

Last active January 9, 2023 15:33
What would you like to do?
Expose Rook Ceph cluster and connect OpenStack

Pt.1: Let's prepare minikube first!

minikube start --force --memory="4096" --cpus="2" -b kubeadm --kubernetes-version="v1.19.2" --driver="kvm2" --feature-gates="BlockVolume=true,CSIBlockVolume=true,VolumeSnapshotDataSource=true,ExpandCSIVolumes=true"

Where rook put all the cluster info?

minikube ssh "sudo mkdir -p /mnt/vda1/var/lib/rook;sudo ln -s /mnt/vda1/var/lib/rook /var/lib/rook"

We need at least one disk for our standalone Ceph cluster!

sudo -S qemu-img create -f raw /var/lib/libvirt/images/minikube-box-vm-disk-50G 50G
virsh -c qemu:///system attach-disk minikube --source /var/lib/libvirt/images/minikube-box-vm-disk-50G --target vdb --cache none
virsh -c qemu:///system reboot --domain minikube

Check the disk is there and restart

minikube ssh "sudo lsblk"
minikube start --force --memory="4096" --cpus="2" -b kubeadm --kubernetes-version="v1.19.2" --driver="kvm2" --feature-gates="BlockVolume=true,CSIBlockVolume=true,VolumeSnapshotDataSource=true,ExpandCSIVolumes=true"

Rook time

git clone

cd deploy/examples/

alias k="kubectl"
alias kc="kubectl create -f"

Patch cluster-test to use hostNetwork: true as the goal is to expose this cluster outside and connect OpenStack to the cluster provided by minikube.

Apply the following to the cluster-test.yaml cluster we're going to create.

diff --git a/deploy/examples/cluster-test.yaml b/deploy/examples/cluster-test.yaml
index cd3a3d111..2254cacd8 100644
--- a/deploy/examples/cluster-test.yaml
+++ b/deploy/examples/cluster-test.yaml
@@ -27,6 +27,8 @@ metadata:
   namespace: rook-ceph # namespace:cluster
   dataDirHostPath: /var/lib/rook
+  network:
+    hostNetwork: true
     allowUnsupported: true

for i in crds common operator cluster-test toolbox; { kubectl create -f $i.yaml; }

And wait for the cluster:

  `$ k get pods -n rook-ceph -w`

  > k get pods -n rook-ceph
  NAME                                            READY   STATUS      RESTARTS   AGE
  csi-cephfsplugin-cmnzc                          3/3     Running     0          4m1s
  csi-cephfsplugin-provisioner-6d4bd9b669-jkwkb   6/6     Running     0          4m1s
  csi-rbdplugin-provisioner-6bcd78bb89-hbn6d      6/6     Running     0          4m1s
  csi-rbdplugin-s85sj                             3/3     Running     0          4m1s
  rook-ceph-mgr-a-595dc9f57f-2rg89                1/1     Running     0          3m12s
  rook-ceph-mon-a-6b4c8b75df-s9qxr                1/1     Running     0          3m28s
  rook-ceph-operator-7bf8ff479-fn2x8              1/1     Running     0          7m23s
  rook-ceph-osd-0-67df5fd4d7-s6brg                1/1     Running     0          106s
  rook-ceph-osd-prepare-minikube-lcm88            0/1     Completed   0          113s
  rook-ceph-tools-5c6844fcd5-bb75h                1/1     Running     0          9s

> kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0]}') bash

[rook@rook-ceph-tools-5c6844fcd5-bb75h /]$ ceph -s
    id:     59998731-37b2-4a5d-a6c6-7d5e15f16408
    health: HEALTH_OK

    mon: 1 daemons, quorum a (age 4m)
    mgr: a(active, since 2m)
    osd: 1 osds: 1 up (since 2m), 1 in (since 2m)

    pools:   1 pools, 32 pgs
    objects: 0 objects, 0 B
    usage:   5.0 MiB used, 50 GiB / 50 GiB avail
    pgs:     32 active+clean

  [rook@rook-ceph-tools-5c6844fcd5-bb75h /]$ cat /etc/ceph/ceph.conf
  mon_host =

  keyring = /etc/ceph/keyring

A few checks

  [rook@rook-ceph-tools-5c6844fcd5-bb75h /]$ ceph osd dump
  epoch 12
  fsid 59998731-37b2-4a5d-a6c6-7d5e15f16408
  created 2022-04-22T08:00:09.332006+0000
  modified 2022-04-22T08:02:01.929903+0000
  flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
  crush_version 4
  full_ratio 0.95
  backfillfull_ratio 0.9
  nearfull_ratio 0.85
  require_min_compat_client luminous
  min_compat_client jewel
  require_osd_release pacific
  stretch_mode_enabled false
  pool 1 'device_health_metrics' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 10 flags hashpspool stripe_width 0 application mgr_devicehealth
  max_osd 1
  osd.0 up   in  weight 1 up_from 8 up_thru 8 down_at 0 last_clean_interval [0,0) [v2:,v1:] [v2:,v1:] exists,up 37bdd9a3-27b8-4a78-9709-c4cef983fa98
  blocklist expires 2022-04-23T08:01:56.083964+0000
  blocklist expires 2022-04-23T08:01:56.083964+0000
  blocklist expires 2022-04-23T08:01:56.083964+0000
  blocklist expires 2022-04-23T08:01:44.618135+0000
  blocklist expires 2022-04-23T08:01:44.618135+0000
  blocklist expires 2022-04-23T08:01:44.618135+0000
  blocklist expires 2022-04-23T08:01:56.083964+0000
  blocklist expires 2022-04-23T08:01:44.618135+0000

  > minikube ssh "sudo ip -o -4 a"
  1: lo    inet scope host lo\       valid_lft forever preferred_lft forever
  2: eth0    inet brd scope global dynamic eth0\       valid_lft 2510sec preferred_lft 2510sec
  3: eth1    inet brd scope global dynamic eth1\       valid_lft 2510sec preferred_lft 2510sec
  5: docker0    inet brd scope global docker0\       valid_lft forever preferred_lft forever

We can see the osd map as well as the ceph monitors exposed to the network, which is the minikube network:

  > s virsh net-list
   Name           State    Autostart   Persistent
   minikube-net   active   yes         yes

  [INS] (0) ~/labs/cube/rook/deploy/examples(v1.8.7)[*|C:8289]
  > s virsh net-dumpxml minikube-net
  <network connections='1'>
    <bridge name='virbr1' stp='on' delay='0'/>
    <mac address='52:54:00:ff:48:28'/>
    <dns enable='no'/>
    <ip address='' netmask=''>
        <range start='' end=''/>

It's almost TripleO time, but first let's create what we need on the Ceph cluster

  1. create pools
for pool in volumes images vms; {
    ceph osd pool create $pool;
    ceph osd pool application enable $pool rbd;
  1. create keys
ceph auth add client.openstack mgr 'allow *' mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images'

ceph auth ls

        key: AQD1ZmJiBVOvMBAAJkaQo567m2sI9uZwVS5tXg==
        caps: [mgr] allow *
        caps: [mon] profile rbd
        caps: [osd] profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images

Prepare the TripleO standalone VM

Let's follow the standalone tripleo guide to start preparing an environment that should be connected to the ceph cluster.

Attach the bridge to the minikube-net to be able to reach the k8s cluster (especially Ceph):

br=$(sudo virsh net-dumpxml minikube-net | awk '/virbr/ {print $2}' | cut -d= -f2)
sudo virsh attach-interface --domain standalonec9_default --type bridge --source $br --model virtio --config --live

and try to reach the Ceph cluster:

  telnet 6789
  Connected to
  Escape character is '^]'.
  ceph v027'G'
  telnet> quit
  Connection closed.

After preparing the environment as per the procedure above, edit the /usr/share/openstack-tripleo-heat-templates/environments/external-ceph.yaml environment file adding the Ceph info of the step before.

  [stack@standalone ~]$ cat /usr/share/openstack-tripleo-heat-templates/environments/external-ceph.yaml | grep -vE "#|^$"

    OS::TripleO::Services::CephExternal: ../deployment/cephadm/ceph-client.yaml
    CephClusterFSID: '59998731-37b2-4a5d-a6c6-7d5e15f16408'
    CephClientKey: 'AQD1ZmJiBVOvMBAAJkaQo567m2sI9uZwVS5tXg=='
    CephExternalMonHost: ''
    NovaEnableRbdBackend: true
    CinderEnableRbdBackend: true
    CinderBackupBackend: ceph
    GlanceBackend: rbd
    NovaRbdPoolName: vms
    CinderRbdPoolName: volumes
    CinderBackupRbdPoolName: backups
    GlanceRbdPoolName: images
    CephClientUserName: openstack
    CinderEnableIscsiBackend: false

and we're ready to deploy!

  export IP=
  export NETMASK=24
  export INTERFACE=eth1
  sudo openstack tripleo deploy \
    --templates \
    --local-ip= \
    -e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \
    -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \
    -e "$HOME/containers-prepare-parameters.yaml" \
    -e "$HOME/standalone_parameters.yaml" \
    -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/external-ceph.yaml \
    --output-dir $HOME \

  export OS_CLOUD=standalone
  openstack endpoint list
  openstack volume create test --size 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment