gistfile1.txt

# Installing Manila backed by CephFS

This document assumes that you have already installed Ceph and Openstack

## Setting up Keepalived

### Install Keepalived and setup its config

These commands are executed from the *ceph deployment node*

``` 
apt install keepalived
systemctl enable keepalived
vim /etc/init/keepalived.conf
```

**/etc/init/keepalived.conf**
```
description "Load-balancing and high-availability service"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

exec /user/local/sbin/keepalived --dont-fork
```

Use an IP from the storage network to serve as the endpoint for the nfs-ganesha service. This IP must be made available to all VMs that are going to use Openstack Manila, so work with the networking team to ensure this happens. This IP will be shared with all of the ceph-mds nodes in the ceph cluster.

Replace `<BIND_ADDR>` with this IP 

```
vim /etc/keepalived/keepalived.conf
```

**/etc/keepalived/keepalived.conf**
```
global_defs {
  enable_script_security
  script_user root
}

vrrp_script pingable_check_script {
    script "/bin/ping -c 2 8.8.8.8 1>&2"
    interval 10  # Check every 10 seconds
    fall 2   # Require 2 failures (default: 3)
    rise 4   # Require 4 successes for OK (default: 6)
    timeout 5  # allow scripts like ping to succeed, before timing out
}

vrrp_instance VI_NFSGANESHA {
    interface <storage_interface>
    state BACKUP
    priority 100
    nopreempt

    virtual_router_id 5

    authentication {
        auth_type PASS
        auth_pass secrete
    }

    virtual_ipaddress {
        <BIND_ADDR>/32 dev <storage_interface>
    }

    track_script {
        pingable_check_script
    }

  notify_master /etc/keepalived/nfs-ganesha_master.sh
  notify_backup /etc/keepalived/nfs-ganesha_backup.sh
  notify_fault /etc/keepalived/nfs-ganesha_fault.sh

}
```

### Create notify scripts

**/etc/keepalived/nfs-ganesha_master.sh**

```
#/bin/bash
/bin/systemctl restart nfs-ganesha.service
```

**/etc/keepalived/nfs-ganesha_backup.sh**

```
#/bin/bash
/bin/systemctl stop nfs-ganesha.service
```

**/etc/keepalived/nfs-ganesha_fault.sh**

```
#/bin/bash
/bin/systemctl stop nfs-ganesha.service
```

### Restart keepalived
```
systemctl restart keepalived.service
```

### Install keepalived on all ceph-mds nodes and push the config files to those servers

```
ssh <ceph-mds-node> apt install keepalived
scp /etc/init/keepalived.conf <ceph-mds-node>:/etc/init/keepalived.conf
scp /etc/keepalived/keepalived.conf <ceph-mds-node>:/etc/keepalived/keepalived.conf
ssh <ceph-mds-node> systemctl restart keepalived.service
```

## Setting up the Ceph Cluster

These commands are executed from the *ceph deployment node*

### Create the Manila keyring for the Ceph cluster

```
read -d '' MON_CAPS << EOF
allow r,
allow command "auth del",
allow command "auth caps",
allow command "auth get",
allow command "auth get-or-create"
EOF

ceph auth get-or-create client.manila -o ceph.client.manila.keyring \
mds 'allow *' \
osd 'allow rw' \
mon "$MON_CAPS"
```

### Edit the ceph-ansible templates for ceph-nfs to always use the bind address

#### In /opt/ceph-ansible/roles/ceph-nfs/templates/ganesha.conf.js edit the NFS_CORE_PARAM block to the following

``` 
vim /opt/ceph-ansible/roles/ceph-nfs/templates/ganesha.conf.js
```

**/opt/ceph-ansible/roles/ceph-nfs/templates/ganesha.conf.js**
```
NFS_CORE_PARAM
{
{% if ceph_nfs_bind_addr is defined %}
       Bind_Addr={{ ceph_nfs_bind_addr }};
{% endif %}
       Enable_NLM = false;
       Enable_RQUOTA = false;
       Protocols = 4;
}
```

### Add NFS variables to /opt/ceph-ansible/group_vars/nfss.yml

```
vim /opt/ceph-ansible/group_vars/nfss.yml
```

**/opt/ceph-ansible/group_vars/nfss.yml**
```
ceph_nfs_bind_addr: <IP FROM KEEPALIVED HERE>
ceph_nfs_secret_key: "<PUT MANILA KEY HERE>"
ceph_nfs_ceph_squash: "No_root_squash"

ganesha_conf_overrides: |
    NFSv4 {
            # Modern versions of libcephfs have delegation support. See below.
            Delegations = true;

            # One can use any recovery backend with this configuration, but being
            # able to store it in RADOS is a nice feature that makes it easy to
            # migrate the daemon to another host.
            #
            # For a single-node or active/passive configuration, rados_ng driver
            # is preferred.
            RecoveryBackend = rados_ng;

            # NFSv4.0 clients do not send a RECLAIM_COMPLETE, so we end up having
            # to wait out the entire grace period if there are any. Avoid them.
            Minor_Versions =  1,2;
    }

    # Config block for FSAL_CEPH
    CEPH {
            # Path to a ceph.conf file for this cluster.
            Ceph_Conf = /etc/ceph/ceph.conf;

            # User file-creation mask. These bits will be masked off from the unix
            # permissions on newly-created inodes.
            # umask = 0;
    }
    RADOS_KV {
            # Path to a ceph.conf file for this cluster.
            # Ceph_Conf = /etc/ceph/ceph.conf;

            # The recoverybackend has its own ceph client. The default is to
            # let libcephfs autogenerate the userid. Note that RADOS_KV block does
            # not have a setting for Secret_Access_Key. A cephx keyring file must
            # be used for authenticated access.
            # UserId = "ganesharecov";

            # Pool ID of the ceph storage pool that contains the recovery objects.
            # The default is "nfs-ganesha".
            pool = "cephfs_data";
    }
```

### Add client.manila to ceph.conf generation

##### Under ceph_conf_overrides (at the end of the section)

**/opt/ceph-ansible/group_vars/all.yml**
```
   "client.manila":
     "client mount uid": "0"
     "client mount gid": "0"
     "log file": "/var/log/ceph/ceph-cleint.manila.log"
     "admin socket": "/var/run/ceph-mds.{{ hostvars[inventory_hostname]['ansible_hostname'] }}.asok"
     "keyring": "/etc/ceph/ceph.client.manila.keyring"
```

### If not already created, create /etc/ceph/ceph.client.manila.keyring on all ceph hosts

**/etc/ceph/ceph.client.manila.keyring**
```
[client.manila]
     key=<PUT MANILA KEY HERE>
```

```
scp /etc/ceph/client.manila.keyring  <mds-host>:/etc/ceph
```

### Confirm that both mdss and nfss sections are defined in the ceph_inventory. Add the sections if they aren't there.

**example (YAML)**
```
    mdss:
      hosts:
        Vaporeon:
        Flareon:
        Jolteon:
        Espeon:
        Umbreon:
    nfss:
      hosts:
        Vaporeon:
        Flareon:
        Jolteon:
        Espeon:
        Umbreon:
```

**example (INI)**

```
[mdss]
Vaporeon
Flareon
Jolteon
Espeon
Umbreon

[nfss]
Vaporeon
Flareon
Jolteon
Espeon
Umbreon
```

```
cat /opt/ceph-ansible/ceph_inventory | egrep 'mdss|nfss' -A3
```

### Run the ceph-ansible playbook

```
source /opt/ceph-toolkit/ceph_deploy/bin/activate
cd /opt/ceph-ansible/
ansible-playbook -i ceph_inventory site.yml
```

### Confirm that the mds and ganesha services have been installed (ganesha will only be up on the server that has the keepalived bind IP)

```
ceph -s
ansible -i ceph_inventory mdss -m shell -a "systemctl status nfs-ganesha.service"
ansible -i ceph_inventory mdss -m shell -a "ip a l <storage-interface>"
ansible -i ceph_inventory mdss -m shell -a "netstat -plnt | grep nfs"
```

### Install sqlite3 on all nfs-ganesha servers

```
ssh <nfs-ganesha-server> apt install sqlite3
```

## Setting up the Openstack Cloud

These commands are executed from the *Openstack deployment node*

### Edit Openstack Configs for Manila

##### Add the following to /etc/openstack_deploy/user_local_variables.yml

**/etc/openstack_deploy/user_local_variables.yml**
```
manila_protocols:
  - NFS
  - CIFS
  - CEPHFS

manila_backends:
  cephfsnfs1:
    driver_handles_share_servers: False
    share_backend_name: CEPHFSNFS1
    share_driver: manila.share.drivers.cephfs.driver.CephFSDriver
    cephfs_protocol_helper_type: NFS
    cephfs_conf_path: /etc/ceph/ceph.conf
    cephfs_auth_id: manila
    cephfs_cluster_name: ceph
    cephfs_enable_snapshots: False
    cephfs_ganesha_server_is_remote: True
    cephfs_ganesha_server_ip: <storage_network_ip> # The IP you set for keepalived

manila_default_store: cephfsnfs1

horizon_enable_manila_ui: False #currently broken
```

##### Add the IP used for Keepalived to the list of used IPs in the openstack_user_config.yml

```
vim /etc/openstack_deploy/openstack_user_config.yml
```

**/etc/openstack_deploy/openstack_user_config.yml**
```
used_ips:
  - "172.29.236.1,172.29.238.255"
  - "172.29.240.0,172.29.242.255"
  - "172.29.244.1,172.29.246.255"
  - "<KEEPALIVED_IP>"
```

##### Define Manila nodes

```
vim /etc/openstack_deploy/conf.d/os-manila.conf
```

**/etc/openstack_deploy/conf.d/os-manila.conf**
```
manila-infra_hosts: &manila-infra_hosts
  controller1:
    ip: <CONTROLLER1_IP>
    no_containers: true
  controller2:
    ip: <CONTROLLER2_IP>
    no_containers: true
  controller3:
    ip: <CONTROLLER3_IP>
    no_containers: true

manila-data_hosts: *manila-infra_hosts
```

##### Check that manila passwords have been defined

``` 
grep manila /etc/openstack_deploy/user_secrets.yml 
```

The following should be declared and defined

```
manila_oslomsg_rpc_password:
manila_container_mysql_password: 
manila_service_password:
manila_profiler_hmac_key:
```

### Add nodes to openstack_inventory.json

```
python /opt/openstack-ansible/inventory/dynamic_inventory.py
cat /etc/openstack_deploy/openstack_inventory.json | grep manila -A5
```

### Run Playbooks

##### Run repo-install playbook

```
cd /opt/openstack-ansible/playbooks
openstack-ansible repo-install.yml
```

##### Run haproxy-install playbook

```
openstack-ansible haproxy-install.yml
```

##### Run os-manila-install playbook

```
openstack-ansible os-manila-install.yml
```

## Configure Manila

These commands are executed from the *Openstack deployment node*

#### The following steps are a bit jank. Workarounds have been documented as best as they can be. Expect these steps to change as the playbooks are fixed and the process is improved.

##### Confirm that the manila-share service is running

``` 
journalctl -f -u manila-share
```

**The service should be complaining that it cannot ssh to the ceph-mds nodes. This is because the playbooks, although creating the manila user, do not create a ssh-key for the manila user to use when connecting to the nfs-ganesha nodes. Therefore, you must become the manila user and create it, as well as push the public key to the nfs-ganesha nodes.**

##### Generate key pair for manila user

```
mkdir /home/manila/.ssh
ssh-keygen -f /home/manila/.ssh/id_rsa
chown -R manila:manila /home/manila/.ssh
```

##### Populate authorized_keys files on nfs-ganesha nodes

```
ansible -i ceph_inventory mdss -m authorized_key -a "user=root state=present key=\"$(cat /home/manila/.ssh/id_rsa.pub)\" "
```

##### Populate the manila key pair on the controller nodes

For each controller node:
```
rsync -av /home/manila/.ssh  controller0X:/home/manila/
```

**Once the ssh keys are in place, Manila should work**

## Post Install QC

These commands are executed from the *Openstack deployment node*

### Create a share type for Ceph

```
manila type-create cephfsnfstype false
manila type-key cephfsnfstype set vendor_name=Ceph storage_protocol=NFS
```

### Create a share and make it available

```
manila create --share-type cephfsnfstype --name cephnfsshare1 nfs 10
manila share-export-location-list cephnfsshare1
```

**Example output**
```
+--------------------------------------+------------------------------------------------------------------------+-----------+
| ID                                   | Path                                                                   | Preferred |
+--------------------------------------+------------------------------------------------------------------------+-----------+
| f5021e54-bf03-43be-91f7-c32a57881a43 | <KEEPALIVED_IP>:/volumes/_nogroup/cb89652d-369d-423f-a9bd-ad5b255d3513 | False     |
+--------------------------------------+------------------------------------------------------------------------+-----------+
```

### Create an Ubuntu VM and mount the filesystem



**The target for the NFS share has one extra part that manila does not report. For example, if Manila gives you the output from above**

```
<KEEPALIVED_IP>:/volumes/_nogroup/cb89652d-369d-423f-a9bd-ad5b255d3513
```

**The actual path of the NFS share is**

```
<KEEPALIVED_IP>:/cephfile/volumes/_nogroup/cb89652d-369d-423f-a9bd-ad5b255d3513
```

These commands are run from *inside the VM*

```
sudo -i
apt install -y <nfs-client>
mkdir /mnt/tmp
mount -t nfs -o nfsvers=4.1,proto=tcp 10.40.0.62:/cephfile/volumes/_nogroup/cb89652d-369d-423f-a9bd-ad5b255d3513 /mnt/tmp
```

### Run fio test to confirm operation

```
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/mnt/tmp/cephfstest --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
```