Burst-from-onprem-osp-to-aws-gluster

Burst-from-onprem-osp-to-aws-gluster

Prerequisites:

Functioning DNS
Functioning OpenStack environment
Functioning AWS environment
Proper permissions

On premise OpenShift on OpenStack environment:

1 Infrastructure node
3 OpenShift Masters
2 OpenShift Application nodes
3 OpenShift Storage nodes
9 cinder volumes @ 75GB each - 3 per storage node

Remote OpenShift on AWS environment:

2 OpenShift Application nodes
3 OpenShift Storage nodes
9 ebs volumes @ 75GB each - 3 per storage node

High level steps:

DNS records for new nodes
Deploy 3 masters on OpenStack
Deploy 1 infra
Deploy 2 app nodes
Deploy 3 storage nodes
Associated 3 cinder volumes to each storage node
Configur all the hosts per normal OpenShift deployment methods to prep OSE install
Deploy OpenShift on the local RHEL OSP cluster
Deploy OpenShift remote on AWS 
Deploy app storage nodes in AWS
Deploy GlusterFS cluster local
Deploy GlusterFS cluster remote in aws
Set up geo-replication between the GlusterFS clusters
Configure persistent volumes local
Configure persistent volumes remote
Start testing apps... 

Detailed steps:

Deploy OpenStack instances -

nova boot --flavor m1.medium --image rhel72 --nic net-id=3ded0e35-e2f6-4ce3-93cf-496392ba131f --nic net-id=4488b814-dab1-43d7-98ce-b8537422924c --key-name scollier scollier-master-1.e2e.bos.redhat.com
nova boot --flavor m1.medium --image rhel72 --nic net-id=3ded0e35-e2f6-4ce3-93cf-496392ba131f --nic net-id=4488b814-dab1-43d7-98ce-b8537422924c --key-name scollier scollier-master-2.e2e.bos.redhat.com
nova boot --flavor m1.medium --image rhel72 --nic net-id=3ded0e35-e2f6-4ce3-93cf-496392ba131f --nic net-id=4488b814-dab1-43d7-98ce-b8537422924c --key-name scollier scollier-master-3.e2e.bos.redhat.com
nova boot --flavor m1.medium --image rhel72 --nic net-id=3ded0e35-e2f6-4ce3-93cf-496392ba131f --nic net-id=4488b814-dab1-43d7-98ce-b8537422924c --key-name scollier scollier-infra-1.e2e.bos.redhat.com
nova boot --flavor m1.medium --image rhel72 --nic net-id=3ded0e35-e2f6-4ce3-93cf-496392ba131f --nic net-id=4488b814-dab1-43d7-98ce-b8537422924c --key-name scollier scollier-app-1.e2e.bos.redhat.com
nova boot --flavor m1.medium --image rhel72 --nic net-id=3ded0e35-e2f6-4ce3-93cf-496392ba131f --nic net-id=4488b814-dab1-43d7-98ce-b8537422924c --key-name scollier scollier-app-2.e2e.bos.redhat.com
nova boot --flavor m1.medium --image rhel72 --nic net-id=3ded0e35-e2f6-4ce3-93cf-496392ba131f --nic net-id=4488b814-dab1-43d7-98ce-b8537422924c --key-name scollier scollier-storage-1.e2e.bos.redhat.com
nova boot --flavor m1.medium --image rhel72 --nic net-id=3ded0e35-e2f6-4ce3-93cf-496392ba131f --nic net-id=4488b814-dab1-43d7-98ce-b8537422924c --key-name scollier scollier-storage-2.e2e.bos.redhat.com
nova boot --flavor m1.medium --image rhel72 --nic net-id=3ded0e35-e2f6-4ce3-93cf-496392ba131f --nic net-id=4488b814-dab1-43d7-98ce-b8537422924c --key-name scollier scollier-storage-3.e2e.bos.redhat.com

Associate floating IPs

nova floating-ip-associate scollier-master-1.e2e.bos.redhat.com 10.19.114.98
nova floating-ip-associate scollier-master-2.e2e.bos.redhat.com 10.19.114.99
nova floating-ip-associate scollier-master-3.e2e.bos.redhat.com 10.19.114.100
nova floating-ip-associate scollier-infra-1.e2e.bos.redhat.com 10.19.114.101
nova floating-ip-associate scollier-app-1.e2e.bos.redhat.com 10.19.114.103
nova floating-ip-associate scollier-app-2.e2e.bos.redhat.com 10.19.114.104

# Get port UUIDs
for i in 11 12 13 14 15; do neutron port-list | grep 10.0.0.$i; done

# Update ports
# for i in 40887a67-5b37-4993-a3dd-5ef708cde45c 738d2b0a-63a2-4324-9583-239c148226c0 c01c0f03-98ff-4ddd-ace7-5de73bfd1ec4 f2355cdc-463f-4de1-8726-c932b953e7d4 3973f6a3-cc16-4b4e-a8a8-fb815bf17257; do neutron port-update $i --no-security-groups --port-security-enabled=False; done

Configure the infrastructure node - 

sed -i -e '/^PEERDNS=/s/=.*/="no"/' \
        /etc/sysconfig/network-scripts/ifcfg-eth0

subscription-manager register --username rhn-engineering-scollier --password Boodle_12345 --auto-attach
subscription-manager subscribe --pool 8a85f9823e3d5e43013e3dce8ff306fd

subscription-manager repos --disable="*"
subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-ose-3.2-rpms" \
    --enable=rhel-7-server-extras-rpms \
    --enable=rhel-7-server-optional-rpms

yum -y update

yum -y install atomic-openshift-utils

mkdir /var/lib/ansible

Configure the master, application and storage nodes -

sed -i -e '/^PEERDNS=/s/=.*/="no"/' \
        /etc/sysconfig/network-scripts/ifcfg-eth0

subscription-manager register --username rhn-engineering-scollier --password Boodle_12345 --auto-attach
subscription-manager subscribe --pool 8a85f9823e3d5e43013e3dce8ff306fd
subscription-manager repos --disable="*"
subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-ose-3.2-rpms" \
    --enable=rhel-7-server-extras-rpms \
    --enable=rhel-7-server-optional-rpms           
yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion httpd-tools
yum -y update

yum -y install docker
systemctl enable docker

echo "INSECURE_REGISTRY='--insecure-registry 0.0.0.0'" >> /etc/sysconfig/docker

On a single Master server that you will run the heketi client from -

subscription-manager repos --enable=rh-gluster-3-for-rhel-7-server-rpms
yum install heketi-client heketi-templates

On the storage nodes -
subscription-manager repos --enable=rh-gluster-3-for-rhel-7-server-rpms
iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2222 -j ACCEPT
service iptables save

Configure networking for OSP instances -

ip a

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO="dhcp"
BOOTPROTOv6="dhcp"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
PEERDNS="no"
IPV6INIT="yes"
PERSISTENT_DHCLIENT="1"
EOF

dhclient eth1

ip a

Create the Cinder volumes -

for i in 1 2 3; do for j in 1 2 3; do cinder create 75 --display-name storage-$i-$j; done; done

Attach the Cinder volumes -

nova volume-attach 92ff0895-1b27-4da0-99df-e42b84c7c5e5 8a622016-4c62-424e-914f-bbca784c5b94
nova volume-attach 92ff0895-1b27-4da0-99df-e42b84c7c5e5 c9e68135-6210-462d-899b-caa84b6682e2 
nova volume-attach 92ff0895-1b27-4da0-99df-e42b84c7c5e5 19c9f0db-b733-47f3-96e5-9a6a894f6e19 
etc... for the other servers - 3 volumes per storage node.


Configure the Ansible inventory file -

[OSv3:children]
infra
masters
nodes
etcd

[OSv3:vars]

ansible_user=cloud-user
# ansible_sudo=true
ansible_become=true
deployment_type=openshift-enterprise
osm_default_subdomain=scollier-apps.e2e.bos.redhat.com
openshift_override_hostname_check=true
openshift_use_openshift_sdn=true
openshift_set_node_ip=true
# openshift_use_flannel=false
openshift_use_dnsmasq=false
# flannel_interface=eth1
openshift_cloudprovider_kind=openstack
openshift_cloudprovider_openstack_auth_url=http://10.19.115.62:5000/v2.0
openshift_cloudprovider_openstack_username=scollier
openshift_cloudprovider_openstack_password=100Root-
openshift_cloudprovider_openstack_tenant_name=scollier
openshift_cloudprovider_openstack_region=RegionOne
openshift_master_cluster_hostname=ose-ha-proxy-scollier.e2e.bos.redhat.com
openshift_master_cluster_public_hostname=ose-ha-proxy-scollier.e2e.bos.redhat.com
openshift_master_cluster_method=native

[infra]
localhost

[masters]
scollier-master-1.e2e.bos.redhat.com openshift_hostname=scollier-master-1.e2e.bos.redhat.com openshift_public_hostname=scollier-master-1.e2e.bos.redhat.com openshift_ip=192.168.0.50
scollier-master-2.e2e.bos.redhat.com openshift_hostname=scollier-master-1.e2e.bos.redhat.com openshift_public_hostname=scollier-master-1.e2e.bos.redhat.com openshift_ip=192.168.0.51
scollier-master-3.e2e.bos.redhat.com openshift_hostname=scollier-master-1.e2e.bos.redhat.com openshift_public_hostname=scollier-master-1.e2e.bos.redhat.com openshift_ip=192.168.0.52

[masters:vars]
openshift_schedulable=true
openshift_router_selector=region=infra

[etcd]
scollier-master-1.e2e.bos.redhat.com
scollier-master-2.e2e.bos.redhat.com
scollier-master-3.e2e.bos.redhat.com

[nodes]
scollier-master-1.e2e.bos.redhat.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_hostname=scollier-master-1.e2e.bos.redhat.com openshift_public_hostname=scollier-master-1.e2e.bos.redhat.com openshift_ip=192.168.0.50
scollier-master-2.e2e.bos.redhat.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_hostname=scollier-master-2.e2e.bos.redhat.com openshift_public_hostname=scollier-master-2.e2e.bos.redhat.com openshift_ip=192.168.0.51
scollier-master-3.e2e.bos.redhat.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_hostname=scollier-master-3.e2e.bos.redhat.com openshift_public_hostname=scollier-master-3.e2e.bos.redhat.com openshift_ip=192.168.0.52

scollier-storage-1.e2e.bos.redhat.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_hostname=scollier-storage-1.e2e.bos.redhat.com openshift_public_hostname=scollier-storage-1.e2e.bos.redhat.com openshift_ip=192.168.0.47
scollier-storage-2.e2e.bos.redhat.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_hostname=scollier-storage-2.e2e.bos.redhat.com openshift_public_hostname=scollier-storage-2.e2e.bos.redhat.com openshift_ip=192.168.0.48
scollier-storage-3.e2e.bos.redhat.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_hostname=scollier-storage-3.e2e.bos.redhat.com openshift_public_hostname=scollier-storage-3.e2e.bos.redhat.com openshift_ip=192.168.0.49

scollier-app-1.e2e.bos.redhat.com openshift_node_labels="{'region': 'primary', 'zone': 'default'}" openshift_hostname=scollier-app-1.e2e.bos.redhat.com openshift_public_hostname=scollier-app-1.e2e.bos.redhat.com openshift_ip=192.168.0.45
scollier-app-2.e2e.bos.redhat.com openshift_node_labels="{'region': 'primary', 'zone': 'default'}" openshift_hostname=scollier-app-2.e2e.bos.redhat.com openshift_public_hostname=scollier-app-2.e2e.bos.redhat.com openshift_ip=192.168.0.46

# [dns]
# localhost


Prepare the infrastructure node to run Ansible -

export ANSIBLE_ROLES_PATH=/usr/share/ansible/openshift-ansible/roles
export ANSIBLE_HOST_KEY_CHECKING=False

eval `ssh-agent`
ssh-add scollier.pem

Run the OpenShift Ansible installer -

ansible-playbook -vvvv --inventory /var/lib/ansible/inventory /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

Configure basic authentication so people can log into the OpenShift GUI -

htpasswd -bc /etc/origin/users.htpasswd admin password

# configure the master-config.yaml file to enable httpd auth
sed -i 's/name: deny_all/name: web_auth/' /etc/origin/master/master-config.yaml
sed -i 's/kind: DenyAllPasswordIdentityProvider/kind: HTPasswdPasswordIdentityProvider/' /etc/origin/master/master-config.yaml
sed -i '/kind: HTPasswdPasswordIdentityProvider/a \     \ file: /etc/origin/users.htpasswd' /etc/origin/master/master-config.yaml

systemctl restart atomic-openshift-master-api.service atomic-openshift-master-controllers.service

Add the proper permissions to the admin user to manage the cluster -

Configure the local GlusterFS cluster on the OpenStack VMs running in the local OpenShift cluster -

oadm policy add-cluster-role-to-user cluster-admin admin
oc login # admin | password
oc new-project gluster-storage
oadm policy add-scc-to-user privileged -z gluster-storage
oc create -f /usr/share/heketi/templates
oc get templates

oc process glusterfs -v GLUSTERFS_NODE=scollier-storage-1.e2e.bos.redhat.com | oc create -f -
oc process glusterfs -v GLUSTERFS_NODE=scollier-storage-2.e2e.bos.redhat.com | oc create -f -
oc process glusterfs -v GLUSTERFS_NODE=scollier-storage-3.e2e.bos.redhat.com | oc create -f -

oc process deploy-heketi -v HEKETI_KUBE_APIHOST='https://ose-ha-proxy-scollier.e2e.bos.redhat.com:8443' HEKETI_KUBE_INSECURE=y HEKETI_KUBE_USER=admin HEKETI_KUBE_PASSWORD=password HEKETI_KUBE_NAMESPACE=gluster-storage | oc create -f -

GOT TO STEP 4.4 here: https://access.redhat.com/documentation/en/red-hat-gluster-storage/3.1/single/deployment-guide-for-containerized-red-hat-gluster-storage-in-openshift-enterprise/