dell/instack RDO install instructions and annoyances

gistfile1.txt

 * br-ctlplane config file is rewritten when the services are
   restarted. There needs to be a comment at the top of the file
   indicating that it is automatically generated, and that points to where
   the configuration actually comes from.
 * Ceph deployment?
 * Any mistake in networking config in instack.answers seems to necessitate
   a complete reinstall -- simply making changes and rerunning
   instack-install-undercloud will not work.
 * Ironic code for full RAID and BIOS control needs to be merged into
   Ironic. As of now, it is in the redhat tree, so we are OK.
 * Reboot an overcloud node, it gets its root filesystem reinstalled.
   This behaviour on bare metal nodes seems designed to drive sysadmins
   right up the wall.  I have not tested that in awhile
 * No username/password for local console (or remote KVM) access.  This
   seems designed to make troubleshooting connectivity issues that much harder,
   especially since the above point makes it impossible to set one without
   creating an entire new image to do so.
 * Logging is wholly inadequate.  All logs from overcloud nodes need to be streamed to
   the undercloud for ease of administration and troubleshooting, and the various openstack
   components need to be alot more verbose about logging what failed when and why.
 * Otherwise unconfigured network interfaces on the overcloud nodes do DHCP.
   This means that you can wind up with working addresses on random networks, or run out
   of leases on the discovery range.  Either way, Hilarity can Ensue, either by blowing away
   your desired default route or by not having any addresses available for future discovery.

Detailed undercloud install steps:

Install Centos 7.1.  1 nic for external communnications (eth0), one nic
for overcloud C&C (eth1).  This will be the single undercloud control
host.
Make sure the user you create has admin rights, as the rest of the install will
run as that user.  Do not use the root user account.

Actual commands will be prefixed with $

 Undercloud C&C address range:

 * 192.168.128.0/22
 * The whole range should be the masquerade network.
 * Undercloud eth1 on 192.168.128.3/22
 * Discovery range from 192.168.131.1 - 192.168.131.254
 * DHCP range from 192.168.129.1 - 192.168.129.254
 * iDRACs in 192.168.128.10 - 192.168.128.254
 * Gateway and PXE IP at 192.168.128.3
 
Give yourself passwordless sudo and update everything before starting the install procedure:

$ sudo visudo
  * Comment out the line that reads:
    %wheel  ALL=(ALL)       ALL
  * Uncomment the line that reads:
    # %wheel        ALL=(ALL)       NOPASSWD: ALL
  * Save and exit changes.
$ sudo yum -y update
$ sudo reboot
 
install the undercloud packages following instructions from:

$ curl https://raw.githubusercontent.com/rdo-management/instack-undercloud/master/scripts/instack-setup-host | bash -x
$ sudo yum install -y instack-undercloud

Second, copy over the sample answers file and customize it to taste.

$ cp /usr/share/instack-undercloud/instack.answers.sample ~/instack.answers

Customize instack.answers to match the undercloud C&C address range info.

Here is a pre-customized instack.answers file that I have been using:

# instack answers file

### DEPLOYMENT_MODE ###
# Deployment mode to setup on this Undercloud.
# Choices are poc and scale:
# poc will allow deployment of a single role to heterogenous hardware
# scale will allow deployment of a single role only to homogenous hardware.
DEPLOYMENT_MODE=poc

### IMAGE_PATH ###
# Local file path to the downloaded images.
# The path should be a directory readable by the current user that contains
# the full set of downloaded images.
IMAGE_PATH=.

### LOCAL_IP ###
# IP address to assign to the interface on the Undercloud that will
# be handling the PXE boots and DHCP for Overcloud instances.
# LOCAL_IP will be assigned to LOCAL_INTERFACE, and must be in
# IP/PREFIX format.
LOCAL_IP=192.168.128.3/22

### LOCAL_INTERFACE ###
# Network interface on the Undercloud that will be handling the PXE boots and
# DHCP for Overcloud instances.
# LOCAL_INTERFACE will be assigned the address from LOCAL_IP
LOCAL_INTERFACE=eth1

### MASQUERADE_NETWORK ###
# Network that will be masqueraded for external access if required.
MASQUERADE_NETWORK=192.168.128.0/22

### DHCP_START ###
# Start of DHCP Allocation range for PXE and DHCP of Overcloud instances
DHCP_START=192.168.129.0

### DHCP_END ###
# End of DHCP Allocation range for PXE and DHCP of Overcloud instances
DHCP_END=192.168.129.255

### NETWORK_CIDR ###
# Network cidr for neutron managed network for Overcloud instances
NETWORK_CIDR=192.168.128.0/22

### NETWORK_GATEWAY ###
# Network gateway for neturon managed network for Overcloud instances
NETWORK_GATEWAY=192.168.128.3

### DISCOVERY_INTERFACE ###
# Network interface on which discovery dnsmasq will listen
# If in doubt, do not change
DISCOVERY_INTERFACE=br-ctlplane

### DISCOVERY_IPRANGE ###
# Temporary IP range that will be given to nodes during discovery process
# Should not overlap with Neutron range, but should be in the same network
DISCOVERY_IPRANGE=192.168.131.0,192.168.131.254

### DISCOVERY_PXEIP ###
# IP address of TFTP server for discovery, chances are high that it is LOCAL_IP
DISCOVERY_PXEIP=192.168.128.3

### Database password ###
# Password used for MySQL databases
# If left unset, one will be automatically generated
UNDERCLOUD_DB_PASSWORD=

### Admin Token ###
# Keystone admin token
# If left unset, one will be automatically generated
UNDERCLOUD_ADMIN_TOKEN=

### Admin password ###
# Keystone admin password
# If left unset, one will be automatically generated
UNDERCLOUD_ADMIN_PASSWORD=

### Glance password ###
# Glance service password
# If left unset, one will be automatically generated
UNDERCLOUD_GLANCE_PASSWORD=

### Heat password ###
# Heat service password
# If left unset, one will be automatically generated
UNDERCLOUD_HEAT_PASSWORD=

### Neutron password ###
# Neutron service password
# If left unset, one will be automatically generated
UNDERCLOUD_NEUTRON_PASSWORD=

### Nova password ###
# Nova service password
# If left unset, one will be automatically generated
UNDERCLOUD_NOVA_PASSWORD=

### Ironic password ###
# Ironic service password
# If left unset, one will be automatically generated
UNDERCLOUD_IRONIC_PASSWORD=

### Tuskar password ###
# Tuskar service password
# If left unset, one will be automatically generated
UNDERCLOUD_TUSKAR_PASSWORD=

### Ceilometer password ###
# Ceilometer service password
# If left unset, one will be automatically generated
UNDERCLOUD_CEILOMETER_PASSWORD=

### Ceilometer metering secret ###
# Ceilometer metering secret
# If left unset, one will be automatically generated
UNDERCLOUD_CEILOMETER_METERING_SECRET=

### Ceilometer snmpd user ###
# Ceilometer snmpd user
# If left unset, one will be automatically generated
UNDERCLOUD_CEILOMETER_SNMPD_USER=

### Ceilometer snmpd password ###
# Ceilometer snmpd password
# If left unset, one will be automatically generated
UNDERCLOUD_CEILOMETER_SNMPD_PASSWORD=

### Swift password ###
# Swift password
# If left unset, one will be automatically generated
UNDERCLOUD_SWIFT_PASSWORD=

### Rabbit Cookie ###
# Rabbit Cookie
# If left unset, one will be automatically generated
UNDERCLOUD_RABBIT_COOKIE=

### Rabbit Password ###
# Rabbit Password
# If left unset, one will be automatically generated
UNDERCLOUD_RABBIT_PASSWORD=

### Rabbit Username ###
# Rabbit Username
# If left unset, one will be automatically generated
UNDERCLOUD_RABBIT_USERNAME=

### Heat Stack Domain Admin Password ###
# Heat Stack Domain Admin Password
# If left unset, one will be automatically generated
UNDERCLOUD_HEAT_STACK_DOMAIN_ADMIN_PASSWORD=

### Swift Hash Suffix ###
# Swift Hash Suffix
# If left unset, one will be automatically generated
UNDERCLOUD_SWIFT_HASH_SUFFIX=

### END instack.answers


On Centos, DO NOT customize or edit the hostname in any way, as it
will cause RabbitMQ to fail startup.

Run the understack install script:

$ instack-install-undercloud

Copy over the passwords and stack configuration:
$ sudo cp /root/tripleo-undercloud-passwords .
$ sudo cp /root/stackrc .

Ship this until https://review.openstack.org/#/c/171303/ has been merged,
and do not reboot your undercloud or it will stop working:
Reboot the undercloud to ensure it comes back up properly:
$ sudo reboot

Create and install the disk images:

$ instack-build-images && instack-prepare-for-overcloud

At this point, we should be ready to deploy the overcloud.  Working assumptions
for overcloud installation are:

* iDRACs on all overcloud nodes are reachable from the undercloud eth1.
* The first embedded 1 gig port on the nodes will be used for the undercloud
  nodes to manage the overcloud.
* All overcloud nodes are Dell 11 gen or higher, with an appropriate mix
  of hardware configurations for the node you will deploy.
* The undercloud network is one big flat network.


Download and install Go, then install idracula:

$ echo 'export GOPATH=$HOME/go' >>$HOME/.bash_profile
$ echo 'export PATH=$PATH:$HOME/go/bin' >> $HOME/.bash_profile
$ . $HOME/.bash_profile
$ sudo yum -y install golang
$ go get github.com/VictorLowther/idracula
$ go install github.com/VictorLowther/idracula

Use idracula to configure remote hosts for PXE booting over the first 1 gig
embedded Ethernet port, and write out the nodes.json.  If the hardware needs to
be configured to PXE boot, this may take awhile.  The username, password, and
address range to scan should be customized to taste.

$ idracula -u root -p 'password' -scan '192.168.128.10-192.168.128.100' >~/nodes.json

Patch os-cloud-config to support pxe_drac.  
This is only needed until https://review.openstack.org/#/c/171774/
is merged.

$ git clone https://review.openstack.org/openstack/os-cloud-config
$ cd os-cloud-config/
$ git fetch https://review.openstack.org/openstack/os-cloud-config refs/changes/74/171774/4
$ git checkout FETCH_HEAD
$ sudo cp -r os_cloud_config/* /usr/lib/python2.7/site-packages/os_cloud_config
$ cd $HOME

Apply the patch https://review.openstack.org/#/c/173047/ to
/usr/lib/python2.7/site-packages/ironic/drivers/modules/drac/management.py

This will let Ironic properly handle not needing to always set the boot device.

Register and discover the nodes:

$ . stackrc
$ instack-ironic-deployment --nodes-json nodes.json --register-nodes
$ instack-ironic-deployment --discover-nodes
$ instack-ironic-deployment --setup-flavors

Validate that the appropriate nodes have been assigned profiles:

$ instack-ironic-deployment --show-profile

If they have not, you can manually assign profiles with:

$ ironic node-update <node uuid> add properties/capabilities=profile:<profile>

Set up and apply the BIOS settings for the nodes:

$ git clone https://github.com/dell-esg/instack-dell-config
$ cd instack-dell-config
$ ./bios.sh
$ cd ..
$ instack-ironic-deployment --configure-bios

If instack-ironic-deployment does not wait for the BIOS settings to apply, you will
need to do so manually:
* Open the iDRAC consoles for each system to monitor their progress.
* Reboot each of the nodes manually with:
  $ ironic node-set-power-state <node uuid> reboot
* Wait for the nodes to finish applying their new BIOS settings via the lifecycle controller
If the output of instack-ironic-deployment --configure-bios does not mention needing to reboot
the machine to apply changes, then you do not need to do the above steps manually.


Copy over the deploy-overcloudrc and customize it to taste. 

$ cp /usr/share/instack-undercloud/deploy-baremetal-overcloudrc ~/deploy-overcloudrc
* <edit the file to taste>
$ . deploy-overcloudrc

An example deploy-overcloudrc file is here:

#!/bin/bash
export NeutronPublicInterface=nic1
export OVERCLOUD_LIBVIRT_TYPE=kvm
export NETWORK_CIDR="10.0.0.0/8"
export FLOATING_IP_START="172.17.0.45"
export FLOATING_IP_END="172.17.0.64"
export FLOATING_IP_CIDR="172.17.0.0/16"
export BM_NETWORK_GATEWAY="172.17.0.1"
export NEUTRON_NETWORK_TYPE=gre
export NEUTRON_TUNNEL_TYPES=gre
export COMPUTESCALE=1
export BLOCKSTORAGESCALE=0
export SWIFTSTORAGESCALE=0
export NODES_JSON=$HOME/nodes.json

Deploy your overcloud:

$ instack-deploy-overcloud

Troubleshoot why your deploy failed. because it almost certianly will.