shannonmitchell

# installed the centos minimal version into virtualbox on windows 10


yum groupinstall "GNOME Desktop"

reboot

yum install cmake python-devel numpy gcc gcc-c++ gtk2-devel libdc1394-devel libv4l-devel \
gstreamer-plugins-base-devel libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel \
libtiff-devel libwebp-devel tbb-devel eigen3-devel python-sphinx texlive python-matplotlib git

shannonmitchell

## Download the base distro

Find the distro you want at:

http://releases.ubuntu.com/

```
wget http://releases.ubuntu.com/trusty/ubuntu-14.04.5-server-amd64.iso

md5sum ubuntu-14.04.5-server-amd64.iso 

shannonmitchell

[root@shannon osic]# cat /data/osic/openstack-deploy.py

#!/usr/bin/python

import os
import sys
import time
import argparse
import datetime
import threading

shannonmitchell

Task: 046dd220-e0de-4f49-bf09-64fc56549ddd.json


# Progress

  - Jan  3 04:07:55: from the deploy box to haproxy 
  - 2017-01-03 04:07:55.733:    network-2_neutron_agents_container-54f08b31 was hit to create a new network resource
  - 2017-01-03 04:07:55.734:    network-2_neutron_agents_container-54f08b31 tried to connect create a network and failed.

shannonmitchell

#################################
# May Need Further Investigating
#################################

- ca4cbc80-f4e7-4b10-b16f-83266b5761a9(research complete): 
  -> NeutronNetworks.create_and_list_networks(1): kill neutron-server service on one node
  -> This one had a lot of downtime around 150 -> 25 seconds of the test on and off.  This was well after the restart and before the 1k limit was hit

- 48feed1c-c902-4bb1-b166-1b5cd62687b5(research complete) 
    -> NeutronNetworks.list_agents(1): restart keystone service on one node

shannonmitchell

##################################
# openstack-ansible config notes:
##################################


Some entries work with setup-hosts for network configuration and such and are never used after.  
Everything else is just ansible variables that get used in all the playbook roles.  Some are put 
in inventory and others get pulled in via the openstack-ansible wrapper when running the playbooks.  

shannonmitchell

(rally) root@deploy:~# for RT in $(rally task list | awk '/failed/{print $2}'); do rally task detailed $RT; done

--------------------------------------------------------------------------------
Task db4c33a6-e09c-4a4f-84f1-158931fc44d5: failed
--------------------------------------------------------------------------------
InvalidTaskException
Task config is invalid: `Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.InternalError'> (HTTP 500) (Request-ID: req-89be9d3f-6fea-477d-aa47-68371f198512)`

For more details run:

shannonmitchell

###########
# Notes
###########

The idea here is to simulate a data loss incident due to a network partition.   The following steps were taken:
  - We will take infra02-rabbit-mq out of the cluster by blocking network access
  - We will restart the service on infra02-rabbitmq
  - We will modify the data on the good partition.
  - We will join the infra02-rabbit-mq server back into the cluster
  - We will then restart both nodes on the good partition.

shannonmitchell

By looking at the keystone restarts, it happened twice on infra-1, once on infra-2 and none on infra-3.

root@deploy:~# ansible -i /opt/openstack-ansible/playbooks/inventory/dynamic_inventory.py keystone_all -m shell -a "grep restart /var/log/salt/minion | grep '2017-01-27 18'"
infra-3_keystone_container-8a194cbe | FAILED | rc=1 >>


infra-1_keystone_container-0ea74c78 | SUCCESS | rc=0 >>
2017-01-27 18:23:52,437 [salt.loaded.int.module.cmdmod][INFO    ] Executing command 'service apache2 restart' in directory '/root'
2017-01-27 18:34:07,240 [salt.loaded.int.module.cmdmod][INFO    ] Executing command 'service apache2 restart' in directory '/root'

shannonmitchell

################################################
# Build out a new AIO environment for testing
################################################

(These steps are only used for testing the playbooks in an AIO environment.  Can throw away after)
(The customer's environment that we are initially testing on should already be in place)

  - Created an OnMetal IO v1 in IAD via the interface.