gistfile1.txt

QE, we want to test a configuration that is a variation on the standard HA
(multi-broker) configuration with F5 BIG-IP plus routing-daemon.  In particular,
we want to set up clustered F5 BIG-IP, we want to colocate an instance of the
routing-daemon on *each* broker host, and we want to ensure that ActiveMQ is
configured with redelivery enabled.

Our goal is to verify that this configuration works as expected and that we can
support it for customers.

The standard openshift.sh installation script can perform most of the
configuration on the OpenShift hosts given the proper parameters, as follows.

When installing the nameserver, configure a router host name—e.g.,
router.hosts.example.com.  If you install the nameserver using openshift.sh, you
can specify router_hostname=router.hosts.example.com. This host name must
resolve to the F5 BIG-IP vip.

When installing each broker host,

• specify both "broker" and "router" components so that the routing-daemon and
  rubygem-openshift-origin-routing-activemq packages will be installed and
  configured on the broker host: "install_components=broker,router";
• specify "router=f5" so that the routing-daemon will be configured to use F5
  BIG-IP;
• specify the router host name—e.g., "router_hostname=router.hosts.example.com";
• specify activemq_replicants—e.g., "activemq_replicants=activemq01.hosts.example.com,activemq02.hosts.example.com,activemq03.hosts.example.com"
  (this is required anyway to configure the broker to use clustered ActiveMQ,
  and it is also used to configure the broker plug-in);
• specify routing_user_pass to set the password with which the broker plug-in
  and routing-daemon will authenticate with ActiveMQ;
  and
• specify "enable_ha=true"
  (this is required anyway to enable HA, irrespective of which external
  load-balancer solution is used).

When installing each ActiveMQ host,

• specify activemq_replicants—e.g., "activemq_replicants=activemq01.hosts.example.com,activemq02.hosts.example.com,activemq03.hosts.example.com"
  (this is required anyway for clustered ActiveMQ);
• specify routing_plugin=true
  (this is required to enable redelivery and add a user that the routing-daemon
  and broker plug-in will use in the ActiveMQ configuration);
  and
• specify routing_user_pass to the same password used when configuring the
  broker hosts.

Other configuration parameters, and the configuration for the nameserver, node,
and datastore hosts will be the same as usual.

Once the hosts are configured by openshift.sh, manual configuration is required
to set the F5 BIG-IP credentials and other parameters in
/etc/openshift/routing-daemon.conf.

On each broker host, after openshift.sh is done, edit
/etc/openshift/routing-daemon.conf and do the following:

• change ACTIVEMQ_DESTINATION=/topic/routinginfo to
  ACTIVEMQ_DESTINATION=/queue/routinginfo;
• verify that LOAD_BALANCER=f5 and the other ACTIVEMQ_* settings were correctly set by
  openshift.sh;
• verify that VIRTUAL_SERVER and VIRTUAL_HTTPS_SERVER match the virtual servers
  configured in F5 BIG-IP;
• verify that CLOUD_DOMAIN is correct;
• change BIGIP_HOST to a comma-separated list of host names or IP addresses for
  the F5 BIG-IP hosts;
• change BIGIP_DEVICE_GROUP to the device group of the F5 cluster; and
• change BIGIP_USERNAME and BIGIP_PASSWORD to the credentials of the F5 BIG-IP
  hosts.

The restart the openshift-routing-daemon service.

On each broker host, after openshift.sh is done, edit
/etc/openshift/plugins.d/openshift-origin-routing-activemq.conf and do the
following:

• change ACTIVEMQ_DESTINATION=/topic/routinginfo to
  ACTIVEMQ_DESTINATION=/queue/routinginfo; and
• verify that the other ACTIVEMQ_* settings were correctly set by openshift.sh.

Then restart the openshift-broker service.

What we want to test is the behaviour with multiple routing-daemon instances
concurrently updating F5 BIG-IP:

Scenario 1: With the above setup, create a scalable application; scale it up;
ensure the pool is created and endpoints are added for the new gears; scale it
down; ensure endpoints corresponding to deleted gears are deleted; delete the
application; ensure the pool is deleted.

We should expect to see the configuration changes in F5 BIG-IP take longer than
usual, because one routing-daemon instance may try to perform one operation
(such as adding an endpoint to a pool) at the same time that another
routing-daemon instance is performing a prerequisite operation (such as creating
the pool), in which case the first routing-daemon instance will need to retry
the operation one or more times before it will succeed.  However, thanks to
ActiveMQ redelivery, the routing-daemon instances should retry their operations
in such scenarios.  Thus we expect to see eventually consistent configuration in
this scenario.

Scenario 2: We should also check what happens when one or more routing-daemon
instances dies (through graceful shutdown or hard kill) during an application
creation, deletion, or scaling operation.  The remaining instance or instances
should continue to perform configuration.  Thus we expect to see no interruption
of service in this scenario.

Scenario 3: We should also check what happens when an F5 BIG-IP host is down
during an application creation, deletion, or scaling operation.  Because we
configure BIGIP_HOST with the entire F5 cluster, we should see the
routing-daemons retry with the second F5 BIG-IP host if the first becomes
unresponsive.  Thus we expect to see no interruption of service in this
scenario.

Scenario 4: We should also check what happens when an F5 BIG-IP host goes down;
application creation, deletion, or scaling operations take place; and the F5
BIG-IP host is restarted and rejoins the cluster.  We want to verify that the
rejoining F5 BIG-IP host properly syncs configuration, that the routing-daemon
instances continue to function properly, and that we continue to see
(eventually) consistent configuration in the F5 cluster.

https://github.com/openshift/openshift-extras/commit/2da96a21681d0838a5f2a3e3665e9fd47ce218f2
https://github.com/openshift/openshift-extras/commit/ae49b7aa49b9b66c5decff4ff4cb327f14d82cf8