Testing Sticky Sessions

gistfile1.md

Overview

1. Sticky sessions are provided in two ways, cookies and stick-tables
2. Routes that run with an http mode backend (unsecure, edge, and reencrypt) utilize cookies
3. Routes that run with a tcp mode backend utilize stick-tables
4. Sticky sessions are implementation specific to a router. We have implemented them in the HAProxy template router.

Testing Cookie Based Sticky Sessions

Cookie based sticky sessions can be tested with curl. When accessing a route a cookie will be saved to disk which should be resubmitted with subsequent curl requests. If you would like to utilize a second http server to produce output different than what is produced by a first http server than the hello-nginx-docker Dockerfile can be rebuilt on minions, modifying the "Hello World Test" section OR you may nsenter the docker container and modify it by hand if that is easier.

Unsecure routes

1. Start your vagrant environment, this can be in a single machine environment as no syncing between router peers is required.
2. Start two hello-nginx-docker pods that are covered by a single service to produce a backend with multiple server entry lines
3. Create the unsecure route
4. Confirm that the haproxy config has the backend and it has a cookie entry like cookie OPENSHIFT_{{$cfgIdx}}_SERVERID insert indirect nocache httponly.
5. Perform a curl to the route with -c to save the cookie to a file
6. Perform a curl to the route with -b to submit the file. You should get sent to the same server as the step above
7. Save a new cookie file and repeat the steps above. You should be balanced to the next server and keep hitting it as long as you submit the same cookie
8. You may also examine the cookie files and see that the cookie is set with the same name indicated in the haproxy config for all routes

########################## Haproxy config
backend be_http_default-route-unsecure
                
  mode http
  balance leastconn
  timeout check 5000ms   
  cookie OPENSHIFT_default-route-unsecure_SERVERID insert indirect nocache                  
  server 172.17.0.15:80 172.17.0.15:80 check inter 5000ms cookie 172.17.0.15:80                
  server 172.17.0.16:80 172.17.0.16:80 check inter 5000ms cookie 172.17.0.16:80
 
########################### curling
[vagrant@openshiftdev sticky]$ curl -c cookies.txt --resolve www.example.com:80:10.0.2.15 http://www.example.com
Hello World
[vagrant@openshiftdev sticky]$ curl -b cookies.txt --resolve www.example.com:80:10.0.2.15 http://www.example.com
Hello World
[vagrant@openshiftdev sticky]$ curl -b cookies.txt --resolve www.example.com:80:10.0.2.15 http://www.example.com
Hello World
[vagrant@openshiftdev sticky]$ curl -b cookies.txt --resolve www.example.com:80:10.0.2.15 http://www.example.com
Hello World
[vagrant@openshiftdev sticky]$ curl -c cookies2.txt --resolve www.example.com:80:10.0.2.15 http://www.example.com
Test2!!!
[vagrant@openshiftdev sticky]$ curl -b cookies2.txt --resolve www.example.com:80:10.0.2.15 http://www.example.com
Test2!!!
[vagrant@openshiftdev sticky]$ curl -b cookies2.txt --resolve www.example.com:80:10.0.2.15 http://www.example.com
Test2!!!
[vagrant@openshiftdev sticky]$ curl -b cookies2.txt --resolve www.example.com:80:10.0.2.15 http://www.example.com
Test2!!!

Edge terminated routes

1. The steps for an unsecure route may be used to test an edge terminated route by creating the new edge terminated route that reuses the unsecure service
2. The cookie for the edge terminated route will be different as shown below

########################## Haproxy config
backend be_edge_http_default-route-edge 
  mode http
  balance leastconn
  timeout check 5000ms  
  cookie OPENSHIFT_EDGE_default-route-edge_SERVERID insert indirect nocache secure  
  server 172.17.0.15:80 172.17.0.15:80 check inter 5000ms cookie 172.17.0.15:80
  server 172.17.0.16:80 172.17.0.16:80 check inter 5000ms cookie 172.17.0.16:80
 
########################### curling
[vagrant@openshiftdev sticky]$ curl -c cookies.txt --resolve www.example.com:443:10.0.2.15 https://www.example.com -k
Hello World
[vagrant@openshiftdev sticky]$ curl -b cookies.txt --resolve www.example.com:443:10.0.2.15 https://www.example.com -k
Hello World
[vagrant@openshiftdev sticky]$ curl -b cookies.txt --resolve www.example.com:443:10.0.2.15 https://www.example.com -k
Hello World
[vagrant@openshiftdev sticky]$ curl -c cookies.txt --resolve www.example.com:443:10.0.2.15 https://www.example.com -k
Test2!!!
[vagrant@openshiftdev sticky]$ curl -b cookies.txt --resolve www.example.com:443:10.0.2.15 https://www.example.com -k
Test2!!!
[vagrant@openshiftdev sticky]$ curl -b cookies.txt --resolve www.example.com:443:10.0.2.15 https://www.example.com -k
Test2!!!

Reencrypt routes

1. Remove the services and routes from previous testing and create the new secure service and route as shown in the hello-nginx-docker repo
2. The cookie for a reencrypt route will look the same as the edge terminated route but have a different name

Testing Stick-table Based Sticky Sessions

In order to test stick-table based sticky sessions you need to run a multi-node openshift environment. This is to test the sycning of stick-table entries between peers in a haproxy multi-router setup. To help, the hello-sni repo has been updated to provide some utility objects https://github.com/pweil-/hello-sni

1. Start your multi node environment
2. Ensure you are running with the latest haproxy image (can be done manually by building the image on each node if necessary)
3. Create a router with multiple replicas which will cause a replica to be placed on each of the two nodes.
4. Create a router with multiple replicas openshift admin router --credentials="$OPENSHIFTCONFIG" --replicas=2 --images=openshift/origin-haproxy-router
5. Build the hello-sni images on both minions (it is not pushed to the repo as of writing). A build script is provided, it should be run with sudo to access docker in the vagrant multi-node setup
6. Create pod.json
7. Create pod2.json (this serves a different message, examine the pod definition for details)
8. Create service.json
9. Create route.json

At this point you should have the following set up and ready to test:

1. A router service with 2 endpoints listed
2. Two routers running and in ready state
3. Two pods running and in ready state
4. The tls-service with 2 endpoints listed

Now, to testing...

1. nsenter both router containers
2. Check the process and ensure it was started with a -L flag that gives the ip/port combination that will be used as the local peer name
3. Check the haproxy.config for the peer section that lists both routers
4. Check the backend for the passthrough route. You should see a stick-table section that references the peer section
5. Check the stick-tables on each router, they should be empty at this point. echo 'show table be_tcp_default-route-passthrough' | socat unix-connect:/var/lib/haproxy/run/haproxy.sock stdio
6. On the master (a node will reference itself and other nodes with different ip addresses, not a good test for sticky-sessions since each ip address gets its own stick table entry but you can use it to verify the other pod is balanced) use the provided client to hit both routers go run client.go -server openshift-minion-1:443 go run client.go -server openshift-minion-2:443 multiple times. You should get the same output from each router you hit.
7. Recheck the stick-tables on each router. You should now see an entry for the master ip address