GNU plot ab performance and CPU for a tsuru installation

README.md

AB benchmark and plotting

Small script to execute a apache benchmark ab from one host to other.

The scripts are designed for benchmark tsuru routers, and compare between hipache+nginx vs. vulcand.

It will gather metrics of request time and CPU in the target so they can be later plot using gnuplot.

Requirements

Software in the remote hosts

You must install:

• dstat in the target host to get CPU metrics: ssh -F ssh.config $TARGET_HOST sudo apt-get install dstat
• ab in the source host to execute the test: ssh -F ssh.config -l ubuntu $SOURCE_HOST sudo apt-get install apache2-utils

Configuration changes

You need to increase the files open limit in both origin and target nodes:

cat <<'EOF' | sudo tee  /etc/security/limits.d/files.conf
* soft nofile 40000
* hard nofile 40000
EOF

sudo reboot # Recommended, I got issues to get this working if not :-m

Routers configuration

If you want to test the routers directly, they allow connections from the source host. Usually they should listen 0.0.0.0

Specifically, on hipache you might need to change /etc/hipache.conf to make it listen 0.0.0.0 in order to do non SSL tests.

Target app

We need to hit some application. For this test, as we use tsuru, we deploy a small static app as target for this test, called mystatic.

tsuru login administrator@gds.tsuru.gov

mkdir -p /tmp/myapp
cd /tmp/myapp

# Add an empty index.html
touch index.html
# And a file of 100k
dd if=/dev/urandom of=bigfile count=100 bs=1k

# And deploy the app
tsuru app-create mystatic static -t admin
tsuru app-deploy . -a mystatic
tsuru app-info -a mystatic

We scale this app to 6-8 units (containers) so we are sure that the backend is not the bottleneck.

tsuru unit-add 6 -a mystatic

SSH configuration

This script expects a file called ssh.config with the SSH configuration like bastion hosts, ssh options, etc.

In our case, copy or link the file from tsuru-ansible: ln -s ~/gds/tsuru-ansible/ssh.config .

Example run

Gather metrics

The script is setup using environment variables, were the mandatory ones are:

• DEPLOY_ENV: your environment
• TARGET_HOST: one or the routers
• SOURCE_HOST: for instance galdalf host

Check all the environment variables in generate_perf_test.sh for all posible options.

A very minimal run:

DEPLOY_ENV=hector270715 \
TARGET_HOST=10.129.0.72 \
SOURCE_HOST=10.129.0.10 \
  ./execute_perf_test.sh

Example suites/plans

In test-plan.sh there are some plans as example.

Generate graphs and markdown

The script generate-md.sh will generate graphs from results directories.

aggregate.plot

#
# Call it with:
#
#
#
# Use filename for plotting
#
if (!exists("filename")) filename='default.csv'
if (!exists("output_file")) output_file='output.jpg'

# Let's output to a jpeg file
set terminal jpeg size 800,600
# This sets the aspect ratio of the graph
set size 1, 1
# The file we'll write to
set output output_file
# The graph title
set title title

# Where to place the legend/key
set key left top
# Draw gridlines oriented on the y axis
set grid y
# Label the x-axis
set xlabel 'requests'
# Label the y-axis
set ylabel "response-time"
# Tell gnuplot to use tabs as the delimiter instead of spaces (default)
set datafile separator '\t'

# Second axis
set y2label 'requests'
set y2tics nomirror
set autoscale y2

# Plot the data
plot \
 filename every ::2 using 9:xticlabels(5) title "request/sec" axes x1y2 with boxes fs solid 0.1, \
 "" every ::2 using 4 title "Failures" axes x1y2 with boxes fs solid 0.1 linecolor rgb "#FF0000", \
 "" every ::2 using 6 title "mean" with lines lw 3, \
 "" every ::2 using 7 title "p95" with lines lw 3, \
 "" every ::2 using 8 title "p98" with lines lw 3
exit

cpu.plot

set xdata time
set timefmt "%s"
set format x "%M:%S"
set output output_file
set terminal jpeg size 500,500
plot input_file using 1:2 title "User" with lines, "" using 1:3 title "Sys" with lines, "" using 1:4 title "Idle" with lines

execute_perf_test.sh

#!/bin/bash -e

# set -x # Uncomment for debug

#
# Executes a ab test in a remote hosts and gathers CPU metrics
#

# Example:
#  DEPLOY_ENV=hector270715 \
#  TARGET_HOST=10.129.0.72 \
#  SOURCE_HOST=hector270715-gandalf.tsuru2.paas.alphagov.co.uk \
#  ./script.sh

die(){
  echo $@
  exit 1
}

DEPLOY_ENV=${DEPLOY_ENV}
[ "$DEPLOY_ENV" ] || die "Must specify DEPLOY_ENV"
TARGET_HOST=${TARGET_HOST}
[ "$TARGET_HOST" ] || die "Must specify TARGET_HOST"
SOURCE_HOST=${SOURCE_HOST}
[ "$SOURCE_HOST" ] || die "Must specify SOURCE_HOST"

TARGET_URL=${TARGET_URL:-https://$TARGET_HOST:443/index.html}

# Result target directory
TARGET_DIR=${TARGET_DIR:-/tmp/benchmark-test}
mkdir -p $TARGET_DIR

# Host header to pass in the request, to hit the backend apps:
HOST_HEADER=${HOST_HEADER:-mystatic.$DEPLOY_ENV-hipache.tsuru2.paas.alphagov.co.uk}

# Enable keepalive: -k
ENABLE_KEEP_ALIVE=${ENABLE_KEEP_ALIVE:-}
# Extra AB Options.
EXTRA_AB_OPTIONS=${EXTRA_AB_OPTIONS:-}

NUM_REQUESTS=${NUM_REQUESTS:-500}
CONCURRENT_REQUESTS=${CONCURRENT_REQUESTS:-20}
TIMEOUT_REQUEST=${TIMEOUT_REQUEST:-3000}

# Special flags to pass to SSH
SSH_FLAGS=${SSH_FLAGS:--F ssh.config }

#########################################################################

# Fail quick if we can not ssh
check_remote_hosts() {
  echo "Trying to ssh to $SOURCE_HOST and $TARGET_HOST"
  echo ssh -T $SSH_FLAGS $SOURCE_HOST true
  ssh -T $SSH_FLAGS $SOURCE_HOST true < /dev/null
  echo ssh -T $SSH_FLAGS $TARGET_HOST true
  ssh -T $SSH_FLAGS $TARGET_HOST true < /dev/null
}

start_cpu_collection() {
  ssh -T $SSH_FLAGS $TARGET_HOST dstat --noheaders -Tc > $TARGET_DIR/dstat.raw &
  CPU_PID=$!
  trap 'kill $(jobs -p) 2> /dev/null' EXIT # Kill childs if we exit the script
  sleep 1
}

stop_cpu_collection(){
  kill $CPU_PID
}

run_test_plan() {
  AB_COMMAND="
time ab -s $TIMEOUT_REQUEST -n $NUM_REQUESTS -c $CONCURRENT_REQUESTS
 ${ENABLE_KEEP_ALIVE:+-k}
 -g /tmp/ab_output.tsv
 ${HOST_HEADER:+-H 'Host: $HOST_HEADER'}
 ${TARGET_URL}
"
  echo "Executing: '$AB_COMMAND'"
  ssh -T $SSH_FLAGS $SOURCE_HOST $AB_COMMAND | tee $TARGET_DIR /ab_report.txt
  scp -qF ssh.config $SOURCE_HOST:/tmp/ab_output.tsv $TARGET_DIR/
}


check_remote_hosts
start_cpu_collection
run_test_plan
stop_cpu_collection

generate-md.sh

#!/bin/bash

#
# Generates graphs and Markdown for all the resulting tests from
# `execute_perf_test.sh`
#
# Given a directory with suites like this:
# ├── suite-vulcand-http-keepalive
# │   ├── 105000-conc-3500-keepalive
# │   ├── 12000-conc-400-keepalive
# │   ├── 120000-conc-4000-keepalive
# │   ├── 60000-conc-2000-keepalive
# │   ├── 75000-conc-2500-keepalive
# │   ├── 9000-conc-300-keepalive
# │   ├── 90000-conc-3000-keepalive
# ├── suite-vulcand-ssl
# │   ├── 10500-conc-350
# │   ├── 1200-conc-40
# │   ├── 12000-conc-400
# │   ├── 6000-conc-200
# │   ├── 7500-conc-250
# │   ├── 9000-conc-300
# └── suite-vulcand-ssl-keepalive
#     ├── 10500-conc-350-keepalive
#     ├── 1200-conc-40-keepalive
#     ├── 12000-conc-400-keepalive
#     ├── 6000-conc-200-keepalive
#     ├── 7500-conc-250-keepalive
#     ├── 9000-conc-300-keepalive
#
# It will create graphs and readmes for:
#  - response time plot per test (in each test in each suite)
#  - CPU per test
#  - Agregate per suite
#

generate_test_graphs() {
    datadir=$1; shift
    gnuplot \
    	-e "input_file='${datadir}/ab_output.tsv'" \
    	-e "output_file='${datadir}/sequence.jpg'" \
    	sequence.plot
    gnuplot \
    	-e "input_file='${datadir}/ab_output.tsv'" \
    	-e "output_file='${datadir}/timeseries.jpg'" \
    	timeseries.plot
    gnuplot \
    	-e "input_file='${datadir}/dstat.raw'" \
    	-e "output_file='${datadir}/cpu.jpg'" \
    	cpu.plot
}

generate_suite_markdown(){
    suite_dir=$1
    if [ -f "$suite_dir/description.txt" ]; then
	DESCRIPTION=$(<$suite_dir/description.txt)
    else
	DESCRIPTION=$suite_dir
    fi

    echo "Processing suite $suite_dir: $DESCRIPTION"

    echo "# $DESCRIPTION" > $suite_dir/README.md
    executions=$(find $suite_dir -name 'dstat.raw' | xargs -n1 dirname | sort -n)

    for i in $executions; do
    	echo "Generating graphs for $i..."
	generate_test_graphs $i
	pathname=$(basename $i)
	cat <<EOF >> $suite_dir/README.md
## $pathname

![cpu]($pathname/cpu.jpg) ![sequence]($pathname/sequence.jpg) ![timeseries]($pathname/timeseries.jpg)

\`\`\`
$(<$i/ab_report.txt)
\`\`\`

EOF
    done
}


get_metrics_from_ab_output() {

    file=$1
    label=$(basename $(dirname $(dirname $file)))

    if grep -q 'Keep-Alive requests' $file; then
	keepalive=1
    else
	keepalive=0
    fi

    total_reqs=$(awk '/Complete requests/ { print $3 }' < $file)
    failed_reqs=$(awk '/Failed requests/ { print $3 }' < $file)
    conc=$(awk '/Concurrency Level/ { print $3 }' < $file)
    mean=$(awk '/Time per request.*\(mean\)/ { print $4 }' < $file)
    p95=$(awk '/95%/ { print $2 };' < $file)
    p98=$(awk '/98%/ { print $2 };' < $file)
    rqs=$(awk '/Requests per second/ { print $4 }' < $file)


    echo -e "$label\t$keepalive\t$total_reqs\t$failed_reqs\t$conc\t$mean\t$p95\t$p98\t$rqs"
}

get_csv_from_test() {
    # Print headers
    echo -e "label\tkeepalive\ttotal_reqs\tfailed_reqs\tconc\tmean\tp95\tp98\trqs"

    for test in $@; do
	for f in $(find $test -name ab_report.txt) ; do
	    get_metrics_from_ab_output $f
	done | sort -k 3 -n
    done
}

generate_aggregate_graph() {
    test_dir=$1
    readme=$2
    if [ -f "$test_dir/description.txt" ]; then
	test_title=$(<$test_dir/description.txt)
    else
	test_title=$(basename $test_dir)
    fi
    echo "Generating aggragate graph for $test_dir"
    get_csv_from_test $test_dir > ${test_dir}/aggregate.csv
    gnuplot \
	-e "filename='${test_dir}/aggregate.csv'" \
	-e "output_file='${test_dir}/aggregate.jpg'" \
	-e "title='${test_title}'" \
	aggregate.plot

    echo "# $test_title" >> $readme
    echo "![$test_title]($(basename ${test_dir})/aggregate.jpg)" >> $readme
}

if [ ! -d "$1" ]; then
    echo "Must specify a directory with the test results"
    exit 1
fi

MAIN_README=$1/README.md
echo -n > $MAIN_README
SUITES=$(find $1 -name suite-* | sort)
for suite in $SUITES; do
    generate_suite_markdown $suite
    generate_aggregate_graph $suite $MAIN_README
done

echo "Done in $MAIN_README"

sequence.plot

# Let's output to a jpeg file
set terminal jpeg size 500,500
# This sets the aspect ratio of the graph
set size 1, 1
# The file we'll write to
set output output_file
# The graph title
set title "Benchmark testing"
# Where to place the legend/key
set key left top
# Draw gridlines oriented on the y axis
set grid y
# Label the x-axis
set xlabel 'requests'
# Label the y-axis
set ylabel "response time (ms)"
# Tell gnuplot to use tabs as the delimiter instead of spaces (default)
set datafile separator '\t'
# Plot the data
plot input_file every ::2 using 5 title 'response time' with lines
exit

test-plan-http.sh

#!/bin/sh
SUITENAME=${1:-suite-unknown}

for CONCURRENT_REQUESTS in 200 300 400 500 1000 1500 2000 2500 3000 3500 4000; do

    echo ===================================================================
    echo ===================================================================
    echo ===================================================================
    echo ===================================================================

    echo $CONCURRENT_REQUESTS
    echo ===================================================================
    echo ===================================================================
    echo ===================================================================
    echo ===================================================================

    export CONCURRENT_REQUESTS
    export NUM_REQUESTS=$(($CONCURRENT_REQUESTS * 30))

    DEPLOY_ENV=hector270715 \
    TARGET_HOST=10.129.0.72 \
    SOURCE_HOST=10.129.0.228 \
    ENABLE_KEEP_ALIVE=yes \
    TARGET_URL='http://10.129.0.72:8080/index.html' \
    TARGET_DIR=../vulcand-hipache-perf-test-results/$SUITENAME/$NUM_REQUESTS-conc-$CONCURRENT_REQUESTS \
    ./generate_perf_test.sh
done

test-plan.sh

#!/bin/sh
for CONCURRENT_REQUESTS in 10 20 40 60 80 100 150 200 250 300 350 400 450 500; do

    echo ===================================================================
    echo ===================================================================
    echo ===================================================================
    echo ===================================================================

    echo $CONCURRENT_REQUESTS
    echo ===================================================================
    echo ===================================================================
    echo ===================================================================
    echo ===================================================================

    export CONCURRENT_REQUESTS
    export NUM_REQUESTS=$(($CONCURRENT_REQUESTS * 30))

    DEPLOY_ENV=hector270715 \
    TARGET_HOST=10.129.0.72 \
    SOURCE_HOST=10.129.0.228 \
    ENABLE_KEEP_ALIVE= \
    TARGET_DIR=../vulcand-hipache-perf-test-results/$NUM_REQUESTS-conc-$CONCURRENT_REQUESTS \
    ./generate_perf_test.sh
done

timeseries.plot

# Let's output to a jpeg file
set terminal jpeg size 500,500
# This sets the aspect ratio of the graph
set size 1, 1
# The file we'll write to
set output output_file
# The graph title
set title "Benchmark testing"
# Where to place the legend/key
set key left top
# Draw gridlines oriented on the y axis
set grid y
# Specify that the x-series data is time data
set xdata time
# Specify the *input* format of the time data
set timefmt "%s"
# Specify the *output* format for the x-axis tick labels
set format x "%S"
# Label the x-axis
set xlabel 'seconds'
# Label the y-axis
set ylabel "response time (ms)"
# Tell gnuplot to use tabs as the delimiter instead of spaces (default)
set datafile separator '\t'
# Plot the data
plot input_file every ::2 using 2:5 title 'response time' with points
exit