Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Dockerized JMeter - A Distributed Load Testing Workflow

Dockerized JMeter - A Distributed Load Testing Workflow

Supported Cloud Providers:

Images used:

Pre-requisites:

  1. Docker and Docker-Machine cli installed on your host (https://docs.docker.com/installation/)

  2. You can choose either Amazon or DigitalOcean

    Amazon:

    • Access Key
    • Secret Key
    • Region
    • VPC ID
    • Subnet ID
    • Zone
    • AMI

    DigitalOcean:

    • Access Token
  3. JMeter test plan on your host (http://jmeter.apache.org/usermanual/build-web-test-plan.html)

Steps:

  1. Provision machine for JMeter Master (Non-gui mode)

    For AWS:

    $ ./launch_jmeter_master_aws
    

    For DigitalOcean:

    $ ./launch_jmeter_master_do
    

    For local machine:

    $ ./launch_jmeter_master_local
    
  2. Provision machine for JMeter Slave (Server mode)

    Optional parameter - indicate number of servers to provision, value between 1 to 20. Default to 1 if no parameter is given.

    For AWS:

    $ ./launch_jmeter_slave_aws 2
    

    For DigitalOcean:

    $ ./launch_jmeter_slave_do 2
    
  3. Copy test plan (jmx file) to jmeter-master machine inside /load_tests

    Connect to jmeter-master machine and create /load_tests directory

    $ eval "$(docker-machine env --swarm jmeter-master)"
    $ docker-machine ssh jmeter-master
    

    For AWS:

    $ sudo mkdir /load_tests && sudo chown ubuntu:ubuntu /load_tests
    

    For DigitalOcean:

    $ mkdir /load_tests
    

    For local machine:

    $ sudo mkdir /load_tests && sudo chown docker:docker /load_tests
    

    Then go back to host

    $ exit
    

    Copy test plan from host to jmeter-master machine

    Syntax:

    docker-machine scp [path to test directory] [machine name]:/load_tests
    

    Example:

    $ docker-machine scp -r /home/user/docker-jmeter-master/load_tests/my_test jmeter-master:/load_tests
    

    Then go back to host

    $ exit
    
  4. Run JMeter load test - execute the following commands, replace values if necessary before executing

    Get JMeter master machine ip

    $ IP=$(docker-machine ip jmeter-master)
    

    Replace the value with the output from ./launch_jmeter_slave - must be comma separated jmeter slave ip addresses

    $ REMOTE_HOSTS=""
    

    Parent directory for the test plan

    $ TEST_DIR="my_test"
    

    Test plan without file extension

    $ TEST_PLAN="test-plan"
    

    Run the JMeter master non-gui and perform load test

    $ docker run \
        --detach \
        --publish 1099:1099 \
        --volume /load_tests/$TEST_DIR:/load_tests/$TEST_DIR \
        --env TEST_DIR=$TEST_DIR \
        --env TEST_PLAN=$TEST_PLAN \
        --env IP=$IP \
        --env REMOTE_HOSTS=$REMOTE_HOSTS \
        --env constraint:type==master \
        hhcordero/docker-jmeter-client
    

    To monitor output, follow the logs:

    Syntax:

    docker logs -f [container name]
    

    Example:

    $ docker logs -f jmeter-master/tender_feynman
    
  5. Save the result, look for the .jtl file inside jmeter-master /load_tests/$TEST_DIR

    Syntax:

    docker-machine scp [machine name]:/load_tests/[test dir]/[test plan result] [path to test directory]
    

    Example:

    $ docker-machine scp jmeter-master:/load_tests/${TEST_DIR}/${TEST_PLAN}.jtl /home/user/docker-jmeter-master/load_tests/my_test/.
    
#!/bin/bash
# replace env file to use, default to digitalocean
app_env='app_do.env'
# Generate token - use swarm for service discovery
TOKEN=$(docker run --rm swarm create)
echo "export TOKEN=$TOKEN" >> ./$app_env
# Provision new machine
echo -e "
Provision machine for JMeter (Master):
docker-machine create
--driver virtualbox
--engine-label type=master
--swarm
--swarm-master
--swarm-discovery token://$TOKEN
jmeter-master"
docker-machine create \
--driver virtualbox \
--engine-label type=master \
--swarm \
--swarm-master \
--swarm-discovery token://$TOKEN \
jmeter-master
echo "To manage swarm, make sure to run: eval \"\$(docker-machine env --swarm jmeter-master)\""
exit
#!/bin/bash
export TOKEN=[auto-populate by running launch_jmeter_master_aws]
export AWS_ACCESS_KEY_ID=[access key here]
export AWS_SECRET_ACCESS_KEY=[secret key here]
export AWS_DEFAULT_REGION=[region here ie. ap-southeast-1]
export AWS_VPC_ID=[vpc id here ie. vpc-xxxxxxxx]
export AWS_SUBNET_ID=[subnet id here ie. subnet-xxxxxxxx]
export AWS_ZONE=[zone here ie. a]
export AWS_AMI=[ami id here ie. ami-b899a2ea]
#!/bin/bash
# Generate token - use swarm for service discovery
TOKEN=$(docker run --rm swarm create)
sed -i "/export TOKEN=/c\export TOKEN=$TOKEN" ./app_aws.env
# Initialise env variables
source ./app_aws.env
# Provision new machine
echo -e "
Provision machine for JMeter (Master):
docker-machine --debug create
--driver amazonec2
--amazonec2-access-key $AWS_ACCESS_KEY_ID
--amazonec2-secret-key $AWS_SECRET_ACCESS_KEY
--amazonec2-region $AWS_DEFAULT_REGION
--amazonec2-vpc-id $AWS_VPC_ID
--amazonec2-subnet-id $AWS_SUBNET_ID
--amazonec2-zone $AWS_ZONE
--amazonec2-ami $AWS_AMI
--engine-label type=master
--swarm
--swarm-master
--swarm-discovery token://$TOKEN
jmeter-master"
docker-machine create \
--driver amazonec2 \
--amazonec2-access-key $AWS_ACCESS_KEY_ID \
--amazonec2-secret-key $AWS_SECRET_ACCESS_KEY \
--amazonec2-region $AWS_DEFAULT_REGION \
--amazonec2-vpc-id $AWS_VPC_ID \
--amazonec2-subnet-id $AWS_SUBNET_ID \
--amazonec2-zone $AWS_ZONE \
--amazonec2-ami $AWS_AMI \
--engine-label type=master \
--swarm \
--swarm-master \
--swarm-discovery token://$TOKEN \
jmeter-master
echo "To manage swarm, make sure to run: eval \"\$(docker-machine env --swarm jmeter-master)\""
exit
#!/bin/bash
# Default to 1 if no parameter is given
count=${1-1}
# Max of 20 machines only, change accordingly
max=20
if ! [[ $count =~ ^[0-9]+$ ]] || ! [[ "$count" -gt 0 && "$count" -le "$max" ]]; then
echo "[ERROR] Parameter is not a positive integer (must be 1 to $max)." >&2; exit 1
fi
if [ "$count" -gt "$max" ]; then
count=$max
fi
# Initialise env variables
source ./app_aws.env
iterator=1
while [ "$iterator" -le "$count" ]; do
# Provision new machine
echo -e "
Provision machine for JMeter (Slave):
docker-machine create
--driver amazonec2
--amazonec2-access-key $AWS_ACCESS_KEY_ID
--amazonec2-secret-key $AWS_SECRET_ACCESS_KEY
--amazonec2-region $AWS_DEFAULT_REGION
--amazonec2-vpc-id $AWS_VPC_ID
--amazonec2-subnet-id $AWS_SUBNET_ID
--amazonec2-zone $AWS_ZONE
--amazonec2-ami $AWS_AMI
--engine-label type=slave
--swarm
--swarm-discovery token://$TOKEN
jmeter-slave-$iterator"
docker-machine create \
--driver amazonec2 \
--amazonec2-access-key $AWS_ACCESS_KEY_ID \
--amazonec2-secret-key $AWS_SECRET_ACCESS_KEY \
--amazonec2-region $AWS_DEFAULT_REGION \
--amazonec2-vpc-id $AWS_VPC_ID \
--amazonec2-subnet-id $AWS_SUBNET_ID \
--amazonec2-zone $AWS_ZONE \
--amazonec2-ami $AWS_AMI \
--engine-label type=slave \
--swarm \
--swarm-discovery token://$TOKEN \
jmeter-slave-$iterator
# Set env variables to point to newly created machine
eval "$(docker-machine env jmeter-slave-$iterator)"
# Get public ip
ip=$(docker-machine ip jmeter-slave-$iterator)
# Run jmeter on new machine
echo -e "
Run JMeter in Server Mode:
docker run
--detach
--publish 1099:1099
--env IP=$ip
hhcordero/docker-jmeter-server"
docker run \
--detach \
--publish 1099:1099 \
--env IP=$ip \
hhcordero/docker-jmeter-server
# Concatenate slave ip addresses
server_ips+="$ip,"
let "iterator += 1"
done
echo "Slave IP's, for use in JMeter Master: $(echo $server_ips | sed 's/,*$//')"
exit
#!/bin/bash
export TOKEN=[auto-populate by running launch_jmeter_master_do]
export DIGITALOCEAN_ACCESS_TOKEN=[access token here]
export DIGITALOCEAN_REGION=[region here]
export DIGITALOCEAN_SIZE=[memory here]
#!/bin/bash
# Generate token - use swarm for service discovery
TOKEN=$(docker run --rm swarm create)
sed -i "/export TOKEN=/c\export TOKEN=$TOKEN" ./app_do.env
# Initialise env variables
source ./app_do.env
# Provision new machine
echo -e "
Provision machine for JMeter (Master):
docker-machine create
--driver digitalocean
--digitalocean-access-token $DIGITALOCEAN_ACCESS_TOKEN
--digitalocean-region $DIGITALOCEAN_REGION
--digitalocean-size $DIGITALOCEAN_SIZE
--engine-label type=master
--swarm
--swarm-master
--swarm-discovery token://$TOKEN
jmeter-master"
docker-machine create \
--driver digitalocean \
--digitalocean-access-token $DIGITALOCEAN_ACCESS_TOKEN \
--digitalocean-region $DIGITALOCEAN_REGION \
--digitalocean-size $DIGITALOCEAN_SIZE \
--engine-label type=master \
--swarm \
--swarm-master \
--swarm-discovery token://$TOKEN \
jmeter-master
echo "To manage swarm, make sure to run: eval \"\$(docker-machine env --swarm jmeter-master)\""
exit
#!/bin/bash
# Default to 1 if no parameter is given
count=${1-1}
# Max of 20 machines only, change accordingly
max=20
if ! [[ $count =~ ^[0-9]+$ ]] || ! [[ "$count" -gt 0 && "$count" -le "$max" ]]; then
echo "[ERROR] Parameter is not a positive integer (must be 1 to $max)." >&2; exit 1
fi
if [ "$count" -gt "$max" ]; then
count=$max
fi
# Initialise env variables
source ./app_do.env
iterator=1
while [ "$iterator" -le "$count" ]; do
# Provision new machine
echo -e "
Provision machine for JMeter (Slave):
docker-machine create
--driver digitalocean
--digitalocean-access-token $DIGITALOCEAN_ACCESS_TOKEN
--digitalocean-region $DIGITALOCEAN_REGION
--digitalocean-size $DIGITALOCEAN_SIZE
--engine-label type=slave
--swarm
--swarm-discovery token://$TOKEN
jmeter-slave-$iterator"
docker-machine create \
--driver digitalocean \
--digitalocean-access-token $DIGITALOCEAN_ACCESS_TOKEN \
--digitalocean-region $DIGITALOCEAN_REGION \
--digitalocean-size $DIGITALOCEAN_SIZE \
--engine-label type=slave \
--swarm \
--swarm-discovery token://$TOKEN \
jmeter-slave-$iterator
# Set env variables to point to newly created machine
eval "$(docker-machine env jmeter-slave-$iterator)"
# Get public ip
ip=$(docker-machine ip jmeter-slave-$iterator)
# Run jmeter on new machine
echo -e "
Run JMeter in Server Mode:
docker run
--detach
--publish 1099:1099
--env IP=$ip
hhcordero/docker-jmeter-server"
docker run \
--detach \
--publish 1099:1099 \
--env IP=$ip \
hhcordero/docker-jmeter-server
# Concatenate slave ip addresses
server_ips+="$ip,"
let "iterator += 1"
done
echo "Slave IP's, for use in JMeter Master: $(echo $server_ips | sed 's/,*$//')"
exit
@burzum619

This comment has been minimized.

Copy link

burzum619 commented May 31, 2016

Excellent project, nice work. When starting the client containers (and executing the test), I seem to be having issues with the server and client nodes communicating. Does an SSH tunnel need to be configured between these machines?

016/05/31 20:48:02 INFO - jmeter.engine.DistributedRunner: Configuring remote engine: 54.xxx.yyy.zzz 2016/05/31 20:50:09 ERROR - jmeter.engine.DistributedRunner: Failed to create engine at 54.xxx.yyy.zzz java.rmi.ConnectException: Connection refused to host: 54.xxx.yyy.zzz; nested exception is: java.net.ConnectException: Operation timed out at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:341) at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at java.rmi.Naming.lookup(Naming.java:101) at org.apache.jmeter.engine.ClientJMeterEngine.getEngine(ClientJMeterEngine.java:54) at org.apache.jmeter.engine.ClientJMeterEngine.<init>(ClientJMeterEngine.java:67) at org.apache.jmeter.engine.DistributedRunner.createEngine(DistributedRunner.java:237) at org.apache.jmeter.engine.DistributedRunner.getClientEngine(DistributedRunner.java:213) at org.apache.jmeter.engine.DistributedRunner.init(DistributedRunner.java:91) at org.apache.jmeter.JMeter.runNonGui(JMeter.java:814) at org.apache.jmeter.JMeter.startNonGui(JMeter.java:737) at org.apache.jmeter.JMeter.start(JMeter.java:395) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.jmeter.NewDriver.main(NewDriver.java:264) Caused by: java.net.ConnectException: Operation timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.<init>(Socket.java:425) at java.net.Socket.<init>(Socket.java:208) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613) ... 18 more

@hhcordero

This comment has been minimized.

Copy link
Owner Author

hhcordero commented Jul 8, 2016

Seems like connectivity issue, did you set the --env IP=$IP parameter when you run the jmeter-client (slave) container?

@burzum619

This comment has been minimized.

Copy link

burzum619 commented Jul 13, 2016

It turned out my security group did not have 1099 inbound port open. Once I took care of that, it worked perfect. Thank you for your work, great job!

@jesperrasmussen

This comment has been minimized.

Copy link

jesperrasmussen commented Aug 12, 2016

Hi Hector!

I'm trying to build a small shellscript, to interactively take a location of a test and run it over a number of servers - to automatically run the setup you're describing.

I noticed that the actual execution of the test might have an issue - you write the following:

$ docker run
--detach
--publish 1099:1099
--volume /load_tests/$TEST_DIR:/load_tests/$TEST_DIR
--env TEST_DIR=$TEST_DIR
--env TEST_PLAN=$TEST_PLAN
--env IP=$IP
--env REMOTE_HOSTS=$REMOTE_HOSTS
--env constraint:type==master
hhcordero/docker-jmeter-master

However, the image hhcordero/docker-jmeter-master is not available on Hub I think?

My guess would be that it should be the "docker-jmeter-client" image, but when I change to using that - Docker tells me:

docker: Error response from daemon: Error: image library/pre-create:latest not found.

Do you know what the cause of this is? Has anything changed regarding the images? :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.