Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
ara benchmarks raw data

Some roughly formatted benchmark data.

ansible without ara enabled

time ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml
# ...
Executed in   11.36 secs   fish           external 
   usr time   36.54 secs    0.00 micros   36.54 secs 
   sys time   16.28 secs  583.00 micros   16.28 secs 

time ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e benchmark_host_count=100
# ...
Executed in   41.32 secs   fish           external 
   usr time  147.68 secs  528.00 micros  147.68 secs 
   sys time   70.04 secs  112.00 micros   70.04 secs 

time ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e benchmark_host_count=200 -e benchmark_task_count=200
# ...
Executed in  360.67 secs   fish           external 
   usr time   20.83 mins    0.00 micros   20.83 mins 
   sys time   10.84 mins  360.00 micros   10.84 mins 

ara 1.5.3 with sqlite and ANSIBLE_FORKS=50

With the default offline API client against localhost (no running API server):

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.3"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.3" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.3" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="client:offline"
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 310 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T11:30:12.313649-04:00 | 00:29:25.955674 |
| 309 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T11:24:58.979017-04:00 | 00:03:22.571607 |
| 308 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T11:22:05.943304-04:00 | 00:00:52.728591 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

With export ARA_API_CLIENT=http against localhost: ara-manage runserver

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,version:1.5.3"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,version:1.5.3" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,version:1.5.3" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="client:http"
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 314 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T12:25:58.707298-04:00 | 00:17:24.335043 |
| 313 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T12:23:34.120747-04:00 | 00:01:37.972033 |
| 311 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T12:14:12.684240-04:00 | 00:00:30.101029 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

With export ARA_API_CLIENT=http against localhost: gunicorn --access-logfile - --workers=9 --bind 127.0.0.1:8000 ara.server.wsgi

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.3"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.3" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.3" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="backend:gunicorn"
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 322 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T14:39:49.979617-04:00 | 00:13:47.462996 |
| 316 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T13:14:17.244041-04:00 | 00:01:09.821960 |
| 315 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T13:12:09.157927-04:00 | 00:00:20.606147 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

ara 1.5.1 with sqlite and ANSIBLE_FORKS=50

With the default offline API client against localhost (no running API server):

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.1"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.1" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.1" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="version:1.5.1"
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 328 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T16:08:19.132407-04:00 | 00:30:22.647469 |
| 327 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T16:04:40.701513-04:00 | 00:03:31.300917 |
| 326 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T16:03:37.630983-04:00 | 00:00:56.030091 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

With export ARA_API_CLIENT=http against localhost: ara-manage runserver:

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,backend:django,version:1.5.1"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,backend:django,version:1.5.1" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,backend:django,version:1.5.1" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="backend:django"
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 325 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T15:00:09.476913-04:00 | 00:17:28.111219 |
| 324 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T14:58:21.897583-04:00 | 00:01:39.595000 |
| 323 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T14:57:42.707173-04:00 | 00:00:32.229069 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

With export ARA_API_CLIENT=http against localhost: gunicorn --access-logfile - --workers=9 --bind 127.0.0.1:8000 ara.server.wsgi:

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.1"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.1" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.1" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="backend:gunicorn" --limit 3
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 334 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T17:22:05.171776-04:00 | 00:21:38.136877 |
| 333 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T17:19:38.631320-04:00 | 00:02:19.691135 |
| 332 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T17:18:55.759791-04:00 | 00:00:36.259820 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

ara 0.16.8 with sqlite and ANSIBLE_FORKS=50

Testing 0.16.8 is also interesting science and also because it doesn't have an API and the callback talks directly to the database.

Screenshot from 2020-10-31 19-39-10

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e benchmark_host_count=200 -e benchmark_host_tasks=200
# ara playbook list
+--------------------------------------+-------------------------------------------------------------------------------+---------------------+----------+----------+-----------------+
| ID                                   | Path                                                                          | Time Start          | Duration | Complete | Ansible Version |
+--------------------------------------+-------------------------------------------------------------------------------+---------------------+----------+----------+-----------------+
| 0f448e3a-f64f-4928-9f5a-251f7eb4372d | /home/dmsimard/dev/git/ansible-community/ara/tests/integration/benchmark.yaml | 2020-10-31 19:06:50 | 0:00:28  | True     | 2.10.2          |
| af6c0d50-eb05-47f8-bba0-e5e3ef609eb9 | /home/dmsimard/dev/git/ansible-community/ara/tests/integration/benchmark.yaml | 2020-10-31 19:07:35 | 0:01:56  | True     | 2.10.2          |
| 20e3ab2e-8296-471a-a3c6-1bc56931fdf4 | /home/dmsimard/dev/git/ansible-community/ara/tests/integration/benchmark.yaml | 2020-10-31 19:14:23 | 0:19:05  | True     | 2.10.2          |
+--------------------------------------+-------------------------------------------------------------------------------+---------------------+----------+----------+-----------------+
#!/bin/bash
BENCHMARK_CMD=${BENCHMARK_CMD:-ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml}
ARA_VERSION=${ARA_VERSION:-1.5.3}
ANSIBLE_VERSION=${ANSIBLE_VERSION:-2.10.2}
export ANSIBLE_FORKS=${ANSIBLE_FORKS:-50}
export ARA_DEFAULT_LABELS=${ARA_DEFAULT_LABELS:-benchmark}
export ARA_API_CLIENT=${ARA_API_CLIENT:-offline}
export ARA_API_SERVER=${ARA_API_SERVER:-http://127.0.0.1:8000}
# Create a virtualenv if none exists
[[ ! -d /tmp/ara-venv ]] && python3 -m venv /tmp/ara-venv
source /tmp/ara-venv/bin/activate
# pip install ara[server]==${ARA_VERSION} ansible==${ANSIBLE_VERSION}
pip install ara[server]==${ARA_VERSION} ansible
export ANSIBLE_CALLBACK_PLUGINS=$(python3 -m ara.setup.callback_plugins)
for hosts in 25 100 200; do
for tasks in 25 100 200; do
$BENCHMARK_CMD -e benchmark_host_count=${hosts} -e benchmark_task_count=${tasks}
done
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.