Skip to content

Instantly share code, notes, and snippets.

@dmsimard

dmsimard/ara-benchmarks.md

Last active November 1, 2020 01:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dmsimard/ab6d53ac2edf3b4ab6076e20dcad8fe9 to your computer and use it in GitHub Desktop.
Save dmsimard/ab6d53ac2edf3b4ab6076e20dcad8fe9 to your computer and use it in GitHub Desktop.
Download ZIP
ara benchmarks raw data
Raw
ara-benchmarks.md

Some roughly formatted benchmark data.

ansible without ara enabled

time ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml
# ...
Executed in   11.36 secs   fish           external 
   usr time   36.54 secs    0.00 micros   36.54 secs 
   sys time   16.28 secs  583.00 micros   16.28 secs 

time ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e benchmark_host_count=100
# ...
Executed in   41.32 secs   fish           external 
   usr time  147.68 secs  528.00 micros  147.68 secs 
   sys time   70.04 secs  112.00 micros   70.04 secs 

time ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e benchmark_host_count=200 -e benchmark_task_count=200
# ...
Executed in  360.67 secs   fish           external 
   usr time   20.83 mins    0.00 micros   20.83 mins 
   sys time   10.84 mins  360.00 micros   10.84 mins

ara 1.5.3 with sqlite and ANSIBLE_FORKS=50

With the default offline API client against localhost (no running API server):

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.3"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.3" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.3" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="client:offline"
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 310 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T11:30:12.313649-04:00 | 00:29:25.955674 |
| 309 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T11:24:58.979017-04:00 | 00:03:22.571607 |
| 308 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T11:22:05.943304-04:00 | 00:00:52.728591 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

With export ARA_API_CLIENT=http against localhost: ara-manage runserver

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,version:1.5.3"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,version:1.5.3" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,version:1.5.3" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="client:http"
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 314 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T12:25:58.707298-04:00 | 00:17:24.335043 |
| 313 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T12:23:34.120747-04:00 | 00:01:37.972033 |
| 311 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T12:14:12.684240-04:00 | 00:00:30.101029 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

With export ARA_API_CLIENT=http against localhost: gunicorn --access-logfile - --workers=9 --bind 127.0.0.1:8000 ara.server.wsgi

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.3"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.3" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.3" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="backend:gunicorn"
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 322 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T14:39:49.979617-04:00 | 00:13:47.462996 |
| 316 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T13:14:17.244041-04:00 | 00:01:09.821960 |
| 315 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T13:12:09.157927-04:00 | 00:00:20.606147 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

ara 1.5.1 with sqlite and ANSIBLE_FORKS=50

With the default offline API client against localhost (no running API server):

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.1"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.1" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:offline,version:1.5.1" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="version:1.5.1"
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 328 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T16:08:19.132407-04:00 | 00:30:22.647469 |
| 327 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T16:04:40.701513-04:00 | 00:03:31.300917 |
| 326 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T16:03:37.630983-04:00 | 00:00:56.030091 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

With export ARA_API_CLIENT=http against localhost: ara-manage runserver:

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,backend:django,version:1.5.1"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,backend:django,version:1.5.1" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=client:http,backend:django,version:1.5.1" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="backend:django"
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 325 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T15:00:09.476913-04:00 | 00:17:28.111219 |
| 324 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T14:58:21.897583-04:00 | 00:01:39.595000 |
| 323 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T14:57:42.707173-04:00 | 00:00:32.229069 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

With export ARA_API_CLIENT=http against localhost: gunicorn --access-logfile - --workers=9 --bind 127.0.0.1:8000 ara.server.wsgi:

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.1"
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.1" -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e "ara_playbook_labels=backend:gunicorn,client:http,version:1.5.1" -e benchmark_host_count=200 -e benchmark_task_count=200
# ara playbook list --label="backend:gunicorn" --limit 3
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
|  id | status    | path                                    | tasks | results | hosts | started                          | duration        |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+
| 334 | completed | ...ara/tests/integration/benchmark.yaml |   202 |   40201 |   201 | 2020-10-31T17:22:05.171776-04:00 | 00:21:38.136877 |
| 333 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    5101 |   101 | 2020-10-31T17:19:38.631320-04:00 | 00:02:19.691135 |
| 332 | completed | ...ara/tests/integration/benchmark.yaml |    52 |    1276 |    26 | 2020-10-31T17:18:55.759791-04:00 | 00:00:36.259820 |
+-----+-----------+-----------------------------------------+-------+---------+-------+----------------------------------+-----------------+

ara 0.16.8 with sqlite and ANSIBLE_FORKS=50

Testing 0.16.8 is also interesting science and also because it doesn't have an API and the callback talks directly to the database.

Screenshot from 2020-10-31 19-39-10

# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e benchmark_host_count=100
# ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml -e benchmark_host_count=200 -e benchmark_host_tasks=200
# ara playbook list
+--------------------------------------+-------------------------------------------------------------------------------+---------------------+----------+----------+-----------------+
| ID                                   | Path                                                                          | Time Start          | Duration | Complete | Ansible Version |
+--------------------------------------+-------------------------------------------------------------------------------+---------------------+----------+----------+-----------------+
| 0f448e3a-f64f-4928-9f5a-251f7eb4372d | /home/dmsimard/dev/git/ansible-community/ara/tests/integration/benchmark.yaml | 2020-10-31 19:06:50 | 0:00:28  | True     | 2.10.2          |
| af6c0d50-eb05-47f8-bba0-e5e3ef609eb9 | /home/dmsimard/dev/git/ansible-community/ara/tests/integration/benchmark.yaml | 2020-10-31 19:07:35 | 0:01:56  | True     | 2.10.2          |
| 20e3ab2e-8296-471a-a3c6-1bc56931fdf4 | /home/dmsimard/dev/git/ansible-community/ara/tests/integration/benchmark.yaml | 2020-10-31 19:14:23 | 0:19:05  | True     | 2.10.2          |
+--------------------------------------+-------------------------------------------------------------------------------+---------------------+----------+----------+-----------------+
Raw
wrapper.sh
#!/bin/bash
BENCHMARK_CMD=${BENCHMARK_CMD:-ansible-playbook -i 'localhost,' -c local tests/integration/benchmark.yaml}
ARA_VERSION=${ARA_VERSION:-1.5.3}
ANSIBLE_VERSION=${ANSIBLE_VERSION:-2.10.2}
export ANSIBLE_FORKS=${ANSIBLE_FORKS:-50}
export ARA_DEFAULT_LABELS=${ARA_DEFAULT_LABELS:-benchmark}
export ARA_API_CLIENT=${ARA_API_CLIENT:-offline}
export ARA_API_SERVER=${ARA_API_SERVER:-http://127.0.0.1:8000}
# Create a virtualenv if none exists
[[ ! -d /tmp/ara-venv ]] && python3 -m venv /tmp/ara-venv
source /tmp/ara-venv/bin/activate
# pip install ara[server]==${ARA_VERSION} ansible==${ANSIBLE_VERSION}
pip install ara[server]==${ARA_VERSION} ansible
export ANSIBLE_CALLBACK_PLUGINS=$(python3 -m ara.setup.callback_plugins)
for hosts in 25 100 200; do
for tasks in 25 100 200; do
$BENCHMARK_CMD -e benchmark_host_count=${hosts} -e benchmark_task_count=${tasks}
done
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment