Skip to content

Instantly share code, notes, and snippets.

@dolph
Last active June 7, 2018 07:06
  • Star 3 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save dolph/02c6d37f49596b3f4298 to your computer and use it in GitHub Desktop.
Keystone performance benchmarking
{
"auth": {
"identity": {
"methods": [
"password"
],
"password": {
"user": {
"domain": {
"id": "default"
},
"password": "password",
"name": "admin"
}
}
},
"scope": {
"project": {
"domain": {
"id": "default"
},
"name": "admin"
}
}
}
}
import os
from keystoneclient.v3 import client
HOST = os.environ.get('HOST', '23.253.157.126')
KEYSTONE_ENDPOINT = os.environ.get(
'KEYSTONE_ENDPOINT', 'http://%s:35357/' % HOST)
project_scoped = client.Client(
username='admin',
password='password',
project_name='admin',
auth_url=KEYSTONE_ENDPOINT + 'v3')
print('%s' % project_scoped.auth_token)
set -e
export HOST=23.253.157.242
export DB_HOST=23.253.157.126
echo "Ensuring Keystone is bootstrapped with the expected credentials..."
python bootstrap.py
echo "Truncating the keystone.token table..."
ssh root@$DB_HOST "mysql keystone -e 'truncate token;'"
echo "Creating a token to run benchmarks with..."
ADMIN_TOKEN=`python authenticate.py`
SUBJECT_TOKEN=`python authenticate.py`
echo "Admin token: $ADMIN_TOKEN"
echo "Subject token: $SUBJECT_TOKEN"
echo "Warming up Apache..."
ab -c 100 -n 1000 -T 'application/json' http://$HOST:35357/ > /dev/null 2>&1
echo "Benchmarking token creation..."
ab -r -c 1 -n 200 -p auth.json -T 'application/json' http://$HOST:35357/v3/auth/tokens > create_token
if ! grep -q 'Failed requests: 0' create_token; then
echo 'WARNING: Failed requests!'
fi
git diff --color create_token | grep --fixed-strings ' [ms] (mean)' || true
echo "Benchmarking token validation..."
ab -r -c 1 -n 10000 -T 'application/json' -H "X-Auth-Token: $ADMIN_TOKEN\n" -H "X-Subject-Token: $SUBJECT_TOKEN\n" http://$HOST:35357/v3/auth/tokens > validate_token
if ! grep -q 'Failed requests: 0' validate_token; then
echo 'WARNING: Failed requests!'
fi
git diff --color validate_token | grep --fixed-strings ' [ms] (mean)' || true
echo "Benchmarking token creation concurrently..."
ab -r -c 100 -n 2000 -p auth.json -T 'application/json' http://$HOST:35357/v3/auth/tokens > create_token_concurrent
if ! grep -q 'Failed requests: 0' create_token_concurrent; then
echo 'WARNING: Failed requests!'
fi
git diff --color create_token_concurrent | grep --fixed-strings ' [#/sec] (mean)' || true
echo "Benchmarking token validation concurrency..."
ab -r -c 100 -n 100000 -T 'application/json' -H "X-Auth-Token: $ADMIN_TOKEN" -H "X-Subject-Token: $SUBJECT_TOKEN" http://$HOST:35357/v3/auth/tokens > validate_token_concurrent
if ! grep -q 'Failed requests: 0' validate_token_concurrent; then
echo 'WARNING: Failed requests!'
fi
git diff --color validate_token_concurrent | grep --fixed-strings ' [#/sec] (mean)' || true
import os
from keystoneclient.v3 import client
HOST = os.environ.get('HOST', '192.168.111.222')
KEYSTONE_ENDPOINT = os.environ.get(
'KEYSTONE_ENDPOINT', 'http://%s:35357/' % HOST)
c = client.Client(
token='ADMIN',
endpoint=KEYSTONE_ENDPOINT + 'v3')
domain = c.domains.get('default')
try:
role = c.roles.create(name='admin')
except Exception:
role = c.roles.list(name='admin')[0]
try:
group = c.groups.create(domain=domain, name='admin')
except Exception:
group = c.groups.list(domain=domain, name='admin')[0]
c.roles.grant(group=group, domain=domain, role=role)
try:
project = c.projects.create(domain=domain, name='admin')
except Exception:
project = c.projects.list(domain=domain, name='admin')[0]
c.roles.grant(group=group, project=project, role=role)
password = 'password'
try:
user = c.users.create(
domain=domain, name='admin', password=password)
except Exception:
user = c.users.list(domain=domain, name='admin')[0]
c.users.add_to_group(user=user, group=group)
services = c.services.list(name='Keystone', type='identity')
if services:
service = services[0]
else:
service = c.services.create(
name='Keystone', type='identity')
endpoints = c.endpoints.list(service=service)
public_endpoints = [e for e in endpoints if e.interface == 'public']
admin_endpoints = [e for e in endpoints if e.interface == 'admin']
if not public_endpoints:
public_endpoint = c.endpoints.create(
service=service,
interface='public',
url=KEYSTONE_ENDPOINT + 'v3')
if not admin_endpoints:
admin_endpoint = c.endpoints.create(
service=service,
interface='admin',
url=KEYSTONE_ENDPOINT + 'v3')
unscoped = client.Client(
username='admin',
password='password',
auth_url=KEYSTONE_ENDPOINT + 'v3')
print('Unscoped token: %s' % unscoped.auth_token)
project_scoped = client.Client(
username='admin',
password='password',
project_name='admin',
auth_url=KEYSTONE_ENDPOINT + 'v3')
print('Project-scoped token: %s' % project_scoped.auth_token)
domain_scoped = client.Client(
username='admin',
password='password',
domain_name='Default',
auth_url=KEYSTONE_ENDPOINT + 'v3')
print('Domain-scoped token: %s' % domain_scoped.auth_token)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 23.253.157.242 (be patient)
Server Software: Apache/2.2.22
Server Hostname: 23.253.157.242
Server Port: 35357
Document Path: /v3/auth/tokens
Document Length: 886 bytes
Concurrency Level: 1
Time taken for tests: 14.596 seconds
Complete requests: 200
Failed requests: 0
Write errors: 0
Total transferred: 553836 bytes
Total POSTed: 123400
HTML transferred: 177200 bytes
Requests per second: 13.70 [#/sec] (mean)
Time per request: 72.981 [ms] (mean)
Time per request: 72.981 [ms] (mean, across all concurrent requests)
Transfer rate: 37.05 [Kbytes/sec] received
8.26 kb/s sent
45.31 kb/s total
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 62 73 19.4 70 284
Waiting: 59 70 19.4 66 280
Total: 62 73 19.4 70 284
Percentage of the requests served within a certain time (ms)
50% 70
66% 71
75% 71
80% 72
90% 80
95% 82
98% 123
99% 214
100% 284 (longest request)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 23.253.157.242 (be patient)
Server Software: Apache/2.2.22
Server Hostname: 23.253.157.242
Server Port: 35357
Document Path: /v3/auth/tokens
Document Length: 886 bytes
Concurrency Level: 100
Time taken for tests: 11.235 seconds
Complete requests: 2000
Failed requests: 0
Write errors: 0
Total transferred: 5538728 bytes
Total POSTed: 1234000
HTML transferred: 1772000 bytes
Requests per second: 178.02 [#/sec] (mean)
Time per request: 561.726 [ms] (mean)
Time per request: 5.617 [ms] (mean, across all concurrent requests)
Transfer rate: 481.46 [Kbytes/sec] received
107.27 kb/s sent
588.72 kb/s total
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 133 555 278.6 477 1862
Waiting: 125 542 274.6 463 1852
Total: 134 555 278.6 477 1862
Percentage of the requests served within a certain time (ms)
50% 477
66% 596
75% 707
80% 771
90% 928
95% 1107
98% 1328
99% 1465
100% 1862 (longest request)
python-keystoneclient
This is ApacheBench, Version 2.3 <$Revision$>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 23.253.157.242 (be patient)
Server Software: Apache/2.2.22
Server Hostname: 23.253.157.242
Server Port: 35357
Document Path: /v3/auth/tokens
Document Length: 886 bytes
Concurrency Level: 1
Time taken for tests: 62.751 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 27620000 bytes
HTML transferred: 8860000 bytes
Requests per second: 159.36 [#/sec] (mean)
Time per request: 6.275 [ms] (mean)
Time per request: 6.275 [ms] (mean, across all concurrent requests)
Transfer rate: 429.84 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 5 6 0.6 6 39
Waiting: 4 5 0.6 5 38
Total: 5 6 0.6 6 39
Percentage of the requests served within a certain time (ms)
50% 6
66% 6
75% 6
80% 6
90% 7
95% 7
98% 7
99% 7
100% 39 (longest request)
This is ApacheBench, Version 2.3 <$Revision$>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 23.253.157.242 (be patient)
Server Software: Apache/2.2.22
Server Hostname: 23.253.157.242
Server Port: 35357
Document Path: /v3/auth/tokens
Document Length: 886 bytes
Concurrency Level: 100
Time taken for tests: 59.193 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 276200000 bytes
HTML transferred: 88600000 bytes
Requests per second: 1689.40 [#/sec] (mean)
Time per request: 59.193 [ms] (mean)
Time per request: 0.592 [ms] (mean, across all concurrent requests)
Transfer rate: 4556.75 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 6 59 79.7 17 470
Waiting: 4 55 79.4 14 469
Total: 6 59 79.7 17 470
Percentage of the requests served within a certain time (ms)
50% 17
66% 30
75% 56
80% 120
90% 212
95% 230
98% 252
99% 267
100% 470 (longest request)
@sxc731
Copy link

sxc731 commented Jan 8, 2018

Hi there, thanks for the v. useful benchmarking code!

Just one q: I've been testing this on an openstack-ansible-deployed Pike environment and Keystone was rejecting all my tokens validation requests ('Invalid user token') until I removed the linefeeds from the request headers (eg from "X-Auth-Token: $ADMIN_TOKEN\n"). Is this an indication of a config issue? I'm trying to understand cause of subpar Keystone validation performance (~500ms single-threaded on lightly loaded metal controllers).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment