Keystone performance benchmarking

auth.json

{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
		    "user": {
			"domain": {
			    "id": "default"
			},
			"password": "password",
			"name": "admin"
		    }
            }
        },
        "scope": {
            "project": {
                "domain": {
                    "id": "default"
                },
                "name": "admin"
            }
        }
    }
}

authenticate.py

import os

from keystoneclient.v3 import client


HOST = os.environ.get('HOST', '23.253.157.126')
KEYSTONE_ENDPOINT = os.environ.get(
    'KEYSTONE_ENDPOINT', 'http://%s:35357/' % HOST)


project_scoped = client.Client(
    username='admin',
    password='password',
    project_name='admin',
    auth_url=KEYSTONE_ENDPOINT + 'v3')
print('%s' % project_scoped.auth_token)

benchmark.sh

set -e

export HOST=23.253.157.242
export DB_HOST=23.253.157.126

echo "Ensuring Keystone is bootstrapped with the expected credentials..."
python bootstrap.py

echo "Truncating the keystone.token table..."
ssh root@$DB_HOST "mysql keystone -e 'truncate token;'"

echo "Creating a token to run benchmarks with..."
ADMIN_TOKEN=`python authenticate.py`
SUBJECT_TOKEN=`python authenticate.py`
echo "Admin token: $ADMIN_TOKEN"
echo "Subject token: $SUBJECT_TOKEN"

echo "Warming up Apache..."
ab -c 100 -n 1000 -T 'application/json' http://$HOST:35357/ > /dev/null 2>&1

echo "Benchmarking token creation..."
ab -r -c 1 -n 200 -p auth.json -T 'application/json' http://$HOST:35357/v3/auth/tokens > latest_create_token
if grep -q 'Non-2xx' latest_create_token; then
echo 'Non-2xx return codes! Aborting.'
fi

echo "Benchmarking token validation..."
ab -r -c 1 -n 10000 -T 'application/json' -H "X-Auth-Token: $ADMIN_TOKEN" -H "X-Subject-Token: $SUBJECT_TOKEN" http://$HOST:35357/v3/auth/tokens > latest_validate_token
if grep -q 'Non-2xx' latest_validate_token; then
echo 'Non-2xx return codes! Aborting.'
fi

echo "Benchmarking token creation concurrently..."
ab -r -c 100 -n 2000 -p auth.json -T 'application/json' http://$HOST:35357/v3/auth/tokens > latest_create_token_concurrent
if grep -q 'Non-2xx' latest_create_token_concurrent; then
  echo 'WARNING: Non-2xx return codes!'
fi

echo "Benchmarking token validation concurrency..."
ab -r -c 100 -n 100000 -T 'application/json' -H "X-Auth-Token: $ADMIN_TOKEN" -H "X-Subject-Token: $SUBJECT_TOKEN" http://$HOST:35357/v3/auth/tokens > latest_validate_token_concurrent
if grep -q 'Non-2xx' latest_validate_token_concurrent; then
  echo 'WARNING: Non-2xx return codes!'
fi

bootstrap.py

import os

from keystoneclient.v3 import client


HOST = os.environ.get('HOST', '192.168.111.222')
KEYSTONE_ENDPOINT = os.environ.get(
    'KEYSTONE_ENDPOINT', 'http://%s:35357/' % HOST)


c = client.Client(
    token='ADMIN',
    endpoint=KEYSTONE_ENDPOINT + 'v3')

domain = c.domains.get('default')

try:
    role = c.roles.create(name='admin')
except Exception:
    role = c.roles.list(name='admin')[0]

try:
    group = c.groups.create(domain=domain, name='admin')
except Exception:
    group = c.groups.list(domain=domain, name='admin')[0]

c.roles.grant(group=group, domain=domain, role=role)

try:
    project = c.projects.create(domain=domain, name='admin')
except Exception:
    project = c.projects.list(domain=domain, name='admin')[0]

c.roles.grant(group=group, project=project, role=role)

password = 'password'
try:
    user = c.users.create(
        domain=domain, name='admin', password=password)
except Exception:
    user = c.users.list(domain=domain, name='admin')[0]

c.users.add_to_group(user=user, group=group)

services = c.services.list(name='Keystone', type='identity')
if services:
    service = services[0]
else:
    service = c.services.create(
        name='Keystone', type='identity')

endpoints = c.endpoints.list(service=service)
public_endpoints = [e for e in endpoints if e.interface == 'public']
admin_endpoints = [e for e in endpoints if e.interface == 'admin']

if not public_endpoints:
    public_endpoint = c.endpoints.create(
        service=service,
        interface='public',
        url=KEYSTONE_ENDPOINT + 'v3')

if not admin_endpoints:
    admin_endpoint = c.endpoints.create(
        service=service,
        interface='admin',
        url=KEYSTONE_ENDPOINT + 'v3')

unscoped = client.Client(
    username='admin',
    password='password',
    auth_url=KEYSTONE_ENDPOINT + 'v3')
print('Unscoped token: %s' % unscoped.auth_token)

project_scoped = client.Client(
    username='admin',
    password='password',
    project_name='admin',
    auth_url=KEYSTONE_ENDPOINT + 'v3')
print('Project-scoped token: %s' % project_scoped.auth_token)

domain_scoped = client.Client(
    username='admin',
    password='password',
    domain_name='Default',
    auth_url=KEYSTONE_ENDPOINT + 'v3')
print('Domain-scoped token: %s' % domain_scoped.auth_token)

latest_create_token

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 23.253.157.242 (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        23.253.157.242
Server Port:            35357

Document Path:          /v3/auth/tokens
Document Length:        862 bytes

Concurrency Level:      1
Time taken for tests:   14.359 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      246400 bytes
Total POSTed:           123400
HTML transferred:       172400 bytes
Requests per second:    13.93 [#/sec] (mean)
Time per request:       71.795 [ms] (mean)
Time per request:       71.795 [ms] (mean, across all concurrent requests)
Transfer rate:          16.76 [Kbytes/sec] received
                        8.39 kb/s sent
                        25.15 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    45   72  58.8     52     460
Waiting:       43   69  58.7     49     460
Total:         45   72  58.8     52     460

Percentage of the requests served within a certain time (ms)
  50%     52
  66%     53
  75%     54
  80%     60
  90%    142
  95%    156
  98%    339
  99%    447
 100%    460 (longest request)

latest_create_token_concurrent

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 23.253.157.242 (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        23.253.157.242
Server Port:            35357

Document Path:          /v3/auth/tokens
Document Length:        862 bytes

Concurrency Level:      100
Time taken for tests:   8.479 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      2464000 bytes
Total POSTed:           1234000
HTML transferred:       1724000 bytes
Requests per second:    235.88 [#/sec] (mean)
Time per request:       423.953 [ms] (mean)
Time per request:       4.240 [ms] (mean, across all concurrent requests)
Transfer rate:          283.79 [Kbytes/sec] received
                        142.12 kb/s sent
                        425.91 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    87  415 221.7    359    1308
Waiting:       81  400 216.7    346    1303
Total:         87  415 221.7    360    1308

Percentage of the requests served within a certain time (ms)
  50%    360
  66%    454
  75%    524
  80%    572
  90%    739
  95%    879
  98%   1023
  99%   1120
 100%   1308 (longest request)

latest_validate_token

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 23.253.157.242 (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        23.253.157.242
Server Port:            35357

Document Path:          /v3/auth/tokens
Document Length:        921 bytes

Concurrency Level:      1
Time taken for tests:   55.558 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      12860000 bytes
HTML transferred:       9210000 bytes
Requests per second:    179.99 [#/sec] (mean)
Time per request:       5.556 [ms] (mean)
Time per request:       5.556 [ms] (mean, across all concurrent requests)
Transfer rate:          226.05 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     4    5   0.9      5      62
Waiting:        3    4   0.9      4      58
Total:          4    6   0.9      6      62

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      6
  75%      6
  80%      6
  90%      6
  95%      6
  98%      6
  99%      7
 100%     62 (longest request)

latest_validate_token_concurrent

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 23.253.157.242 (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        23.253.157.242
Server Port:            35357

Document Path:          /v3/auth/tokens
Document Length:        921 bytes

Concurrency Level:      100
Time taken for tests:   52.634 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      128600000 bytes
HTML transferred:       92100000 bytes
Requests per second:    1899.93 [#/sec] (mean)
Time per request:       52.634 [ms] (mean)
Time per request:       0.526 [ms] (mean, across all concurrent requests)
Transfer rate:          2386.04 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:     5   53  68.5     14     374
Waiting:        4   49  68.0     11     373
Total:          5   53  68.5     14     374

Percentage of the requests served within a certain time (ms)
  50%     14
  66%     27
  75%     68
  80%    142
  90%    177
  95%    193
  98%    210
  99%    223
 100%    374 (longest request)

original_create_token

$ ab -c 1 -n 200 -p auth.json -T 'application/json' http://23.253.157.126:35357/v3/auth/tokens
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 23.253.157.126 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests


Server Software:        Apache/2.2.22
Server Hostname:        23.253.157.126
Server Port:            35357

Document Path:          /v3/auth/tokens
Document Length:        1270 bytes

Concurrency Level:      1
Time taken for tests:   66.744 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      301400 bytes
Total POSTed:           123400
HTML transferred:       254000 bytes
Requests per second:    3.00 [#/sec] (mean)
Time per request:       333.719 [ms] (mean)
Time per request:       333.719 [ms] (mean, across all concurrent requests)
Transfer rate:          4.41 [Kbytes/sec] received
                        1.81 kb/s sent
                        6.22 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   281  333  44.1    325     423
Waiting:      281  333  44.1    325     423
Total:        281  334  44.1    325     423

Percentage of the requests served within a certain time (ms)
  50%    325
  66%    335
  75%    389
  80%    390
  90%    392
  95%    393
  98%    397
  99%    412
 100%    423 (longest request)

original_validate_token

$ ab -c 1 -n 10000 -T 'application/json' -H 'X-Auth-Token: ADMIN' -H 'X-Subject-Token: 9314ae88cfe04ef3a2fbb74358c62dcb' http://23.253.157.126:35357/v3/auth/tokens
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 23.253.157.126 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        Apache/2.2.22
Server Hostname:        23.253.157.126
Server Port:            35357

Document Path:          /v3/auth/tokens
Document Length:        1654 bytes

Concurrency Level:      1
Time taken for tests:   30.411 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      18860000 bytes
HTML transferred:       16540000 bytes
Requests per second:    328.83 [#/sec] (mean)
Time per request:       3.041 [ms] (mean)
Time per request:       3.041 [ms] (mean, across all concurrent requests)
Transfer rate:          605.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0      26
Processing:     2    3   1.0      3      42
Waiting:        2    3   0.9      3      42
Total:          3    3   1.1      3      42

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      3
  75%      3
  80%      3
  90%      3
  95%      3
  98%      4
  99%      6
 100%     42 (longest request)

requirements.txt

python-keystoneclient

Hi there, thanks for the v. useful benchmarking code!

Just one q: I've been testing this on an openstack-ansible-deployed Pike environment and Keystone was rejecting all my tokens validation requests ('Invalid user token') until I removed the linefeeds from the request headers (eg from "X-Auth-Token: $ADMIN_TOKEN\n"). Is this an indication of a config issue? I'm trying to understand cause of subpar Keystone validation performance (~500ms single-threaded on lightly loaded metal controllers).