Newton Rolling Upgrades

README.md

Problem Context

We had an operator come to us with an interesting issue regarding rolling upgrades from Mitaka to Newton. The conversation was logged in #openstack-keystone. This notepad is my attempt to recreate the issue and document whatever I find.

Setup

I documented most of the installation process while testing the migration for encrypted credentials. I'm going to essentially use the same steps to install keystone. The steps after setup are specific to creating test data for credentials. Here we are going to document the upgrade process separately since it doesn't have to be specific to credentials.

Once we have Mitaka up and running - we can go ahead and populate it with some data:

openstack project create accounting
openstack project create marketing
openstack project create sales
openstack project create engineering

openstack user create --project accounting --password password --enable bob
openstack user create --project accounting --password password --enable susan
openstack user create --project marketing --password password --enable jeff
openstack user create --project marketing --password password --enable jerry
openstack user create --project marketing --password password --enable jim
openstack user create --project marketing --password password --enable jane
openstack user create --project sales --password password --enable tom
openstack user create --project engineering --password password --enable jill
openstack user create --project engineering --password password --enable jack
openstack user create --project engineering --password password --enable george
openstack user create --project engineering --password password --enable sarah

openstack role create member

openstack role add --user bob --project accounting member
openstack role add --user susan --project accounting member
openstack role add --user jeff --project marketing member
openstack role add --user jerry --project marketing member
openstack role add --user jim --project marketing member
openstack role add --user jane --project marketing member
openstack role add --user tom --project sales member
openstack role add --user jill --project engineering member
openstack role add --user jack --project engineering member
openstack role add --user george --project engineering member
openstack role add --user sarah --project engineering member

Upgrade

I'm going to follow the steps as they are in keystone's documentation.

(1) Make a backup of your database.

sudo mysqldump keystone > keystone-backup-1477326267.sql

(2) Stop the keystone processes on the first node.

Since I'm using an eventlet process for testing (keystone-wsgi-admin), this just consists of stopping that process.

(mitaka) ubuntu@upgrade-1:~$ ps aux | grep keystone
ubuntu    51020  0.0  0.0  12944  1088 pts/1    S+   16:40   0:00 grep --color=auto keystone

(3) Upgrade your first node to the next release, but do not start any keystone processes.

git clone -b stable/newton https://github.com/openstack/keystone keystone-newton

Lets create a new virtualenv for our Newton source:

virtualenv newton
source newton/bin/activate
pip install -e keystone-newton/
pip install python-memcached osprofiler mysql-python

(4) Update your configuration files on the first node (/etc/keystone/)

(5) Run keystone-manage doctor on the first node

(newton) ubuntu@upgrade-1:~$ keystone-manage doctor
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Checking for caching disabled...
Checking for caching enabled without a backend...
Checking for keys in credential fernet key repository...
Checking for unique key repositories...
Checking for usability of credential fernet key repository...
Checking for database connection is not SQLite...
Checking for comma in SAML private key file path...
Checking for comma in SAML public certificate path...
Checking for LDAP group members are ids disabled...
Checking for LDAP user enabled emulation dn ignored...
Checking for LDAP user enabled emulation use group config ignored...
Checking for invalid password regular expression...
Checking for minimum password age should be less than password expires days...
Checking for password regular expression description not set...
Checking for unreasonable max token size...
Checking for keys in Fernet key repository...
Checking for usability of Fernet key repository...

(6) Run keystone-manage db_sync --expand on the first node

(newton) ubuntu@upgrade-1:~$ keystone-manage db_sync --expand
2016-10-24 16:43:35.031 51249 INFO migrate.versioning.api [-] 97 -> 98... 
2016-10-24 16:43:35.037 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.037 51249 INFO migrate.versioning.api [-] 98 -> 99... 
2016-10-24 16:43:35.042 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.042 51249 INFO migrate.versioning.api [-] 99 -> 100... 
2016-10-24 16:43:35.047 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.047 51249 INFO migrate.versioning.api [-] 100 -> 101... 
2016-10-24 16:43:35.055 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.055 51249 INFO migrate.versioning.api [-] 101 -> 102... 
2016-10-24 16:43:35.068 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.068 51249 INFO migrate.versioning.api [-] 102 -> 103... 
2016-10-24 16:43:35.092 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.093 51249 INFO migrate.versioning.api [-] 103 -> 104... 
2016-10-24 16:43:35.100 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.100 51249 INFO migrate.versioning.api [-] 104 -> 105... 
2016-10-24 16:43:35.170 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.170 51249 INFO migrate.versioning.api [-] 105 -> 106... 
2016-10-24 16:43:35.208 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.208 51249 INFO migrate.versioning.api [-] 106 -> 107... 
2016-10-24 16:43:35.264 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.265 51249 INFO migrate.versioning.api [-] 107 -> 108... 
2016-10-24 16:43:35.352 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.353 51249 INFO migrate.versioning.api [-] 108 -> 109... 
2016-10-24 16:43:35.396 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.423 51249 INFO migrate.versioning.api [-] 0 -> 1... 
2016-10-24 16:43:35.428 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.428 51249 INFO migrate.versioning.api [-] 1 -> 2... 
2016-10-24 16:43:35.433 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.434 51249 INFO migrate.versioning.api [-] 2 -> 3... 
2016-10-24 16:43:35.490 51249 INFO migrate.versioning.api [-] done
2016-10-24 16:43:35.490 51249 INFO migrate.versioning.api [-] 3 -> 4... 
2016-10-24 16:43:35.495 51249 INFO migrate.versioning.api [-] done

I'm going to make sure I can still get a token from upgrade-2:

(osc) ubuntu@upgrade-2:~$ openstack token issue
+------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2016-10-24T17:44:59+0000         |
| id         | b6cf77d762f64466b8b3bc0eccb62a24 |
| project_id | 5ec043fb4c6a4077b258bc300ec8feb9 |
| user_id    | fa1003125a1c4f089654ba6881b2eeda |
+------------+----------------------------------+

(7) Run keystone-manage db_sync --migrate on the first node

(newton) ubuntu@upgrade-1:~$ keystone-manage db_sync --migrate
2016-10-24 16:45:29.417 51379 INFO migrate.versioning.api [-] 0 -> 1... 
2016-10-24 16:45:29.423 51379 INFO migrate.versioning.api [-] done
2016-10-24 16:45:29.423 51379 INFO migrate.versioning.api [-] 1 -> 2... 
2016-10-24 16:45:29.429 51379 INFO migrate.versioning.api [-] done
2016-10-24 16:45:29.429 51379 INFO migrate.versioning.api [-] 2 -> 3... 
2016-10-24 16:45:29.440 51379 INFO migrate.versioning.api [-] done
2016-10-24 16:45:29.440 51379 INFO migrate.versioning.api [-] 3 -> 4... 
2016-10-24 16:45:29.446 51379 INFO migrate.versioning.api [-] done

The migration seems to be fine - let's get another token from upgrade-2:

(osc) ubuntu@upgrade-2:~$ openstack token issue
+------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2016-10-24T17:46:23+0000         |
| id         | 1013d1d3ca294a9eab0670b7d7003247 |
| project_id | 5ec043fb4c6a4077b258bc300ec8feb9 |
| user_id    | fa1003125a1c4f089654ba6881b2eeda |
+------------+----------------------------------+

At this point I should be able to safely start the keystone-wsgi-admin process on upgrade-1. We can verify keystone is running by grabbing a token from upgrade-1.

(osc) ubuntu@upgrade-1:~$ openstack token issue
+------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2016-10-24T17:48:30+0000         |
| id         | aec62a358e564e52b1db913d772add21 |
| project_id | 5ec043fb4c6a4077b258bc300ec8feb9 |
| user_id    | fa1003125a1c4f089654ba6881b2eeda |
+------------+----------------------------------+

(8) Update your configuration files (/etc/keystone/) on all nodes

(9) Upgrade all keystone nodes to the next release

I did the same virtualenv steps for Newton, already documented in step 3:

git clone -b stable/newton https://github.com/openstack/keystone keystone-newton

Lets create a new virtualenv for our Newton source:

virtualenv newton
source newton/bin/activate
pip install -e keystone-newton/
pip install python-memcached osprofiler mysql-python
keystone-wsgi-admin -p 35357

Just double checking that I can still get a token from upgrade-2 after Newton code is running:

(osc) ubuntu@upgrade-2:~$ openstack token issue
+------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2016-10-24T17:49:00+0000         |
| id         | c1cc3560c37447a896afab69e5aef00a |
| project_id | 5ec043fb4c6a4077b258bc300ec8feb9 |
| user_id    | fa1003125a1c4f089654ba6881b2eeda |
+------------+----------------------------------+

(10) Run keystone-manage db_sync --contract

(newton) ubuntu@upgrade-1:~$ keystone-manage db_sync --contract
2016-10-24 16:50:46.563 51789 INFO migrate.versioning.api [-] 0 -> 1... 
2016-10-24 16:50:46.570 51789 INFO migrate.versioning.api [-] done
2016-10-24 16:50:46.570 51789 INFO migrate.versioning.api [-] 1 -> 2... 
2016-10-24 16:50:46.651 51789 INFO migrate.versioning.api [-] done
2016-10-24 16:50:46.651 51789 INFO migrate.versioning.api [-] 2 -> 3... 
2016-10-24 16:50:46.709 51789 INFO migrate.versioning.api [-] done
2016-10-24 16:50:46.709 51789 INFO migrate.versioning.api [-] 3 -> 4... 
2016-10-24 16:50:46.818 51789 INFO migrate.versioning.api [-] done

At this point there are no Mitaka nodes running and the database has been upgraded to Newton's schema. I can verify information on both nodes.

Information from upgrade-1

(osc) ubuntu@upgrade-1:~$ openstack user list
+----------------------------------+--------+
| ID                               | Name   |
+----------------------------------+--------+
| 0ef34fc7e35444c6a328bdb302da1287 | jill   |
| 1db6450319c240de870f3b47832c0fb8 | jane   |
| 23ed3d0d478142dfa05c06c98d9437b7 | jeff   |
| 26771559a7e04c64a64de8c6e9d5612e | jack   |
| 37329ad64ae146e8bcc2555fcb744ed1 | susan  |
| 4db2489ad11e426f9fd5da9492eb484b | tom    |
| 5f0a2ee52cda4c7cbe97626e8919fc1d | sarah  |
| 65755898e12f42cba58a94160ea49ac6 | jerry  |
| 904aa66ae6dc4f959d655eb2df36f54b | bob    |
| a45bf02a17694282be21611cc9d2d2ce | george |
| b53b8ccdf1174a20884e73e482bdbea7 | jim    |
| fa1003125a1c4f089654ba6881b2eeda | admin  |
+----------------------------------+--------+
(osc) ubuntu@upgrade-1:~$ openstack project list
+----------------------------------+-------------+
| ID                               | Name        |
+----------------------------------+-------------+
| 52be18361d504e4bac727fdad7c0972b | sales       |
| 5ec043fb4c6a4077b258bc300ec8feb9 | admin       |
| 630714b2a6404bc6bf88518054a4a418 | accounting  |
| a51dd026cb4f45d6a716e1afbff4242c | marketing   |
| f409f94f48bd46bdb7265d51e5bea386 | engineering |
+----------------------------------+-------------+
(osc) ubuntu@upgrade-1:~$ openstack role assignment list
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| Role                             | User                             | Group | Project                          | Domain | Inherited |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| c85c300f3bfb4ecab094a15e48a23918 | 0ef34fc7e35444c6a328bdb302da1287 |       | f409f94f48bd46bdb7265d51e5bea386 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 1db6450319c240de870f3b47832c0fb8 |       | a51dd026cb4f45d6a716e1afbff4242c |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 23ed3d0d478142dfa05c06c98d9437b7 |       | a51dd026cb4f45d6a716e1afbff4242c |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 26771559a7e04c64a64de8c6e9d5612e |       | f409f94f48bd46bdb7265d51e5bea386 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 37329ad64ae146e8bcc2555fcb744ed1 |       | 630714b2a6404bc6bf88518054a4a418 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 4db2489ad11e426f9fd5da9492eb484b |       | 52be18361d504e4bac727fdad7c0972b |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 5f0a2ee52cda4c7cbe97626e8919fc1d |       | f409f94f48bd46bdb7265d51e5bea386 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 65755898e12f42cba58a94160ea49ac6 |       | a51dd026cb4f45d6a716e1afbff4242c |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 904aa66ae6dc4f959d655eb2df36f54b |       | 630714b2a6404bc6bf88518054a4a418 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | a45bf02a17694282be21611cc9d2d2ce |       | f409f94f48bd46bdb7265d51e5bea386 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | b53b8ccdf1174a20884e73e482bdbea7 |       | a51dd026cb4f45d6a716e1afbff4242c |        | False     |
| b9c158cf7f9f4b9f82ffbb15c3fbb6c8 | fa1003125a1c4f089654ba6881b2eeda |       | 5ec043fb4c6a4077b258bc300ec8feb9 |        | False     |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
(osc) ubuntu@upgrade-1:~$ openstack token issue
+------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2016-10-24T17:54:38+0000         |
| id         | 25fd2bdcb71a4be09c58f545f1d5ab11 |
| project_id | 5ec043fb4c6a4077b258bc300ec8feb9 |
| user_id    | fa1003125a1c4f089654ba6881b2eeda |
+------------+----------------------------------+

Information from upgrade-2

(osc) ubuntu@upgrade-2:~$ openstack user list
+----------------------------------+--------+
| ID                               | Name   |
+----------------------------------+--------+
| 0ef34fc7e35444c6a328bdb302da1287 | jill   |
| 1db6450319c240de870f3b47832c0fb8 | jane   |
| 23ed3d0d478142dfa05c06c98d9437b7 | jeff   |
| 26771559a7e04c64a64de8c6e9d5612e | jack   |
| 37329ad64ae146e8bcc2555fcb744ed1 | susan  |
| 4db2489ad11e426f9fd5da9492eb484b | tom    |
| 5f0a2ee52cda4c7cbe97626e8919fc1d | sarah  |
| 65755898e12f42cba58a94160ea49ac6 | jerry  |
| 904aa66ae6dc4f959d655eb2df36f54b | bob    |
| a45bf02a17694282be21611cc9d2d2ce | george |
| b53b8ccdf1174a20884e73e482bdbea7 | jim    |
| fa1003125a1c4f089654ba6881b2eeda | admin  |
+----------------------------------+--------+
(osc) ubuntu@upgrade-2:~$ openstack project list
+----------------------------------+-------------+
| ID                               | Name        |
+----------------------------------+-------------+
| 52be18361d504e4bac727fdad7c0972b | sales       |
| 5ec043fb4c6a4077b258bc300ec8feb9 | admin       |
| 630714b2a6404bc6bf88518054a4a418 | accounting  |
| a51dd026cb4f45d6a716e1afbff4242c | marketing   |
| f409f94f48bd46bdb7265d51e5bea386 | engineering |
+----------------------------------+-------------+
(osc) ubuntu@upgrade-2:~$ openstack role assignment list
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| Role                             | User                             | Group | Project                          | Domain | Inherited |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| c85c300f3bfb4ecab094a15e48a23918 | 0ef34fc7e35444c6a328bdb302da1287 |       | f409f94f48bd46bdb7265d51e5bea386 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 1db6450319c240de870f3b47832c0fb8 |       | a51dd026cb4f45d6a716e1afbff4242c |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 23ed3d0d478142dfa05c06c98d9437b7 |       | a51dd026cb4f45d6a716e1afbff4242c |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 26771559a7e04c64a64de8c6e9d5612e |       | f409f94f48bd46bdb7265d51e5bea386 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 37329ad64ae146e8bcc2555fcb744ed1 |       | 630714b2a6404bc6bf88518054a4a418 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 4db2489ad11e426f9fd5da9492eb484b |       | 52be18361d504e4bac727fdad7c0972b |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 5f0a2ee52cda4c7cbe97626e8919fc1d |       | f409f94f48bd46bdb7265d51e5bea386 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 65755898e12f42cba58a94160ea49ac6 |       | a51dd026cb4f45d6a716e1afbff4242c |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | 904aa66ae6dc4f959d655eb2df36f54b |       | 630714b2a6404bc6bf88518054a4a418 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | a45bf02a17694282be21611cc9d2d2ce |       | f409f94f48bd46bdb7265d51e5bea386 |        | False     |
| c85c300f3bfb4ecab094a15e48a23918 | b53b8ccdf1174a20884e73e482bdbea7 |       | a51dd026cb4f45d6a716e1afbff4242c |        | False     |
| b9c158cf7f9f4b9f82ffbb15c3fbb6c8 | fa1003125a1c4f089654ba6881b2eeda |       | 5ec043fb4c6a4077b258bc300ec8feb9 |        | False     |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
(osc) ubuntu@upgrade-2:~$ openstack token issue
+------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2016-10-24T17:54:41+0000         |
| id         | 11a0f0f98c1d48039a31a1d0f346bd5d |
| project_id | 5ec043fb4c6a4077b258bc300ec8feb9 |
| user_id    | fa1003125a1c4f089654ba6881b2eeda |
+------------+----------------------------------+

adminrc

export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_PROJECT_NAME=admin
export OS_DEFAULT_DOMAIN=default
export OS_AUTH_URL=http://localhost:35357/v3/
export OS_IDENTITY_API_VERSION=3

keystone-1.conf

[DEFAULT]
debug=True
verbose=True
# We don't really use two endpoints, as we're only deploying v3
public_endpoint=http://localhost:35357/
admin_endpoint=http://localhost:35357/
fatal_deprecations = true
# truncate collection responses for performance
list_limit=20
strict_password_check=True
max_token_size = 32
fatal_deprecations=false
policy_file=policy.json
read_only_mode=true

[eventlet_server]
admin_port = 35357

[assignment]
driver=sql

[identity]
driver=sql

[auth]
methods=password,token

[cache]
enabled=true
backend=dogpile.cache.memcached
expiration_time=600
backend_argument=url:127.0.0.1:11211

[catalog]
driver=sql

[database]
connection=mysql://keystone:keystone@127.0.0.1/keystone
mysql_sql_mode = TRADITIONAL

[paste_deploy]
config_file=/etc/keystone/paste.ini

[token]
expiration=3600
provider = uuid

[trust]
enabled=true

keystone-2.conf

[DEFAULT]
debug=True
verbose=True
# We don't really use two endpoints, as we're only deploying v3
public_endpoint=http://localhost:35357/
admin_endpoint=http://localhost:35357/
fatal_deprecations = true
# truncate collection responses for performance
list_limit=20
strict_password_check=True
max_token_size = 32
fatal_deprecations=false
policy_file=policy.json
read_only_mode=true

[eventlet_server]
admin_port = 35357

[assignment]
driver=sql

[identity]
driver=sql

[auth]
methods=password,token

[cache]
enabled=true
backend=dogpile.cache.memcached
expiration_time=600
backend_argument=url:127.0.0.1:11211

[catalog]
driver=sql

[database]
connection=mysql://keystone:keystone@104.130.31.14/keystone
mysql_sql_mode = TRADITIONAL

[paste_deploy]
config_file=/etc/keystone/paste.ini

[token]
expiration=3600
provider = uuid

[trust]
enabled=true

paste.ini

# Keystone PasteDeploy configuration file.

[filter:debug]
use = egg:oslo.middleware#debug

[filter:request_id]
use = egg:oslo.middleware#request_id

[filter:build_auth_context]
use = egg:keystone#build_auth_context

[filter:token_auth]
use = egg:keystone#token_auth

[filter:admin_token_auth]
# This is deprecated in the M release and will be removed in the O release.
# Use `keystone-manage bootstrap` and remove this from the pipelines below.
use = egg:keystone#admin_token_auth

[filter:json_body]
use = egg:keystone#json_body

[filter:cors]
use = egg:oslo.middleware#cors
oslo_config_project = keystone

[filter:ec2_extension]
use = egg:keystone#ec2_extension

[filter:ec2_extension_v3]
use = egg:keystone#ec2_extension_v3

[filter:s3_extension]
use = egg:keystone#s3_extension

[filter:url_normalize]
use = egg:keystone#url_normalize

[filter:sizelimit]
use = egg:oslo.middleware#sizelimit

[app:public_service]
use = egg:keystone#public_service

[app:service_v3]
use = egg:keystone#service_v3

[app:admin_service]
use = egg:keystone#admin_service

[pipeline:public_api]
# The last item in this pipeline must be public_service or an equivalent
# application. It cannot be a filter.
pipeline = cors sizelimit osprofiler url_normalize request_id admin_token_auth build_auth_context token_auth json_body ec2_extension public_service

[pipeline:admin_api]
# The last item in this pipeline must be admin_service or an equivalent
# application. It cannot be a filter.
pipeline = cors sizelimit osprofiler url_normalize request_id admin_token_auth build_auth_context token_auth json_body ec2_extension s3_extension admin_service

[pipeline:api_v3]
# The last item in this pipeline must be service_v3 or an equivalent
# application. It cannot be a filter.
pipeline = cors sizelimit osprofiler url_normalize request_id admin_token_auth build_auth_context token_auth json_body ec2_extension_v3 s3_extension service_v3

[app:public_version_service]
use = egg:keystone#public_version_service

[app:admin_version_service]
use = egg:keystone#admin_version_service

[pipeline:public_version_api]
pipeline = cors sizelimit osprofiler url_normalize public_version_service

[pipeline:admin_version_api]
pipeline = cors sizelimit osprofiler url_normalize admin_version_service

[composite:main]
use = egg:Paste#urlmap
/v2.0 = public_api
/v3 = api_v3
/ = public_version_api

[composite:admin]
use = egg:Paste#urlmap
/v2.0 = admin_api
/v3 = api_v3
/ = admin_version_api

[filter:osprofiler]
paste.filter_factory = osprofiler.web:WsgiMiddleware.factory

policy.json

{
    "admin_required": "role:admin or is_admin:1",
    "service_role": "role:service",
    "service_or_admin": "rule:admin_required or rule:service_role",
    "owner" : "user_id:%(user_id)s",
    "admin_or_owner": "rule:admin_required or rule:owner",
    "token_subject": "user_id:%(target.token.user_id)s",
    "admin_or_token_subject": "rule:admin_required or rule:token_subject",
    "service_admin_or_token_subject": "rule:service_or_admin or rule:token_subject",

    "default": "rule:admin_required",

    "identity:get_region": "",
    "identity:list_regions": "",
    "identity:create_region": "rule:admin_required",
    "identity:update_region": "rule:admin_required",
    "identity:delete_region": "rule:admin_required",

    "identity:get_service": "rule:admin_required",
    "identity:list_services": "rule:admin_required",
    "identity:create_service": "rule:admin_required",
    "identity:update_service": "rule:admin_required",
    "identity:delete_service": "rule:admin_required",

    "identity:get_endpoint": "rule:admin_required",
    "identity:list_endpoints": "rule:admin_required",
    "identity:create_endpoint": "rule:admin_required",
    "identity:update_endpoint": "rule:admin_required",
    "identity:delete_endpoint": "rule:admin_required",

    "identity:get_domain": "rule:admin_required",
    "identity:list_domains": "rule:admin_required",
    "identity:create_domain": "rule:admin_required",
    "identity:update_domain": "rule:admin_required",
    "identity:delete_domain": "rule:admin_required",

    "identity:get_project": "rule:admin_required",
    "identity:list_projects": "rule:admin_required",
    "identity:list_user_projects": "rule:admin_or_owner",
    "identity:create_project": "rule:admin_required",
    "identity:update_project": "rule:admin_required",
    "identity:delete_project": "rule:admin_required",

    "identity:get_user": "rule:admin_required",
    "identity:list_users": "rule:admin_required",
    "identity:create_user": "rule:admin_required",
    "identity:update_user": "rule:admin_required",
    "identity:delete_user": "rule:admin_required",
    "identity:change_password": "rule:admin_or_owner",

    "identity:get_group": "rule:admin_required",
    "identity:list_groups": "rule:admin_required",
    "identity:list_groups_for_user": "rule:admin_or_owner",
    "identity:create_group": "rule:admin_required",
    "identity:update_group": "rule:admin_required",
    "identity:delete_group": "rule:admin_required",
    "identity:list_users_in_group": "rule:admin_required",
    "identity:remove_user_from_group": "rule:admin_required",
    "identity:check_user_in_group": "rule:admin_required",
    "identity:add_user_to_group": "rule:admin_required",

    "identity:get_credential": "rule:admin_required",
    "identity:list_credentials": "rule:admin_required",
    "identity:create_credential": "rule:admin_required",
    "identity:update_credential": "rule:admin_required",
    "identity:delete_credential": "rule:admin_required",

    "identity:ec2_get_credential": "rule:admin_required or (rule:owner and user_id:%(target.credential.user_id)s)",
    "identity:ec2_list_credentials": "rule:admin_or_owner",
    "identity:ec2_create_credential": "rule:admin_or_owner",
    "identity:ec2_delete_credential": "rule:admin_required or (rule:owner and user_id:%(target.credential.user_id)s)",

    "identity:get_role": "rule:admin_required",
    "identity:list_roles": "rule:admin_required",
    "identity:create_role": "rule:admin_required",
    "identity:update_role": "rule:admin_required",
    "identity:delete_role": "rule:admin_required",

    "identity:check_grant": "rule:admin_required",
    "identity:list_grants": "rule:admin_required",
    "identity:create_grant": "rule:admin_required",
    "identity:revoke_grant": "rule:admin_required",

    "identity:list_role_assignments": "rule:admin_required",

    "identity:get_policy": "rule:admin_required",
    "identity:list_policies": "rule:admin_required",
    "identity:create_policy": "rule:admin_required",
    "identity:update_policy": "rule:admin_required",
    "identity:delete_policy": "rule:admin_required",

    "identity:check_token": "rule:admin_or_token_subject",
    "identity:validate_token": "rule:service_admin_or_token_subject",
    "identity:validate_token_head": "rule:service_or_admin",
    "identity:revocation_list": "rule:service_or_admin",
    "identity:revoke_token": "rule:admin_or_token_subject",

    "identity:create_trust": "user_id:%(trust.trustor_user_id)s",
    "identity:list_trusts": "",
    "identity:list_roles_for_trust": "",
    "identity:get_role_for_trust": "",
    "identity:delete_trust": "",

    "identity:create_consumer": "rule:admin_required",
    "identity:get_consumer": "rule:admin_required",
    "identity:list_consumers": "rule:admin_required",
    "identity:delete_consumer": "rule:admin_required",
    "identity:update_consumer": "rule:admin_required",

    "identity:authorize_request_token": "rule:admin_required",
    "identity:list_access_token_roles": "rule:admin_required",
    "identity:get_access_token_role": "rule:admin_required",
    "identity:list_access_tokens": "rule:admin_required",
    "identity:get_access_token": "rule:admin_required",
    "identity:delete_access_token": "rule:admin_required",

    "identity:list_projects_for_endpoint": "rule:admin_required",
    "identity:add_endpoint_to_project": "rule:admin_required",
    "identity:check_endpoint_in_project": "rule:admin_required",
    "identity:list_endpoints_for_project": "rule:admin_required",
    "identity:remove_endpoint_from_project": "rule:admin_required",

    "identity:create_endpoint_group": "rule:admin_required",
    "identity:list_endpoint_groups": "rule:admin_required",
    "identity:get_endpoint_group": "rule:admin_required",
    "identity:update_endpoint_group": "rule:admin_required",
    "identity:delete_endpoint_group": "rule:admin_required",
    "identity:list_projects_associated_with_endpoint_group": "rule:admin_required",
    "identity:list_endpoints_associated_with_endpoint_group": "rule:admin_required",
    "identity:get_endpoint_group_in_project": "rule:admin_required",
    "identity:list_endpoint_groups_for_project": "rule:admin_required",
    "identity:add_endpoint_group_to_project": "rule:admin_required",
    "identity:remove_endpoint_group_from_project": "rule:admin_required",

    "identity:create_identity_provider": "rule:admin_required",
    "identity:list_identity_providers": "rule:admin_required",
    "identity:get_identity_providers": "rule:admin_required",
    "identity:update_identity_provider": "rule:admin_required",
    "identity:delete_identity_provider": "rule:admin_required",

    "identity:create_protocol": "rule:admin_required",
    "identity:update_protocol": "rule:admin_required",
    "identity:get_protocol": "rule:admin_required",
    "identity:list_protocols": "rule:admin_required",
    "identity:delete_protocol": "rule:admin_required",

    "identity:create_mapping": "rule:admin_required",
    "identity:get_mapping": "rule:admin_required",
    "identity:list_mappings": "rule:admin_required",
    "identity:delete_mapping": "rule:admin_required",
    "identity:update_mapping": "rule:admin_required",

    "identity:create_service_provider": "rule:admin_required",
    "identity:list_service_providers": "rule:admin_required",
    "identity:get_service_provider": "rule:admin_required",
    "identity:update_service_provider": "rule:admin_required",
    "identity:delete_service_provider": "rule:admin_required",

    "identity:get_auth_catalog": "",
    "identity:get_auth_projects": "",
    "identity:get_auth_domains": "",

    "identity:list_projects_for_groups": "",
    "identity:list_domains_for_groups": "",

    "identity:list_revoke_events": "",

    "identity:create_policy_association_for_endpoint": "rule:admin_required",
    "identity:check_policy_association_for_endpoint": "rule:admin_required",
    "identity:delete_policy_association_for_endpoint": "rule:admin_required",
    "identity:create_policy_association_for_service": "rule:admin_required",
    "identity:check_policy_association_for_service": "rule:admin_required",
    "identity:delete_policy_association_for_service": "rule:admin_required",
    "identity:create_policy_association_for_region_and_service": "rule:admin_required",
    "identity:check_policy_association_for_region_and_service": "rule:admin_required",
    "identity:delete_policy_association_for_region_and_service": "rule:admin_required",
    "identity:get_policy_for_endpoint": "rule:admin_required",
    "identity:list_endpoints_for_policy": "rule:admin_required",

    "identity:create_domain_config": "rule:admin_required",
    "identity:get_domain_config": "rule:admin_required",
    "identity:update_domain_config": "rule:admin_required",
    "identity:delete_domain_config": "rule:admin_required"
}