Attempt to restrict identity:create_user

## Applying the following Keystone policy overrides with openstack-ansible.

keystone_policy_overrides:
  cloud_admin: "role:admin and (is_admin_project:True or domain_id:238eb8a7ec424998b439d716c423dbde)"
  admin_required: "role:admin"
  admin_and_matching_user_domain_id: "rule:admin_required and domain_id:%(user.domain_id)s"
  identity:create_user: "rule:cloud_admin or rule:admin_and_matching_user_domain_id"
  

## Confirm the cloud_admin alias' domain_id matches the `cloud_admin` domain.

$ openstack domain show cloud_admin
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | Cloud admin domain               |
| enabled     | True                             |
| id          | 238eb8a7ec424998b439d716c423dbde |
| name        | cloud_admin                      |
| tags        | []                               |
+-------------+----------------------------------+

$ openstack domain show product_test
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | Niko product test domain         |
| enabled     | True                             |
| id          | 5abad140eb3349c2b55ba9cf419f493c |
| name        | product_test                     |
| tags        | []                               |
+-------------+----------------------------------+


## Create project and user within the product_test domain.

$ openstack project create --domain product_test product_test2
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description |                                  |
| domain_id   | 5abad140eb3349c2b55ba9cf419f493c |
| enabled     | True                             |
| id          | 2b3dda2239374ba98240dfd476af487d |
| is_domain   | False                            |
| name        | product_test2                    |
| parent_id   | 5abad140eb3349c2b55ba9cf419f493c |
| tags        | []                               |
+-------------+----------------------------------+

$ openstack user create --project product_test2 --password product_test2 --domain product_test product_test2
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| default_project_id  | 2b3dda2239374ba98240dfd476af487d |
| domain_id           | 5abad140eb3349c2b55ba9cf419f493c |
| enabled             | True                             |
| id                  | d6f0bc2a452c4775be2e22d61fd10993 |
| name                | product_test2                    |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

$ openstack role add admin --user product_test2 --user-domain product_test --project product_test2


## Load the environment variable configuration file.

$ cat openrc-product-test2
export LC_ALL=C

# COMMON CINDER ENVS
export CINDER_ENDPOINT_TYPE=internalURL

# COMMON NOVA ENVS
export NOVA_ENDPOINT_TYPE=internalURL

# COMMON OPENSTACK ENVS
export OS_ENDPOINT_TYPE=internalURL
export OS_INTERFACE=internalURL
export OS_USERNAME=product_test2
export OS_PASSWORD=product_test2
export OS_PROJECT_NAME=product_test2
export OS_TENANT_NAME=product_test2
export OS_AUTH_TYPE=password
export OS_AUTH_URL=http://10.103.0.8:5000/v3
export OS_NO_CACHE=1
export OS_USER_DOMAIN_NAME=product_test
export OS_PROJECT_DOMAIN_NAME=product_test
export OS_REGION_NAME=RegionOne

# For openstackclient
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3

$ source openrc-product-test2

$ openstack user create product_test3 --password product_test3
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| enabled             | True                             |
| id                  | d7eaab191f8544c1b86cf01520d3b065 |
| name                | product_test3                    |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

In the above example, we can see the product_test2 user, who was created in the product_test domain, create a user in the default domain - even though policy should restrict this.

I've attempted to restrict create_user to just the cloud_admin rule while troubleshooting.

identity:create_user: "rule:cloud_admin"

However, now the playbook fails.

$ sudo openstack-ansible /opt/openstack-ansible/playbooks/os-keystone-install.yml --tags "keystone-config"

TASK [os_keystone : Bootstrap keystone admin and endpoint] ****************************************************************************
Friday 30 November 2018  23:52:31 +0200 (0:00:00.680)       0:00:45.221 *******
FAILED - RETRYING: Bootstrap keystone admin and endpoint (5 retries left).
FAILED - RETRYING: Bootstrap keystone admin and endpoint (4 retries left).
FAILED - RETRYING: Bootstrap keystone admin and endpoint (3 retries left).
FAILED - RETRYING: Bootstrap keystone admin and endpoint (2 retries left).
FAILED - RETRYING: Bootstrap keystone admin and endpoint (1 retries left).
fatal: [os1_keystone_container-4044b189]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["/openstack/venvs/keystone-17.1.3/bin/keystone-manage", "bootstrap", "--bootstrap-username", "admin", "--bootstrap-password", "40b68878d45f1a1b04539690", "--bootstrap-project-name", "admin", "--bootstrap-role-name", "admin", "--bootstrap-service-name", "keystone", "--bootstrap-region-id", "RegionOne", "--bootstrap-admin-url", "http://10.103.0.8:35357", "--bootstrap-public-url", "https://cloud.auto-h.net:5000", "--bootstrap-internal-url", "http://10.103.0.8:5000"], "delta": "0:00:04.285494", "end": "2018-11-30 23:53:48.865457", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-11-30 23:53:44.579963", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

NO MORE HOSTS LEFT ********************************************************************************************************************

PLAY RECAP ****************************************************************************************************************************
os1_keystone_container-4044b189 : ok=53   changed=4    unreachable=0    failed=1

Reviewing keystone.log on the Keystone containers, we see the following error.

2018-11-30 23:53:05.267 40370 ERROR keystone ValidationError: Role 3837d79a26c84688bae2beeee4c06c07 is a domain-specific role. Unable to use a domain-specific role in a system assignment.

I'd previously created an admin role within the cloud_admin domain, which appears to cause the issue.

$ openstack role show 3837d79a26c84688bae2beeee4c06c07
+-----------+----------------------------------+
| Field     | Value                            |
+-----------+----------------------------------+
| domain_id | 238eb8a7ec424998b439d716c423dbde |
| id        | 3837d79a26c84688bae2beeee4c06c07 |
| name      | admin                            |
+-----------+----------------------------------+

$ openstack domain show 238eb8a7ec424998b439d716c423dbde
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | Cloud admin domain               |
| enabled     | True                             |
| id          | 238eb8a7ec424998b439d716c423dbde |
| name        | cloud_admin                      |
| tags        | []                               |
+-------------+----------------------------------+

Deleted the role and ran playbook again.

$ openstack role delete 3837d79a26c84688bae2beeee4c06c07

Resolved the issue - playbook applied successfully.

However, the product_test user can still create users in default domain.

$ openstack user create product_test5 --password product_test5 --domain default
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| enabled             | True                             |
| id                  | ecbfce5038524ad484516c47448e95e8 |
| name                | product_test5                    |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

Just to confirm we're using the expected token...

$ openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                                   |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires    | 2018-12-01T10:11:37+0000                                                                                                                                                                |
| id         | gAAAAABcAbWZzmOJBWIj0rlL3aLollj1tesGGa5jNchM8IpLXMEh5RepTTHRxUBRhy45U8XDAyjhSyzd6BaBxg4U8dPiWomeLUgauu5bCPEbENPOSl0rlA3ycHZSHMaoFxWrYkdpSlG-JiOHue5W7fQjJdOGRTLMaBLdi28HapsvgfmYft8lG1c |
| project_id | 2b3dda2239374ba98240dfd476af487d                                                                                                                                                        |
| user_id    | d6f0bc2a452c4775be2e22d61fd10993                                                                                                                                                        |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ openstack user show d6f0bc2a452c4775be2e22d61fd10993
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| default_project_id  | 2b3dda2239374ba98240dfd476af487d |
| domain_id           | 5abad140eb3349c2b55ba9cf419f493c |
| enabled             | True                             |
| id                  | d6f0bc2a452c4775be2e22d61fd10993 |
| name                | product_test2                    |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

$ openstack project show 2b3dda2239374ba98240dfd476af487d
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description |                                  |
| domain_id   | 5abad140eb3349c2b55ba9cf419f493c |
| enabled     | True                             |
| id          | 2b3dda2239374ba98240dfd476af487d |
| is_domain   | False                            |
| name        | product_test2                    |
| parent_id   | 5abad140eb3349c2b55ba9cf419f493c |
| tags        | []                               |
+-------------+----------------------------------+

Lets remove refereneces to is_admin_project.

$ git diff HEAD~1
 keystone_policy_overrides:
-  cloud_admin: "role:admin and (is_admin_project:True or domain_id:238eb8a7ec424998b439d716c423dbde)"
+  cloud_admin: "role:admin and domain_id:238eb8a7ec424998b439d716c423dbde"
   admin_required: "role:admin"
   admin_and_matching_user_domain_id: "rule:admin_required and domain_id:%(user.domain_id)s"
   identity:create_user: "rule:cloud_admin" # or rule:admin_and_matching_user_domain_id"

Apply changes via Ansible, and...

$ openstack user create product_test3 --password product_test3
You are not authorized to perform the requested action: identity:create_user. (HTTP 403) (Request-ID: req-a04a194a-5a0f-4df7-8806-6c5788c461c0)

Amazing! Our test_product user is unable to create users (finally).

So I switch to my other terminal with nsmeds credentials, who exists in cloud_admin domain and should be able to create users in any domain...

$ openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                                   |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires    | 2018-12-01T10:29:33+0000                                                                                                                                                                |
| id         | gAAAAABcAbnN4U9zKAKsjN6Hyd-H29eWnDN9EcO6uWVABQlx5HBHj2xhRwvrMbUaL7TyjrjR8D8b2XOrZXYAsx-BKq9Y2qtavc0LRfgeH0HF3IN7fTChNN1IZzXwifj-GHZlAl7CbYXRWkV3zVs_jkWNLt3j5eh_bZAE7LUr3_aeT2DkYyjVEDM |
| project_id | ec1bd3dd5e4949b4907fed4cf7c95569                                                                                                                                                        |
| user_id    | 22668f96cfa741c392d26a81a7274a1a                                                                                                                                                        |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ openstack user show 22668f96cfa741c392d26a81a7274a1a
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| default_project_id  | ec1bd3dd5e4949b4907fed4cf7c95569 |
| domain_id           | 238eb8a7ec424998b439d716c423dbde |
| enabled             | True                             |
| id                  | 22668f96cfa741c392d26a81a7274a1a |
| name                | nsmeds                           |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

$ openstack user create nsmeds3 --password nsmeds3
You are not authorized to perform the requested action: identity:create_user. (HTTP 403) (Request-ID: req-b520a4c3-0b56-4527-b3f3-dfbb0208524a)

Fuck.

Configuration information for admin_project can be found in https://docs.openstack.org/ocata/config-reference/identity/api.html
Also mentioned in https://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/system-scope.html

Except it does not appear to be configured.

root@os1-keystone-container-4044b189:/# grep admin_project /etc/keystone/keystone.conf
root@os1-keystone-container-4044b189:/#

How about allowing admin users to create users within their own domain?

$ git diff HEAD~1
   cloud_admin: "role:admin and domain_id:238eb8a7ec424998b439d716c423dbde"
   admin_required: "role:admin"
   admin_and_matching_user_domain_id: "rule:admin_required and domain_id:%(user.domain_id)s"
-  identity:create_user: "rule:cloud_admin" # or rule:admin_and_matching_user_domain_id"
+  identity:create_user: "rule:cloud_admin or rule:admin_and_matching_user_domain_id"

Applied successfully. Trying to create a user...

$ openstack user create nsmeds3 --password nsmeds3 --domain cloud_admin
You are not authorized to perform the requested action: identity:create_user. (HTTP 403) (Request-ID: req-b75f5cf1-a0ad-44a6-9a19-54301d0f5811)

This shows up in Keystone's log.

c1bd3dd5e4949b4907fed4cf7c95569 - 238eb8a7ec424998b439d716c423dbde 238eb8a7ec424998b439d716c423dbde] Could not find domain: cloud_admin.: DomainNotFound: Could not find domain: cloud_admin.

Yet it's there.

$ openstack domain show cloud_admin
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | Cloud admin domain               |
| enabled     | True                             |
| id          | 238eb8a7ec424998b439d716c423dbde |
| name        | cloud_admin                      |
| tags        | []                               |
+-------------+----------------------------------+

sigh

Let's start from the beginning. Can we restrict admin to a specific user?

$ openstack user show nsmeds --domain cloud_admin
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| default_project_id  | ec1bd3dd5e4949b4907fed4cf7c95569 |
| domain_id           | 238eb8a7ec424998b439d716c423dbde |
| enabled             | True                             |
| id                  | 22668f96cfa741c392d26a81a7274a1a |
| name                | nsmeds                           |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

$ git diff HEAD~1
diff --git a/roles/system-osa/templates/user_variables.yml.j2 b/roles/system-osa/templates/user_variables.yml.j2
index d773d50..110d37c 100644
--- a/roles/system-osa/templates/user_variables.yml.j2
+++ b/roles/system-osa/templates/user_variables.yml.j2
@@ -197,9 +197,10 @@ keystone_keystone_conf_overrides:

 keystone_policy_overrides:
   cloud_admin: "role:admin and domain_id:238eb8a7ec424998b439d716c423dbde"
+  admin_user: "user_id:22668f96cfa741c392d26a81a7274a1a"
   admin_required: "role:admin"
   admin_and_matching_user_domain_id: "rule:admin_required and domain_id:%(user.domain_id)s"
-  identity:create_user: "rule:cloud_admin or rule:admin_and_matching_user_domain_id"
+  identity:create_user: "rule:admin_user"

Using niko.smeds@webnifico.ca credentials.

$ source openrc-personal
$ openstack user create nsmeds2
No password was supplied, authentication will fail when a user does not have a password.
You are not authorized to perform the requested action: identity:create_user. (HTTP 403) (Request-ID: req-74881243-89c5-42f3-b104-81b0b7f8874d)

Using nsmeds credentials (exists in cloud_admin domain).

$ source openrc-nsmeds-cloud-admin
$ openstack user create nsmeds2 --password nsmeds2
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| enabled             | True                             |
| id                  | 6872620c2a104b32aab7688e5b25d8a6 |
| name                | nsmeds2                          |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

$ openstack user create nsmeds2 --password nsmeds2 --domain cloud_admin
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | 238eb8a7ec424998b439d716c423dbde |
| enabled             | True                             |
| id                  | 34354a7aa7e34721b93d1c4fe526be72 |
| name                | nsmeds2                          |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

Ok great - RBAC is working as expected here.

Can we restrict user creation to members of cloud_admin domain, regardless of their role(s)?

$ openstack domain show cloud_admin
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | Cloud admin domain               |
| enabled     | True                             |
| id          | 238eb8a7ec424998b439d716c423dbde |
| name        | cloud_admin                      |
| tags        | []                               |
+-------------+---------------------------------+

$ git diff HEAD~1
diff --git a/roles/system-osa/templates/user_variables.yml.j2 b/roles/system-osa/templates/user_variables.yml.j2
index 110d37c..2496b59 100644
--- a/roles/system-osa/templates/user_variables.yml.j2
+++ b/roles/system-osa/templates/user_variables.yml.j2
@@ -196,11 +196,10 @@ keystone_keystone_conf_overrides:
     group_allow_delete: False

 keystone_policy_overrides:
-  cloud_admin: "role:admin and domain_id:238eb8a7ec424998b439d716c423dbde"
-  admin_user: "user_id:22668f96cfa741c392d26a81a7274a1a"
+  cloud_admin: "domain_id:238eb8a7ec424998b439d716c423dbde"
   admin_required: "role:admin"
   admin_and_matching_user_domain_id: "rule:admin_required and domain_id:%(user.domain_id)s"
-  identity:create_user: "rule:admin_user"
+  identity:create_user: "rule:cloud_admin"

Using niko.smeds@webnifico.ca credentials (we expect this to fail).

$ source openrc-personal
$ openstack user create nsmeds2 --password nsmeds2
You are not authorized to perform the requested action: identity:create_user. (HTTP 403) (Request-ID: req-6c3797e4-bb47-448e-a3bc-32834f3c744e)

Using nsmeds credentials (we expect this to succeed).

$ source openrc-nsmeds-cloud-admin
$ openstack user create nsmeds2 --password nsmeds2
You are not authorized to perform the requested action: identity:create_user. (HTTP 403) (Request-ID: req-88df79e2-00b6-48fe-86d4-b5b4bceacf7b)

This must mean that our token for nsmeds user does not contain domain ID. Let's confirm.

$ openstack user show nsmeds
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| default_project_id  | ec1bd3dd5e4949b4907fed4cf7c95569 |
| domain_id           | 238eb8a7ec424998b439d716c423dbde |
| enabled             | True                             |
| id                  | 22668f96cfa741c392d26a81a7274a1a |
| name                | nsmeds                           |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

$ openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                                   |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires    | 2018-12-04T04:56:23+0000                                                                                                                                                                |
| id         | gAAAAABcBWA3OCTfz_8Ts4bFNfpvV3VvpeoABn4_ewNtulVPiEJilGZV21mXT7U-RVByWEDRlXfTfRZmKCDKFEC_wjgIamhAxO9sHRTiCnQvEgJwls2Dtdck38E-0n9J1tydxpmB6_R4o94JLxw7k-JWkkNQF0ru0Alk3U8cWJ5zQIpHT5jt8zg |
| project_id | ec1bd3dd5e4949b4907fed4cf7c95569                                                                                                                                                        |
| user_id    | 22668f96cfa741c392d26a81a7274a1a                                                                                                                                                        |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Previously I've added admin role to nsmeds user on each level (project, domain, and system). Perhaps this causes the issue?

$ openstack role assignment list --names --user nsmeds
+-------+--------------------+-------+-------------------------+-------------+--------+-----------+
| Role  | User               | Group | Project                 | Domain      | System | Inherited |
+-------+--------------------+-------+-------------------------+-------------+--------+-----------+
| admin | nsmeds@cloud_admin |       | cloud_admin@cloud_admin |             |        | False     |
| admin | nsmeds@cloud_admin |       |                         | cloud_admin |        | False     |
| admin | nsmeds@cloud_admin |       |                         |             | all    | False     |
+-------+--------------------+-------+-------------------------+-------------+--------+-----------+

$ openstack role remove admin --project cloud_admin --project-domain cloud_admin --user nsmeds
$ openstack role assignment list --names --user nsmeds
The request you have made requires authentication. (HTTP 401) (Request-ID: req-b1c1fd78-c135-49d9-8892-86e923f267c8)
$ openstack token issue
The request you have made requires authentication. (HTTP 401) (Request-ID: req-d22c37c0-fac8-427c-becf-05f9165ea858)

Ah, this is caused by my environment credentials still containing the OS_PROJECT_NAME, OS_PROJECT_DOMAIN_NAME, and OS_TENANT_NAME. Comment out those lines and open a new terminal.

$ source openrc-nsmeds-cloud-admin
$ openstack token issue
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field   | Value                                                                                                                                                              |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires | 2018-12-04T05:06:46+0000                                                                                                                                           |
| id      | gAAAAABcBWKmjJDq3-9FRijY0Y0y7Z80Ly5QmKXbtL8-WFuvg3oNyBHaEgak3MJg34mstncD6dMQt9qDf-OOxLcclpOWD9FONVJjTMQqiRnV-1-wJ2JNiYSTlRI0oV-GIvTCiq_PnLOOGCWAv469dt_XEmzk1Bscqw |
| user_id | 22668f96cfa741c392d26a81a7274a1a                                                                                                                                   |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
$ openstack role assignment list --names --user nsmeds
The service catalog is empty.

Ok, we're worse off than we previously were. It's likely I've overlooked or misunderstood something from documentation. Reviewing https://docs.openstack.org/keystone/pike/admin/identity-service-api-protection.html, Keystone service token expects user_id, domain_id, and project_id. So... what's going on here?

There's an example of requesting domain-scoped token with cURL in https://docs.openstack.org/keystone/pike/api_curl_examples.html, which mentions having "role assignment in that domain" as requirement, which we have.

$ openstack role assignment list --domain cloud_admin --names
+-------+--------------------+-------+---------+-------------+--------+-----------+
| Role  | User               | Group | Project | Domain      | System | Inherited |
+-------+--------------------+-------+---------+-------------+--------+-----------+
| admin | nsmeds@cloud_admin |       |         | cloud_admin |        | False     |
+-------+--------------------+-------+---------+-------------+--------+-----------+

Will keep troubleshooting this issue.

Ok, we have progress. The OS_USER_DOMAIN_NAME variable only defines where the user exists, we also need OS_DOMAIN_NAME to define where the token is scoped.

$ grep DOMAIN_NAME openrc-nsmeds-cloud-admin
export OS_DOMAIN_NAME=cloud_admin
export OS_USER_DOMAIN_NAME=cloud_admin

$ openstack token issue
+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field     | Value                                                                                                                                                                                   |
+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| domain_id | 238eb8a7ec424998b439d716c423dbde                                                                                                                                                        |
| expires   | 2018-12-04T05:47:48+0000                                                                                                                                                                |
| id        | gAAAAABcBWxEQtvh9zRieE6UnN3kZzEv8HW_vylm_w_WYoKySwG8juC0VjKMa4HrFEjMbFLTavjuWof6ABAfM40XhGzfJkF9YC-VX1sNHGBhdTJb2MzRLU1edoivcsfO-DXOXQTCFxDYf686wMPI89fV3WBAKm2DL_E0HIQKIXykw__Vop3fVS0 |
| user_id   | 22668f96cfa741c392d26a81a7274a1a                                                                                                                                                        |
+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Retesting the following changes.

+keystone_policy_overrides:
+  cloud_admin: "domain_id:238eb8a7ec424998b439d716c423dbde"
+  identity:create_user: "rule:cloud_admin"

Great success.

• nsmeds can create users
• any user not in the cloud_admin domain fails to create user

Hoping for smoother sailing going forward.

It's time to get all Keystone policy changes (https://gist.github.com/nikosmeds/74881a4de5c777b8e808cdee442cdcdb) applied and working - however, once again there are issues to investigate and resolve.

My nsmeds user in the cloud_admin domain can successfully create and delete users in any domain, great.
The product_test user in product_test domain is unable to list users in any domain, including product_test domain. Not so great.

$ source openrc-product-test
$ openstack token issue
+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field     | Value                                                                                                                                                                                   |
+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| domain_id | 5abad140eb3349c2b55ba9cf419f493c                                                                                                                                                        |
| expires   | 2018-12-04T09:06:42+0000                                                                                                                                                                |
| id        | gAAAAABcBZri5lIzT7S3EUWdhQf_WZ_Bd8tb7RcbvccJzwbIyBj7YX-rhPA-GRL0Rg_dKb7zVNow8BDJISjIXk2XXkEXuLGeM7AksB79IjFBl4XLnEvr5tI_HkpTG3FbAjljMex91SBKIu_IUeFuPrkiwrwrxP1dyPDJVN9nggIh_ehscyRhMDM |
| user_id   | 1bbd32cd2371468fab764ea28586c235                                                                                                                                                        |
+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ openstack domain show 5abad140eb3349c2b55ba9cf419f493c
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | Niko product test domain         |
| enabled     | True                             |
| id          | 5abad140eb3349c2b55ba9cf419f493c |
| name        | product_test                     |
| tags        | []                               |
+-------------+----------------------------------+

$ openstack user list --domain product_test
You are not authorized to perform the requested action: identity:list_users. (HTTP 403) (Request-ID: req-d8235214-2858-4d80-8976-29fc0921a880)

We see the following in Keystone container's log file.

$ tail -f /var/log/keystone/keystone.log
2018-12-03 23:08:18.098 5302 WARNING py.warnings [req-d8235214-2858-4d80-8976-29fc0921a880 1bbd32cd2371468fab764ea28586c235 - 5abad140eb3349c2b55ba9cf419f493c 5abad140eb3349c2b55ba9cf419f493c -] /openstack/venvs/keystone-17.1.3/lib/python2.7/site-packages/oslo_policy/policy.py:869: UserWarning: Policy identity:list_users failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required
  warnings.warn(msg)

2018-12-03 23:08:18.099 5302 WARNING keystone.common.wsgi [req-d8235214-2858-4d80-8976-29fc0921a880 1bbd32cd2371468fab764ea28586c235 - 5abad140eb3349c2b55ba9cf419f493c 5abad140eb3349c2b55ba9cf419f493c -] You are not authorized to perform the requested action: identity:list_users.: ForbiddenAction: You are not authorized to perform the requested action: identity:list_users.

Am I having issues again with correctly scoping the token?

Commenting out the OS_DOMAIN_NAME variable allows us to grab a system-scoped token.

$ grep OS_DOMAIN_NAME openrc-product-test
#export OS_DOMAIN_NAME=product_test
$ source openrc-product-test
$ openstack token issue
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field   | Value                                                                                                                                                              |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires | 2018-12-04T09:10:26+0000                                                                                                                                           |
| id      | gAAAAABcBZvC5YRMpQwnz2YNtMfqv96BY-X9dUew1hGl42Tqa5veWKBV0KV1TnYkgfQJ3RYMWf0XUbcaYlW0x0BlHivx1LszsVfddWzp3xo31km2m5B5R1t-eRV4yuRBiRmgegSj9_F13k1iN3qun8iM2xhdcNtcyw |
| user_id | 1bbd32cd2371468fab764ea28586c235                                                                                                                                   |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ openstack user list --domain product_test
The service catalog is empty.

Nope. Alright, here we go again. So below we can see the policy rules for listing users.

identity:list_users: "rule:cloud_admin or rule:admin_and_matching_domain_id"

admin_and_matching_domain_id: "rule:admin_required and domain_id:%(domain_id)s"

admin_required: "role:admin"

Going to troubleshoot by removing requirements until product_test user can list users in their own domain. Because we're comparing the token's domain ID to the API request's target domain, we need to ensure that we use a domain-scoped token.

$ git diff HEAD~1
diff --git a/roles/system-osa/templates/user_variables.yml.j2 b/roles/system-osa/templates/user_variables.yml.j2
index dd7a995..b004e5e 100644
--- a/roles/system-osa/templates/user_variables.yml.j2
+++ b/roles/system-osa/templates/user_variables.yml.j2
@@ -202,7 +202,7 @@ keystone_policy_overrides:
   cloud_admin: "user_id:2035bc66d7b64ce6add3b5519c5fcbf7 or (role:admin and domain_id:238eb8a7ec424998b439d716c423dbde)"
   owner: "user_id:%(user_id)s or user_id:%(target.token.user_id)s"
   admin_or_owner: "(rule:admin_required and domain_id:%(target.token.user.domain.id)s) or rule:owner"
-  admin_and_matching_domain_id: "rule:admin_required and domain_id:%(domain_id)s"
+  admin_and_matching_domain_id: "domain_id:%(domain_id)s"

Nope, same permission denied error.

Let's check the reverse. Also, personal note: this is tedious as hell to troubleshoot.

$ git diff HEAD~1
diff --git a/roles/system-osa/templates/user_variables.yml.j2 b/roles/system-osa/templates/user_variables.yml.j2
index b004e5e..e915c0d 100644
--- a/roles/system-osa/templates/user_variables.yml.j2
+++ b/roles/system-osa/templates/user_variables.yml.j2
@@ -202,7 +202,7 @@ keystone_policy_overrides:
   cloud_admin: "user_id:2035bc66d7b64ce6add3b5519c5fcbf7 or (role:admin and domain_id:238eb8a7ec424998b439d716c423dbde)"
   owner: "user_id:%(user_id)s or user_id:%(target.token.user_id)s"
   admin_or_owner: "(rule:admin_required and domain_id:%(target.token.user.domain.id)s) or rule:owner"
-  admin_and_matching_domain_id: "domain_id:%(domain_id)s"
+  admin_and_matching_domain_id: "rule:admin_required"

Apply changes and confirm product_test user can list users now.

$ openstack user list
+----------------------------------+---------------+
| ID                               | Name          |
+----------------------------------+---------------+
| 1bbd32cd2371468fab764ea28586c235 | product_test  |
| d6f0bc2a452c4775be2e22d61fd10993 | product_test2 |
+----------------------------------+---------------+

Great, so the issue is with "domain_id:%(domain_id)s", which should evaluate True when the token's domain ID matches the API calls domain ID. There are a few variations of this in policy.v3cloudsample, and either they have a mistake in their example or I'm once again misunderstanding something. Here's my current understanding of these variations.

Rule	Details
domain_id:%(target.user.domain_id)s	Token's domain ID must match the target resource's (i.e. existing user's) domain ID.
domain_id:%(domain_id)s	Token's domain ID must match the target domain ID (i.e. creating new users in a domain).
domain_id:%(target.project.domain_id)s	Token's domain ID must match parent domain of target project (i.e. existing project).
domain_id:%(project.domain_id)s	Token's domain ID must match parent domain of target project (i.e. creating new project in a domain).
domain_id:%(target.token.user.domain.id)s	Token's domain ID must match target token's domain ID. This rule is applied to the actions check_token, validate_token, and revoke_token.

The best related documentation I've been able to find is https://docs.openstack.org/keystone/queens/admin/identity-service-api-protection.html. Also, reviewing what rules are applied to what actions in https://github.com/openstack/keystone/blob/master/etc/policy.v3cloudsample.json gave me a better understanding of the above domain_id examples.

... except it isn't working.

Throwing darts in the dark here: let's allow any of the domain_id variations (shown in previous comment) just to see if ANY of them return True.

$ git diff HEAD~1
diff --git a/roles/system-osa/templates/user_variables.yml.j2 b/roles/system-osa/templates/user_variables.yml.j2
index dd7a995..9b688c8 100644
--- a/roles/system-osa/templates/user_variables.yml.j2
+++ b/roles/system-osa/templates/user_variables.yml.j2
@@ -261,7 +261,7 @@ keystone_policy_overrides:
   admin_and_matching_target_user_domain_id: "rule:admin_required and domain_id:%(target.user.domain_id)s"
   admin_and_matching_user_domain_id: "rule:admin_required and domain_id:%(user.domain_id)s"
   identity:get_user: "rule:cloud_admin or rule:admin_and_matching_target_user_domain_id or rule:owner"
-  identity:list_users: "rule:cloud_admin or rule:admin_and_matching_domain_id"
+  identity:list_users: "rule:cloud_admin or rule:admin_and_matching_domain_id or rule:admin_and_matching_target_user_domain_id or rule:admin_and_matching_user_domain_id or rule:admin_or_owner"

Applied changes, and...

$ openstack user list
You are not authorized to perform the requested action: identity:list_users. (HTTP 403) (Request-ID: req-6dc1b47a-a74a-4780-9864-a4a71d694c84)
$ openstack user list --domain product_test
You are not authorized to perform the requested action: identity:list_users. (HTTP 403) (Request-ID: req-8f8b48a2-a257-4216-aea3-1f2ef5115442)

Ok... not good. How about we test deleting of existing users, since this applies policies to an actual target resource. Start by creating a new user (using cloud admin credentials).

$ openstack user create nsmeds3 --password nsmeds3 --domain product_test
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | 5abad140eb3349c2b55ba9cf419f493c |
| enabled             | True                             |
| id                  | 47c27fc0576041de8eff4ca792b7a4ed |
| name                | nsmeds3                          |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

Switch to product_test credentials, with domain-scoped token.

$ openstack user delete nsmeds3 --domain product_test
Failed to delete user with name or ID 'nsmeds3': You are not authorized to perform the requested action: identity:list_users. (HTTP 403) (Request-ID: req-7311227c-acb2-4603-8e26-de721d35f18a)
1 of 1 users failed to delete.

Oh common. Okay, we're opening the list_users action to any admins.

$ git diff HEAD~1
diff --git a/roles/system-osa/templates/user_variables.yml.j2 b/roles/system-osa/templates/user_variables.yml.j2
index 9b688c8..e57edd8 100644
--- a/roles/system-osa/templates/user_variables.yml.j2
+++ b/roles/system-osa/templates/user_variables.yml.j2
@@ -261,7 +261,7 @@ keystone_policy_overrides:
-  identity:list_users: "rule:cloud_admin or rule:admin_and_matching_domain_id or rule:admin_and_matching_target_user_domain_id or rule:admin_and_matching_user_domain_id or rule:admin_or_owner"
+  identity:list_users: "rule:admin_required"

Applied above changes. Again, using the product_test user credentials with domain-scoped token.

$ openstack user list
+----------------------------------+---------------+
| ID                               | Name          |
+----------------------------------+---------------+
| 1bbd32cd2371468fab764ea28586c235 | product_test  |
| 47c27fc0576041de8eff4ca792b7a4ed | nsmeds3       |
| d6f0bc2a452c4775be2e22d61fd10993 | product_test2 |
+----------------------------------+---------------+

$ openstack user list
+----------------------------------+---------------+
| ID                               | Name          |
+----------------------------------+---------------+
| 1bbd32cd2371468fab764ea28586c235 | product_test  |
| 47c27fc0576041de8eff4ca792b7a4ed | nsmeds3       |
| d6f0bc2a452c4775be2e22d61fd10993 | product_test2 |
+----------------------------------+---------------+
$ openstack user delete nsmeds3 --domain product_test
Failed to delete user with name or ID 'nsmeds3': No user with a name or ID of 'nsmeds3' exists.
1 of 1 users failed to delete.
$ openstack user delete nsmeds3
$ openstack user list
+----------------------------------+---------------+
| ID                               | Name          |
+----------------------------------+---------------+
| 1bbd32cd2371468fab764ea28586c235 | product_test  |
| d6f0bc2a452c4775be2e22d61fd10993 | product_test2 |
+----------------------------------+---------------+

Apparently when you specify your default domain with --domain <domain> things break. Good times. But happy to see we can in fact delete a user created within the same domain! Let's test a bit more functionality with the same credentials.

Using cloud_admin credentials, creating user in cloud_admin domain.

$ openstack user create nsmeds5 --password nsmeds5 --domain cloud_admin
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | 238eb8a7ec424998b439d716c423dbde |
| enabled             | True                             |
| id                  | 52816efd1a374339b274820124210841 |
| name                | nsmeds5                          |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

Switch back to product_test credentials and attempt to delete the previously created user.

$ openstack user delete nsmeds5 --domain cloud_admin
Failed to delete user with name or ID 'nsmeds5': No user with a name or ID of 'nsmeds5' exists.
1 of 1 users failed to delete.

Interesting - not permission denied, but simply states cannot find the user. Switching back to cloud_admin credentials to delete with same command.

$ openstack user delete nsmeds5 --domain cloud_admin
$

How about user creation? Switching to product_test credentials again.

$ openstack user create product_test5 --password product_test5 --domain default
You are not authorized to perform the requested action: identity:create_user. (HTTP 403) (Request-ID: req-605db68a-522d-4cd9-8111-c6ba2ce7d9b4)
$ openstack user create product_test5 --password product_test5
You are not authorized to perform the requested action: identity:create_user. (HTTP 403) (Request-ID: req-e5024d7e-30ee-4c06-8178-d3c436ab033f)

Yep, create_user suffers from same issue as list_users (since they target a domain and not a specific user resource). Will need to figure that out.

Reminder to self: add --debug to any CLI command.

It isn't helping with my above problem, but hey - it's good to remember.

Alright... I don't want to admit to this, but I will: decided to directly test user creation with HTTP API (using curl). Was curious if issue somehow related to Python client.

1. Request domain-scoped token (response not shown below, but contains token).

curl -i \
  -H "Content-Type: application/json" \
  -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "product_test",
          "domain": { "name": "product_test" },
          "password": "product_test"
        }
      }
    },
    "scope": {
      "domain": {
        "name": "product_test"
      }
    }
  }
}' \
  "https://cloud.auto-h.net:5000/v3/auth/tokens" ; echo

2. Add token as header, send user creation request.

curl -i \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-Auth-Token: gAAAAABcBcAq72OtOU7GE5RAogLhEMvIQtbY5CdLk0Hvq4n0frBN_EPe2sWhI03PeEmYtHSylZQ6yL4hJZv1UKr9YvqVz1AvcioogUORwybiwuvJLxZbEsO5eqGe7Us2vu5WRTy8NqV5TlNkKMXkW3D4WUFOymZfi99eHxqWRwxM6jvj1mfVh0Q" \
  -d '
{ "user": {
    "default_project_id": "cc91843aee474f3295eb3fc46961bab3",
    "domain_id": "5abad140eb3349c2b55ba9cf419f493c",
    "enabled": true,
    "name": "Product REST",
    "password": "productrest"
    }
}' \
  "https://cloud.auto-h.net:5000/v3/users" ; echo

HTTP/1.1 201 Created
Server: nginx/1.10.3 (Ubuntu)
Date: Mon, 03 Dec 2018 23:49:15 GMT
Content-Type: application/json
Content-Length: 338
Vary: X-Auth-Token
x-openstack-request-id: req-cd8ab7e9-37fa-4bb0-815b-ed09b653c9f8
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Security-Policy: default-src 'self' https: wss:;
X-Frame-Options: DENY

{"user": {"name": "Product REST", "links": {"self": "https://cloud.auto-h.net:5000/v3/users/940f01d8819f4700a9fd017fea474e37"}, "domain_id": "5abad140eb3349c2b55ba9cf419f493c", "enabled": true, "options": {}, "default_project_id": "cc91843aee474f3295eb3fc46961bab3", "id": "940f01d8819f4700a9fd017fea474e37", "password_expires_at": null}}

Yeah I've left passwords in this and previous comments - will destroy these users once testing complete.

... but more importantly - it worked.

$ openstack user list --domain product_test
+----------------------------------+---------------+
| ID                               | Name          |
+----------------------------------+---------------+
| 1bbd32cd2371468fab764ea28586c235 | product_test  |
| 940f01d8819f4700a9fd017fea474e37 | Product REST  |
| d6f0bc2a452c4775be2e22d61fd10993 | product_test2 |
+----------------------------------+---------------+

And ensuring the product_test token cannot create a user in cloud_admin domain.

curl -i \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-Auth-Token: gAAAAABcBcAq72OtOU7GE5RAogLhEMvIQtbY5CdLk0Hvq4n0frBN_EPe2sWhI03PeEmYtHSylZQ6yL4hJZv1UKr9YvqVz1AvcioogUORwybiwuvJLxZbEsO5eqGe7Us2vu5WRTy8NqV5TlNkKMXkW3D4WUFOymZfi99eHxqWRwxM6jvj1mfVh0Q" \
  -d '
{ "user": {
    "default_project_id": "ec1bd3dd5e4949b4907fed4cf7c95569",
    "domain_id": "238eb8a7ec424998b439d716c423dbde",
    "enabled": true,
    "name": "Product REST",
    "password": "productrest"
    }
}' \
  "https://cloud.auto-h.net:5000/v3/users" ; echo

HTTP/1.1 403 Forbidden
Server: nginx/1.10.3 (Ubuntu)
Date: Mon, 03 Dec 2018 23:57:10 GMT
Content-Type: application/json
Content-Length: 138
Vary: X-Auth-Token
x-openstack-request-id: req-f54c1b7e-7f27-4e1b-bd2c-25db860b9f63

{"error": {"message": "You are not authorized to perform the requested action: identity:create_user.", "code": 403, "title": "Forbidden"}}

Amazing - RBAC is working as expected. Either the Python client has an issue, or I'm configuring the environment variables wrong.

This has been rough.

It came to me, like a vision of embarrassment and shame, that most of my confusion stems from one possibility.

Ready?

$ source openrc-product-test

$ openstack token issue
+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field     | Value                                                                                                                                                                                   |
+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| domain_id | 5abad140eb3349c2b55ba9cf419f493c                                                                                                                                                        |
| expires   | 2018-12-05T04:04:59+0000                                                                                                                                                                |
| id        | gAAAAABcBqWrOKQGJOoSy72gF59ieqQ_KNUGVMlzZdLNN_UAlhxZkqAZvWSI7NAxilBYm5YfCiZUNaC7_lRk7eaexP2e2gmaYJRXAy7GrCaGHgbpw4ESvDgGCWFMtP4cY5bz4AnYlEtP08cMBXRCyIjEAzQusmSh4tpueDQ38Ca6WmWvP4yzpbU |
| user_id   | 1bbd32cd2371468fab764ea28586c235                                                                                                                                                        |
+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ openstack user list --domain product_test
You are not authorized to perform the requested action: identity:list_users. (HTTP 403) (Request-ID: req-eb49060b-7b46-4e0a-94d6-5c502f1e909c)

$ openstack user list --domain 5abad140eb3349c2b55ba9cf419f493c
+----------------------------------+---------------+
| ID                               | Name          |
+----------------------------------+---------------+
| 1bbd32cd2371468fab764ea28586c235 | product_test  |
| d2fba554a133468f9567453be72ee0b1 | product-test3 |
| d6f0bc2a452c4775be2e22d61fd10993 | product_test2 |
+----------------------------------+---------------+

The Keystone policy files compare domain ID, and therefore we must provide the API with a domain ID. Even though names and IDs have a one-to-one relationship, Keystone doesn't make that conversion and will deny actions when provided with a domain name.

This also explains why my cURL request in the previous comment was successful - I provided a domain ID.