We now have end-to-end integration testing of our API, however, we also need to protect our API from unauthenticated access.
For this, we will use the OIDC framework (OpenID Connect v1.0), integrating the API Gateway (Kong) with the IAM solution (Keycloak).
To integrate Kong with Keycloak, we will need to install a community plugin from the image we already use (kong:latest). For that, we will need to create a customized Dockerfile that will install the jwt-keycloak plugin.
Below the Dockerfile for installing the plugin:
FROM kong:1.4.0
RUN apk add --no-cache git
RUN git config --global url."https://".insteadOf git://
RUN luarocks install kong-plugin-jwt-keycloak
ENV KONG_PLUGINS="bundled,jwt-keycloak"We will now need to change the Makefile to:
- Build the custom Kong image from the Dockerfile we created
- Use our generated image of Kong instead of the official image
Below Updated Makefile:
#!make
run-integration-tests: integration-tests clean
integration-tests:
# we created a network to be shared by Kong, Postgres, Newman and Prism
docker network create kong-net
# we created the Kong database
docker run -d --name kong-database --network=kong-net -p 5432:5432 -e "POSTGRES_USER=kong" -e "POSTGRES_DB=kong" postgres:9.6
sleep 5
# we set up the kong database (bootstrap)
docker run --rm --network=kong-net -e "KONG_DATABASE=postgres" -e "KONG_PG_HOST=kong-database" -e "KONG_CASSANDRA_CONTACT_POINTS=kong-database" kong:latest kong migrations bootstrap
# kong
docker build -f Dockerfile -t kong-tutorial-apifirst .
docker run -d --name kong --network=kong-net -e "KONG_DATABASE=postgres" -e "KONG_ANONYMOUS_REPORTS=off" -e "KONG_PG_HOST=kong-database" -e "KONG_CASSANDRA_CONTACT_POINTS=kong-database" -e "KONG_PROXY_ACCESS_LOG=/dev/stdout" -e "KONG_ADMIN_ACCESS_LOG=/dev/stdout" -e "KONG_PROXY_ERROR_LOG=/dev/stderr" -e "KONG_ADMIN_ERROR_LOG=/dev/stderr" -e "KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 ssl" -p 8000:8000 -p 8443:8443 -p 8001:8001 -p 8444:8444 kong-tutorial-apifirst:latest
# prism
docker run --network=kong-net --rm --name prism -d -p 4010:4010 -v "$$(pwd)/petstore-v3.0.yaml":/etc/openapi/spec.file stoplight/prism:3 mock -h 0.0.0.0 "/etc/openapi/spec.file"
# swagger-to-kong
docker run --rm --name swagger-to-kong --network kong-net -e OPENAPI_SERVER=http://prism:4010 -e KONG_HOST=http://kong:8001 -v "$$(pwd)/petstore-v3.0.yaml":/etc/openapi/spec.file registry.domain.com/ci-tools/swagger-to-kong:latest python3 swaggertokong.py /etc/openapi/spec.file
# newman
docker run --net kong-net --rm -v "$$(pwd)/localhost.postman_environment.json:/etc/newman/env.json" -v "$$(pwd)/petstore-collection.json":/etc/newman/collection.json -t postman/newman:alpine run -e env.json collection.json
clean:
-docker rm --force prism
-docker rm --force kong
-docker rm --force kong-database
-docker network rm kong-netWe need to provision the Keycloak (IAM) locally so that we can request tokens from our client (newman) and validate them at the gateway (Kong).
Keycloak uses the concept of realms to group security settings logically, similar to Kubernetes workspaces.
To facilitate the tutorial, a realm (realm-export.json) called test has already been created, which has a client called newman configured to use OIDC and with the secret 5b7aad6a-9efc-4d14-9fd4-641ed48754c5
Our Makefile will provision the Keycloak with a memory database (H2) and will automatically import this pre-existing realm.
We also added a waiting time after the Keycloak is started (20 seconds), to wait for it to be properly provisioned.
Here is an updated Makefile:
#!make
run-integration-tests: integration-tests clean
integration-tests:
# we created a network to be shared by Kong, Postgres, Newman and Prism
docker network create kong-net
# we created the Kong database
docker run -d --name kong-database --network=kong-net -p 5432:5432 -e "POSTGRES_USER=kong" -e "POSTGRES_DB=kong" postgres:9.6
sleep 5
# we set up the kong database (bootstrap)
docker run --rm --network=kong-net -e "KONG_DATABASE=postgres" -e "KONG_PG_HOST=kong-database" -e "KONG_CASSANDRA_CONTACT_POINTS=kong-database" kong:latest kong migrations bootstrap
# kong
docker build -f Dockerfile -t kong-tutorial-apifirst .
docker run -d --name kong --network=kong-net -e "KONG_DATABASE=postgres" -e "KONG_ANONYMOUS_REPORTS=off" -e "KONG_PG_HOST=kong-database" -e "KONG_CASSANDRA_CONTACT_POINTS=kong-database" -e "KONG_PROXY_ACCESS_LOG=/dev/stdout" -e "KONG_ADMIN_ACCESS_LOG=/dev/stdout" -e "KONG_PROXY_ERROR_LOG=/dev/stderr" -e "KONG_ADMIN_ERROR_LOG=/dev/stderr" -e "KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 ssl" -p 8000:8000 -p 8443:8443 -p 8001:8001 -p 8444:8444 kong-tutorial-apifirst:latest
# prism
docker run --network=kong-net --rm --name prism -d -p 4010:4010 -v "$$(pwd)/petstore-v3.0.yaml":/etc/openapi/spec.file stoplight/prism:3 mock -h 0.0.0.0 "/etc/openapi/spec.file"
# keycloak
docker run -d --name keycloak --net kong-net -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin -e DB_VENDOR=h2 -e KEYCLOAK_IMPORT=/tmp/realm-export.json -v "$$(pwd)/realm-export.json:/tmp/realm-export.json" -e "JAVA_OPTS=-Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Djboss.http.port=8083 -Djboss.https.port=8445 -Djboss.bind.address.private=127.0.0.1 -Djboss.bind.address=127.0.0.1 -Djboss.bind.address.management=127.0.0.1 -Djboss.bind.address.unsecure=127.0.0.1" -p 8083:8083 jboss/keycloak
sleep 20
# swagger-to-kong
docker run --rm --name swagger-to-kong --network kong-net -e OPENAPI_SERVER=http://prism:4010 -e KONG_HOST=http://kong:8001 -v "$$(pwd)/petstore-v3.0.yaml":/etc/openapi/spec.file registry.domain.com/ci-tools/swagger-to-kong:latest python3 swaggertokong.py /etc/openapi/spec.file
# newman
docker run --net kong-net --rm -v "$$(pwd)/localhost.postman_environment.json:/etc/newman/env.json" -v "$$(pwd)/petstore-collection.json":/etc/newman/collection.json -t postman/newman:alpine run -e env.json collection.json
clean:
-docker rm --force prism
-docker rm --force kong
-docker rm --force kong-database
-docker rm --force keycloak
-docker network rm kong-netWe have the following drawing of the current solution:
We can run the Makefile to verify that everything is set up correctly:
make -k run-integration-testsWe now have the IAM configured and provisioned along with our test suite, gateway and mock.
However, we still need to configure our API at the gateway so that it is secure. In Kong, this is done through plugins.
The swagger-to-kong component is also capable of configuring plugins via Route, which is exactly what we need.
For that, we will create the file json kong-route-config.json with the plugins that we want to enable by Route.
The Route name is obtained by sanitizing the API + info.title. + the paths.PATH.METHOD.operationId of the operation. Sanitization is the process of removing special characters, exchanging spaces for - and exchanging accented characters for their respective non-accented characters, all in lowercase.
The openapi section below would become the Route named swagger-petstore.listPets:
openapi: "3.0.0"
info:
version: 1.0.0
title: Swagger Petstore
license:
name: MIT
servers:
- url: http://petstore.swagger.io/v1
paths:
/pets:
get:
summary: List all pets
operationId: listPets
tags:
- pets
parameters:
- name: limit
in: query
description: How many items to return at one time (max 100)
required: false
schema:
type: integer
format: int32
responses:
'200':
description: A paged array of pets
headers:
x-next:
description: A link to the next page of responses
schema:
type: string
content:
application/json:
schema:
$ref: "#/components/schemas/Pets"
default:
description: unexpected error
content:
application/json:
schema:
$ref: "#/components/schemas/Error"Using the regex features available in swagger-to-kong, we will have the following content for the file kong-route-config.json (which contains the plugins that will be enabled/configured by Route):
{
".*": {
"plugins": [
{
"name": "jwt-keycloak",
"config": {
"allowed_iss": [
"http://keycloak:8083/auth/realms/test"
],
"cookie_names": [
"oauthtoken"
]
}
}
]
},
"swagger-petstore.listPets": {
"plugins": [
{
"config": {
"methods": [],
"origins": [
"*"
],
"credentials": false,
"preflight_continue": false
},
"name": "cors",
"protocols": [
"grpc",
"grpcs",
"http",
"https"
],
"enabled": true,
"run_on": "first"
}
]
}
}This configuration file tells the component swagger-to-kong:
- Enable the Prometheus metrics plugin for all Routes (this setting is the default for swagger-to-kong and is defined via the [kong-route-config-default.json] file (https://github.com/3bit-techs/swagger-to-kong/blob/master/kong-route-config-default.json))
- For Route swagger-petstore.showPetById, swagger-petstore.listPets and swagger-petstore.createPets (via the
.*Regex) configure the security plugin (jwt-keycloak) already pointing to the IAM (keycloak) and also stating that the Access Token can be extracted from Cookie oauthtoken (to facilitate integration with [oidc-proxy] (https://github.com/3bit-techs/oidc-proxy)) - For Route swagger-petstore.listPets configure the additional CORS plugin.
This file must be kept under versioning and one can be created for each environment (according to the IAM and the settings of the other plugins) or it can be transformed through some templating process (e.g. mustache).
The configuration of the CORS plugin demonstrated in the previous item, was extracted directly from the Konga interface.
With that, we can manually configure the plugins via UI and then export only the JSON that represents each plugin, to facilitate the use of the tool.
Necessary steps:
- In Konga, select the desired Service or Route and browse Plugins
- Click the Raw view icon (eye icon)
- Copy the generated JSON
- Remove unnecessary attributes and use JSON in the kong-route-config.json file
Export example:
JSON (raw) example:
{
"created_at": 1568171036,
"config": {
"methods": [],
"exposed_headers": null,
"max_age": null,
"headers": null,
"origins": [
"*"
],
"credentials": false,
"preflight_continue": false
},
"id": "d1cb564e-1d7e-4e2b-b61a-68a254bc1e5b",
"service": {
"id": "c84d8797-38cf-41d0-b41e-0ed815e84060"
},
"name": "cors",
"protocols": [
"grpc",
"grpcs",
"http",
"https"
],
"enabled": true,
"run_on": "first",
"consumer": null,
"route": null,
"tags": null
}JSON example (no attributes with null value, ids, parent Service/Route item and timestamps):
{
"config": {
"methods": [],
"origins": [
"*"
],
"credentials": false,
"preflight_continue": false
},
"name": "cors",
"protocols": [
"grpc",
"grpcs",
"http",
"https"
],
"enabled": true,
"run_on": "first"
}We now need to change the Makefile to reflect the necessary changes, informing swagger-to-kong the plugin configuration file:
#!make
run-integration-tests: integration-tests clean
integration-tests:
# we created a network to be shared by Kong, Postgres, Newman and Prism
docker network create kong-net
# we created the Kong database
docker run -d --name kong-database --network=kong-net -p 5432:5432 -e "POSTGRES_USER=kong" -e "POSTGRES_DB=kong" postgres:9.6
sleep 5
# we set up the kong database (bootstrap)
docker run --rm --network=kong-net -e "KONG_DATABASE=postgres" -e "KONG_PG_HOST=kong-database" -e "KONG_CASSANDRA_CONTACT_POINTS=kong-database" kong:latest kong migrations bootstrap
# kong
docker build -f Dockerfile -t kong-tutorial-apifirst .
docker run -d --name kong --network=kong-net -e "KONG_DATABASE=postgres" -e "KONG_ANONYMOUS_REPORTS=off" -e "KONG_PG_HOST=kong-database" -e "KONG_CASSANDRA_CONTACT_POINTS=kong-database" -e "KONG_PROXY_ACCESS_LOG=/dev/stdout" -e "KONG_ADMIN_ACCESS_LOG=/dev/stdout" -e "KONG_PROXY_ERROR_LOG=/dev/stderr" -e "KONG_ADMIN_ERROR_LOG=/dev/stderr" -e "KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 ssl" -p 8000:8000 -p 8443:8443 -p 8001:8001 -p 8444:8444 kong-tutorial-apifirst:latest
# prism
docker run --network=kong-net --rm --name prism -d -p 4010:4010 -v "$$(pwd)/petstore-v3.0.yaml":/etc/openapi/spec.file stoplight/prism:3 mock -h 0.0.0.0 "/etc/openapi/spec.file"
# keycloak
docker run -d --name keycloak --net kong-net -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin -e DB_VENDOR=h2 -e KEYCLOAK_IMPORT=/tmp/realm-export.json -v "$$(pwd)/realm-export.json:/tmp/realm-export.json" -e "JAVA_OPTS=-Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Djboss.http.port=8083 -Djboss.https.port=8445 -Djboss.bind.address.private=127.0.0.1 -Djboss.bind.address=127.0.0.1 -Djboss.bind.address.management=127.0.0.1 -Djboss.bind.address.unsecure=127.0.0.1" -p 8083:8083 jboss/keycloak
sleep 20
# swagger-to-kong
docker run --rm --name swagger-to-kong --network kong-net -e OPENAPI_SERVER=http://prism:4010 -e KONG_HOST=http://kong:8001 -v "$$(pwd)/kong-route-config.json":/etc/openapi/kong-route-config.json -v "$$(pwd)/petstore-v3.0.yaml":/etc/openapi/spec.file registry.domain.com/ci-tools/swagger-to-kong:latest python3 swaggertokong.py /etc/openapi/spec.file
# newman
docker run --net kong-net --rm -v "$$(pwd)/localhost.postman_environment.json:/etc/newman/env.json" -v "$$(pwd)/petstore-collection.json":/etc/newman/collection.json -t postman/newman:alpine run -e env.json collection.json
clean:
-docker rm --force prism
-docker rm --force kong
-docker rm --force kong-database
-docker rm --force keycloak
-docker network rm kong-netIf we run Makefile now, we will have to see the tests fail due to the fact that our API is protected with OIDC and the client (newman) is not configured to pass on valid credentials:
make -k run-integration-tests
For our integration test to work again, we need to configure our collection in Postman so that it:
- Have credentials (client_id and client_secret) as Variable in the environment localhost
- Invoke a Pre-request Script for all operations, which will talk to the IAM (Keycloak) to negotiate the credentials for an Access Token
- Propagate the Access Token for all operations as Header
Authorization: Bearer $TOKEN - Use an oauthtoken cookie for one of the operations instead of the Header to propagate the Access Token
In the postman, we will add the variables to the localhost environment by following the steps below:
- Click on Manage Environments
- In the Popup, select the environment localhost
- Add the variable client_id with the value newman
- Add the variable client_secret with the value 5b7aad6a-9efc-4d14-9fd4-641ed48754c5
- Add the tenant variable with the value http://keycloak:8083/auth/realms/test/protocol/openid-connect/token
- Click Update to make the changes effective
- Export the environment as done in the step Creating an environment and exporting
Now we will create a pre-script directly in the Collection Swagger Petstore, which will cause it to be executed before each execution.
Steps to create the Pre-script:
- In the Postman side navigation menu, right click on the collection Swagger Petstore and then click Edit
- In the popup, select the Pre-request Scripts tab
- Add the script below:
var client_id = pm.environment.get("client_id");
var client_secret = pm.environment.get("client_secret");
var tenant = pm.environment.get("tenant");
pm.sendRequest({
url: tenant,
method: 'POST',
header: {
'content-type': 'application/x-www-form-urlencoded'
},
body: {
mode: 'urlencoded',
urlencoded: [
{ key: "grant_type", value: "client_credentials" },
{ key: "client_id", value: client_id, disabled: false },
{ key: "client_secret", value: client_secret, disabled: false }
]
}
}, function (err, res) {
pm.environment.set("token", res.json().access_token);
})- Click on update to make the changes effective
Now we will set up Bearer authentication directly in the Collection Swagger Petstore, which will make it used together with each run.
Steps to configure authentication:
- In the Postman side navigation menu, right click on the collection Swagger Petstore and then click Edit
- In the popup, select the Authorization tab
- In the Type combo, select Bearer Token
- In the Token field, enter the value {{token}}
- Click Update to make the changes effective
As our openapi spec is not secure, by default Postman does not define the authentication configuration, leaving it as In Auth, we have to change this configuration in all requests:
- In the collection Swagger Petstore, in the folder pets select the requests (List all pets and Create a pet) and one by one navigate to the tab Authorization
- In the TYPE combo select the type Inherit auth from parent
- Click Save to make the changes
We will configure one of the requests so that it uses Cookie oauthtoken to authenticate instead of Header Authorization, for this:
- In the collection Swagger Petstore, in the folder pets select the request Info for a specific pet and navigate to the tab Authorization
- In the TYPE combo select the type No Auth
- Navigate to the Headers tab
- Add a Header with KEY
oauthtokenand VALUE{{token}} - Click Save to make the changes effective
Now we can export the collection again, according to the steps described in Exporting the collection
Now if everything went correctly, when executing our Makefile we will have the desired result with the API authenticated in Keycloak and validated by Kong.
Success
We saw in this tutorial how:
- Use Postman to import and execute definitions from an openapi spec
- How to create automated tests (collection) from Postman and export them for execution via the command line
- How to create API mocks with Prism
- How to use the swagger-to-kong tool to import openapi definitions into Kong (automatically creating Service and Route)
- How to use the swagger-to-kong tool to configure plugins in the generated routes
- How to use newman to run the collection created in Postman
- How to protect the API (authentication) with Keycloak, Kong + plugin jwt-keycloak and swagger-to-kong (plugins)
- How to run integration tests with Kong + Postgres, Keycloak, Prism, swagger-to-kong and newman
For the next steps, we will address the API authorization theme based on the Keycloak openapi x Realm roles spec, completing the API first cycle without a backend code implemented!