This guide describes setting out a credential storage mechanism that is shared between the root
account, aka releasr
, and jenkins
account. It intends to support and persist access credentials that are needed when authenticating with external services, primarily Docker Hub, for users wishing to access remote Docker images as part of their containerised services.
The following diagram aims to describe the control flow this setup intends to support.
This guide is a first, and a tad complicated sorry, pass at setting this up. It got us over the line for a couple of uses cases as well as working with automated build jobs.
The following dependencies need to be present on the target host machine prior to configuration.
The following environment variables have been added globally to vmjenkins
master node so they are made available to all build jobs. The main credentials store which is used by Docker credentials store is pass which in turn uses gpg2 to verify user based access controls when retrieving or storing credentials.
# location of docker client configuration file that is set relative to the user account.
$ sed -i '0,/{/s/{/{\n\t"credsStore": "pass",/' ${HOME}/.docker/config.json
$ export DOCKER_CONFIG = ${HOME}/.docker
$ export GNUPGHOME = ${HOME}/.gnupg
...
The following section describes how the required credentials are created. The credentials are used to identify key users who have been provided access to retrieve or store sensitive secrets. The first configuration steps involve creating the identities that will ultimately be given access to manage pass
backed secrets.
Adding the releasr
GPG Key to the local keystore is required. This key will be used whenever sensitive login credentials are stored within pass
. It can be used by any service or account with either root
privileges or like root
privileges.
1: Create a new public / private signature for releasr
using gpg2
.
$ gpg2 --gen-key
# real name: Release Team
# email: releasr@domain.co.uk
# comment: releasr
# pass phrase: gpg.releasr@domain.co.uk [LastPass]
# verify the key exists and is correct
$ gpg2 --list-key
2: Generate a revocation certificate for releasr
.
# THIS FILE NEEDS TO BE KEPT SECRET - EXAMPLE OUTPUT PATH: gpg-revoke.releasr@domain.co.uk [LastPass]
$ gpg2 --output ~/.gnupg/releasr-revoke.asc --gen-revoke <TR_KEY_FINGERPRINT>
3: Export the public / private keys for releasr
.
# export public key from the pubring.gpg
$ gpg2 --output ~/.gnupg/releasr.gpg --export <TR_KEY_FINGERPRINT>
# export public / private keys
# THIS FILE NEEDS TO BE KEPT SECRET - EXAMPLE OUTPUT PATH: gpg-sig.releasr@domain.co.uk [LastPass]
$ gpg2 --export <TR_KEY_FINGERPRINT> > ~/.gnupg/releasr.asc
Adding a Jenkins GPG Key to the local keystore is required. This key will be used during the execution of build jobs so approve access to Docker credentials held in pass so these can be managed properly. Please note that there are a few considerations to bare in mind when setting up the jenkins identity that are described in more detail below.
1: Prior to creating a new public / private GPG signature the jenkins
the user needs a supported shell
. This is a temporary measure as we need to be the actual jenkins
user when creating the new identity. Once the overall gpg2
and pass
configuration has been applied the jenkins
user should be modified so it doesnt have access to a shell
to improve this accounts security, e.g., usermod -s /sbin/nologin jenkins
.
# ran prior to configuration
$ usermod -s /bin/bash jenkins
# assume the jenkins user account
$ sudo su - jenkins
2: As su
does not open or apply a new pty
, or a pseudo-teletype
terminal, for the su
-ed user session this will mean that issuing gpg2 --gen-key
will ultimately fail, primarily at the point of entering a user pass phrase. To workaround this the following command needs to be issued prior to generating the public / private GPG signature for jenkins
.
# once this command is issued key generation can be accomplished
$ script /dev/null
2: As described in the 'Configuration Files' section please ensure the environment is supported.
3: Create a new public / private signature for jenkins
using gpg2
.
$ gpg2 --gen-key
# real name: Release Team CI
# email: jenkins@domain.co.uk
# comment: jenkins
# pass phrase: gpg.jenkins@domain.co.uk [LastPass]
# verify the key exists and is correct
$ gpg2 --list-key
4: Generate a revocation certificate for jenkins
.
# THIS FILE NEEDS TO BE KEPT SECRET - EXAMPLE OUTPUT PATH: gpg-revoke.jenkins@domain.co.uk [LastPass]
$ gpg2 --output ${HOME}/.gnupg/jenkins-revoke.asc --gen-revoke <JENKINS_KEY_FINGERPRINT>
5: Export the public / private keys for jenkins
.
# export public key from the pubring.gpg
$ gpg2 --output ${HOME}/.gnupg/jenkins.gpg --export <JENKINS_KEY_FINGERPRINT>
# export public / private keys
# THIS FILE NEEDS TO BE KEPT SECRET - EXAMPLE OUTPUT PATH: gpg-sig.jenkins@domain.co.uk [LastPass]
$ gpg2 --export <JENKINS_KEY_FINGERPRINT> > ${HOME}/.gnupg/jenkins-sig.asc
At this stage we should have valid gpg2
identities / signatures for both releasr
and jenkins
. It's worth saying at this point that the jenkins
user has been added to the docker
group and that docker
runs with like root
privileges. With this relationship established jenkins
avoids having to issue sudo
whenever it runs docker
commands.
releasr
user:${HOME}/.gnupg/releasr.gpg
jenkins
user:${HOME}/.gnupg/jenkins.gpg
The public signatures for both accounts will now need to be mutually trusted and locally signed as each user will need to access pass
stored credentials from various avenues. As each key is locally signed by each user: releasr
and jenkins
each of the accounts corresponding public keys need to be present, e.g., trying to access pass
as jenkins
without the releasr
public key being present and in the same GPG account will fail. This is done on both sides which may or may not be required.
1: Check the public fingerprints on both sides.
$ gpg2 --fingerprint <TR_KEY_FINGERPRINT>
$ gpg2 --fingerprint <JENKINS_KEY_FINGERPRINT>
2: As root
import the jenkins
public identity into the releasr
GPG2 account.
# as root import the jenkins public key
$ gpg2 --import /path/to/.gnupg/jenkins.gpg
# as root verify the imported key fingerprint matches expectations: check, etc
$ gpg2 --fingerprint <JENKINS_KEY_FINGERPRINT>
...
3: Locally sign jenkins
public key.
$ gpg2 --edit-key <JENKINS_KEY_FINGERPRINT>
> lsign
> save
4: Trust the jenkins
public key.
$ gpg2 --edit-key <JENKINS_KEY_FINGERPRINT>
> trust
# enter 4 = I trust fully
> 4
> save
5: As jenkins
import the releasr
public identity into the jenkins
GPG2 account.
# as jenkins import the releasr public key
$ gpg2 --import /path/to/.gnupg/releasr.gpg
# as root verify the imported key fingerprint matches expectations: check, etc
$ gpg2 --fingerprint <TR_KEY_FINGERPRINT>
...
6: Locally sign root
public key.
$ gpg2 --edit-key <TR_KEY_FINGERPRINT>
> lsign
> save
7: Trust the root
public key.
$ gpg2 --edit-key <TR_KEY_FINGERPRINT>
> trust
# enter 4 = I trust fully
> 4
> save
As the root
user initialise the pass
storage for use by both releasr
and jenkins
users. We will need to configure this as root
initially and then add jenkins
with shared access as a later step.
# initiate the pass store, here we're using docker-credential-helpers the default path
$ pass init -p docker-credential-helpers <TR_KEY_FINGERPRINT>
# list the newly created store
$ pass list
# ** on insert enter 'pass is initialised' when prompted
$ pass insert docker-credential-helpers/docker-pass-initialized-check
# ** on show enter the users gpg2 pass phrase when prompted
$ pass show docker-credential-helpers/docker-pass-initialized-check
# verify that docker-credential-pass has access to the empty pass store
$ docker-credential-pass list
(**) Please note that the reference to docker-credential-helpers/docker-pass-initialized-check
, creds @Ayrat-Kh, has presumed to be a setup step for accessing pass that both accounts can use to test their access to pass
against. Whenever the path (docker-credential-helpers
) is reinitialised the file maybe removed (docker-pass-initialized-check
). On build we will confer against the legit docker-credential-helpers
inspired path to check we have permission to decrypt it's content. If we can't access this it may mean the users gpg2
pass phrase has expired, there's a problem with the gpg2
identity or we never had permission in the first place.
$ pass
Password Store
└── docker-credential-helpers
└── aHR0cHM6Ly9pbmRleC5kb2NrZXIuaW8vdjEv
└── username
During build we check access to the credentials stored in pass
using this command. If it fails then a problem, as described above, has been encountered.
$ pass show docker-credential-helpers/aHR0cHM6Ly9pbmRleC5kb2NrZXIuaW8vdjEv
To provide both releasr
and jenkins
with access to pass
credentials for the docker-credential-helpers
path we now need to add jenkins
.
1: Add the jenkins
user to the existing pass
path for docker-credential-helpers
.
$ echo <JENKINS_KEY_FINGERPRINT> >> ${HOME}/.password-store/docker-credential-helpers/.gpg-id
2: Initialise the pass
store for the GPG IDs that have been granted access.
$ pass init -p docker-credential-helpers $(cat ${HOME}/.password-store/docker-credential-helpers/.gpg-id)
Please note that this may also need to be done if the server, username and/or secret password were to change, for example, this is the json
payload that Docker stores so if this changes then the reference to the credentials change.
{
"ServerURL": "https://index.docker.io/v1",
"Username": "username",
"Secret": "password"
}
3: Now login to Docker Hub from the root
account.
$ docker login --username <DOCKER_HUB_USERNAME>
Password: <DOCKER_HUB_PASSWORD>
4: Next, as jenkins
, create the same file structure for the jenkins
version of it's pass
store.
$ pass init -p docker-credential-helpers <JENKINS_KEY_FINGERPRINT>
5: As root
copy the pass files to the jenkins
account.
# hammer like copy
$ cp ${HOME}/.password-store/docker-credential-helpers/* /var/lib/jenkins/.password-store/docker-credential-helpers/
6: Now login to Docker Hub from the jenkins
account.
$ docker login --username <DOCKER_HUB_USERNAME>
Password: <DOCKER_HUB_PASSWORD>
You may need to export the gpg2
jenkins
owner trust file as jenkins
will not be ran interactively. If this is required please ensure all but jenkins
specific credentials are removed!
$ gpg2 --export-ownertrust > /path/to/.gnupg/owner-trust.txt
# depth: 0 valid: 2 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 2u
# 2u - ultimate trust level
$ gpg2 --update-trustdb
The following key needs to be loaded prior to the jenkins
CI service being started.
$ gpg2 --batch --import /path/to/.gnupg/jenkins.gpg
$ gpg2 --batch --import /path/to/.gnupg/releasr.gpg
$ gpg2 --import-ownertrust /path/to/.gnupg/owner-trust.txt
A custom Unit File has been provided to ensure that preset pass phrases are referenced on boot. This file can be found at the following location.
Each time the server is rebooted the following commands will need to be ran. This is currently a manual process but will soon be automated.
# persist releasr gpg pass phrase
$ gpg2 --fingerprint --fingerprint <TR_KEY_FINGERPRINT>
$ /usr/libexec/gpg-preset-passphrase --preset --passphrase <RELEASR_PASS_PHRASE> <LONG_TR_SUB_KEY_FINGERPRINT>
# persist jenkins gpg pass phrase
$ gpg2 --fingerprint --fingerprint <JENKINS_KEY_FINGERPRINT>
$ /usr/libexec/gpg-preset-passphrase --preset --passphrase <JENKINS_PASS_PHRASE> <LONG_JENKINS_SUB_KEY_FINGERPRINT>
# or forget them
$ /usr/libexec/gpg-preset-passphrase --forget <LONG_TR_SUB_KEY_FINGERPRINT>
$ /usr/libexec/gpg-preset-passphrase --forget <LONG_JENKINS_SUB_KEY_FINGERPRINT>
The following command may also be used to protect the pass phrase.
$ echo <RELEASR_PASS_PHRASE> | gpg --batch --pinentry-mode loopback --passphrase-fd 0 -d /path/to/.gnupg/releasr.gpg
$ echo <JENKINS_PASS_PHRASE> | gpg --batch --pinentry-mode loopback --passphrase-fd 0 -d /path/to/.gnupg/jenkins.gpg
Each account as a reference to GPG and GPG Agent configuration files. These are pretty much identical for both root
and jenkins
and can be found at the following locations.
These are some of the common errors experienced during execution of gpg2
and pass
/ docker-credential-helpers
.
This error indicates that the currently active GPG account was not able to access the pass
backed credentials store. This potentially means the pass phrase has expired or the user doesnt currently have permission to access this path. This is likely to be experienced when running pass show docker-credential-helpers/aHR0cHM6Ly9pbmRleC5kb2NrZXIuaW8vdjEv
with an expired or invalid access route.
pass is uninitialised
This error indicates that the currently active GPG account doesnt have access to the public key of the corresponding account: releasr
or jenkins
. This potentially means the corresponding GPG public key was not found via the users GPG account.
Error saving credentials: error storing credentials - err: exit status 1, out: `exit status 1: gpg: <KEY_FINGERPRINT>: skipped: No public key
This error indicates that the currently active GPG account doesnt have access to a locally signed signature for the corresponding account: releasr
or jenkins
. This potentially means the corresponding GPG public key will need to be locally signed before it can be used.
Error saving credentials: error storing credentials - err: exit status 1, out: `exit status 1: gpg: <KEY_FINGERPRINT>: There is no assurance this key belongs to the named user
- https://eklitzke.org/using-gpg-agent-effectively
- https://davesteele.github.io/gpg/2014/09/20/anatomy-of-a-gpg-key/
- docker/docker-credential-helpers#102
- https://medium.com/@jon_gille/decrypting-secrets-in-your-ci-cd-pipeline-c57da11e1794
- https://medium.com/@davidpiegza/using-pass-in-a-team-1aa7adf36592
- https://thoughtbot.com/blog/pgp-and-you
- https://github.com/rstacruz/cheatsheets/blob/master/gnupg.md
- https://lists.gnupg.org/pipermail/gnupg-users/2018-February/059918.html
- Author: @dataday
- Owner: Release Team
- Jira: https://jira.domain.co.uk/browse/TR