This guide describes setting out a credential storage mechanism that is shared between the root account, aka releasr, and jenkins account. It intends to support and persist access credentials that are needed when authenticating with external services, primarily Docker Hub, for users wishing to access remote Docker images as part of their containerised services.
The following diagram aims to describe the control flow this setup intends to support.
This guide is a first, and a tad complicated sorry, pass at setting this up. It got us over the line for a couple of uses cases as well as working with automated build jobs.
The following dependencies need to be present on the target host machine prior to configuration.
The following environment variables have been added globally to vmjenkins master node so they are made available to all build jobs. The main credentials store which is used by Docker credentials store is pass which in turn uses gpg2 to verify user based access controls when retrieving or storing credentials.
# location of docker client configuration file that is set relative to the user account.
$ sed -i '0,/{/s/{/{\n\t"credsStore": "pass",/' ${HOME}/.docker/config.json
$ export DOCKER_CONFIG = ${HOME}/.docker
$ export GNUPGHOME = ${HOME}/.gnupg
...The following section describes how the required credentials are created. The credentials are used to identify key users who have been provided access to retrieve or store sensitive secrets. The first configuration steps involve creating the identities that will ultimately be given access to manage pass backed secrets.
Adding the releasr GPG Key to the local keystore is required. This key will be used whenever sensitive login credentials are stored within pass. It can be used by any service or account with either root privileges or like root privileges.
1: Create a new public / private signature for releasr using gpg2.
$ gpg2 --gen-key
# real name: Release Team
# email: releasr@domain.co.uk
# comment: releasr
# pass phrase: gpg.releasr@domain.co.uk [LastPass]
# verify the key exists and is correct
$ gpg2 --list-key2: Generate a revocation certificate for releasr.
# THIS FILE NEEDS TO BE KEPT SECRET - EXAMPLE OUTPUT PATH: gpg-revoke.releasr@domain.co.uk [LastPass]
$ gpg2 --output ~/.gnupg/releasr-revoke.asc --gen-revoke <TR_KEY_FINGERPRINT>3: Export the public / private keys for releasr.
# export public key from the pubring.gpg
$ gpg2 --output ~/.gnupg/releasr.gpg --export <TR_KEY_FINGERPRINT>
# export public / private keys
# THIS FILE NEEDS TO BE KEPT SECRET - EXAMPLE OUTPUT PATH: gpg-sig.releasr@domain.co.uk [LastPass]
$ gpg2 --export <TR_KEY_FINGERPRINT> > ~/.gnupg/releasr.ascAdding a Jenkins GPG Key to the local keystore is required. This key will be used during the execution of build jobs so approve access to Docker credentials held in pass so these can be managed properly. Please note that there are a few considerations to bare in mind when setting up the jenkins identity that are described in more detail below.
1: Prior to creating a new public / private GPG signature the jenkins the user needs a supported shell. This is a temporary measure as we need to be the actual jenkins user when creating the new identity. Once the overall gpg2 and pass configuration has been applied the jenkins user should be modified so it doesnt have access to a shell to improve this accounts security, e.g., usermod -s /sbin/nologin jenkins.
# ran prior to configuration
$ usermod -s /bin/bash jenkins
# assume the jenkins user account
$ sudo su - jenkins2: As su does not open or apply a new pty, or a pseudo-teletype terminal, for the su-ed user session this will mean that issuing gpg2 --gen-key will ultimately fail, primarily at the point of entering a user pass phrase. To workaround this the following command needs to be issued prior to generating the public / private GPG signature for jenkins.
# once this command is issued key generation can be accomplished
$ script /dev/null2: As described in the 'Configuration Files' section please ensure the environment is supported.
3: Create a new public / private signature for jenkins using gpg2.
$ gpg2 --gen-key
# real name: Release Team CI
# email: jenkins@domain.co.uk
# comment: jenkins
# pass phrase: gpg.jenkins@domain.co.uk [LastPass]
# verify the key exists and is correct
$ gpg2 --list-key4: Generate a revocation certificate for jenkins.
# THIS FILE NEEDS TO BE KEPT SECRET - EXAMPLE OUTPUT PATH: gpg-revoke.jenkins@domain.co.uk [LastPass]
$ gpg2 --output ${HOME}/.gnupg/jenkins-revoke.asc --gen-revoke <JENKINS_KEY_FINGERPRINT>5: Export the public / private keys for jenkins.
# export public key from the pubring.gpg
$ gpg2 --output ${HOME}/.gnupg/jenkins.gpg --export <JENKINS_KEY_FINGERPRINT>
# export public / private keys
# THIS FILE NEEDS TO BE KEPT SECRET - EXAMPLE OUTPUT PATH: gpg-sig.jenkins@domain.co.uk [LastPass]
$ gpg2 --export <JENKINS_KEY_FINGERPRINT> > ${HOME}/.gnupg/jenkins-sig.ascAt this stage we should have valid gpg2 identities / signatures for both releasr and jenkins. It's worth saying at this point that the jenkins user has been added to the docker group and that docker runs with like root privileges. With this relationship established jenkins avoids having to issue sudo whenever it runs docker commands.
releasruser:${HOME}/.gnupg/releasr.gpgjenkinsuser:${HOME}/.gnupg/jenkins.gpg
The public signatures for both accounts will now need to be mutually trusted and locally signed as each user will need to access pass stored credentials from various avenues. As each key is locally signed by each user: releasr and jenkins each of the accounts corresponding public keys need to be present, e.g., trying to access pass as jenkins without the releasr public key being present and in the same GPG account will fail. This is done on both sides which may or may not be required.
1: Check the public fingerprints on both sides.
$ gpg2 --fingerprint <TR_KEY_FINGERPRINT>
$ gpg2 --fingerprint <JENKINS_KEY_FINGERPRINT>2: As root import the jenkins public identity into the releasr GPG2 account.
# as root import the jenkins public key
$ gpg2 --import /path/to/.gnupg/jenkins.gpg
# as root verify the imported key fingerprint matches expectations: check, etc
$ gpg2 --fingerprint <JENKINS_KEY_FINGERPRINT>
...3: Locally sign jenkins public key.
$ gpg2 --edit-key <JENKINS_KEY_FINGERPRINT>
> lsign
> save4: Trust the jenkins public key.
$ gpg2 --edit-key <JENKINS_KEY_FINGERPRINT>
> trust
# enter 4 = I trust fully
> 4
> save5: As jenkins import the releasr public identity into the jenkins GPG2 account.
# as jenkins import the releasr public key
$ gpg2 --import /path/to/.gnupg/releasr.gpg
# as root verify the imported key fingerprint matches expectations: check, etc
$ gpg2 --fingerprint <TR_KEY_FINGERPRINT>
...6: Locally sign root public key.
$ gpg2 --edit-key <TR_KEY_FINGERPRINT>
> lsign
> save7: Trust the root public key.
$ gpg2 --edit-key <TR_KEY_FINGERPRINT>
> trust
# enter 4 = I trust fully
> 4
> saveAs the root user initialise the pass storage for use by both releasr and jenkins users. We will need to configure this as root initially and then add jenkins with shared access as a later step.
# initiate the pass store, here we're using docker-credential-helpers the default path
$ pass init -p docker-credential-helpers <TR_KEY_FINGERPRINT>
# list the newly created store
$ pass list
# ** on insert enter 'pass is initialised' when prompted
$ pass insert docker-credential-helpers/docker-pass-initialized-check
# ** on show enter the users gpg2 pass phrase when prompted
$ pass show docker-credential-helpers/docker-pass-initialized-check
# verify that docker-credential-pass has access to the empty pass store
$ docker-credential-pass list(**) Please note that the reference to docker-credential-helpers/docker-pass-initialized-check, creds @Ayrat-Kh, has presumed to be a setup step for accessing pass that both accounts can use to test their access to pass against. Whenever the path (docker-credential-helpers) is reinitialised the file maybe removed (docker-pass-initialized-check). On build we will confer against the legit docker-credential-helpers inspired path to check we have permission to decrypt it's content. If we can't access this it may mean the users gpg2 pass phrase has expired, there's a problem with the gpg2 identity or we never had permission in the first place.
$ pass
Password Store
└── docker-credential-helpers
└── aHR0cHM6Ly9pbmRleC5kb2NrZXIuaW8vdjEv
└── usernameDuring build we check access to the credentials stored in pass using this command. If it fails then a problem, as described above, has been encountered.
$ pass show docker-credential-helpers/aHR0cHM6Ly9pbmRleC5kb2NrZXIuaW8vdjEvTo provide both releasr and jenkins with access to pass credentials for the docker-credential-helpers path we now need to add jenkins.
1: Add the jenkins user to the existing pass path for docker-credential-helpers.
$ echo <JENKINS_KEY_FINGERPRINT> >> ${HOME}/.password-store/docker-credential-helpers/.gpg-id2: Initialise the pass store for the GPG IDs that have been granted access.
$ pass init -p docker-credential-helpers $(cat ${HOME}/.password-store/docker-credential-helpers/.gpg-id)Please note that this may also need to be done if the server, username and/or secret password were to change, for example, this is the json payload that Docker stores so if this changes then the reference to the credentials change.
{
"ServerURL": "https://index.docker.io/v1",
"Username": "username",
"Secret": "password"
}3: Now login to Docker Hub from the root account.
$ docker login --username <DOCKER_HUB_USERNAME>
Password: <DOCKER_HUB_PASSWORD>4: Next, as jenkins, create the same file structure for the jenkins version of it's pass store.
$ pass init -p docker-credential-helpers <JENKINS_KEY_FINGERPRINT>5: As root copy the pass files to the jenkins account.
# hammer like copy
$ cp ${HOME}/.password-store/docker-credential-helpers/* /var/lib/jenkins/.password-store/docker-credential-helpers/6: Now login to Docker Hub from the jenkins account.
$ docker login --username <DOCKER_HUB_USERNAME>
Password: <DOCKER_HUB_PASSWORD>You may need to export the gpg2 jenkins owner trust file as jenkins will not be ran interactively. If this is required please ensure all but jenkins specific credentials are removed!
$ gpg2 --export-ownertrust > /path/to/.gnupg/owner-trust.txt
# depth: 0 valid: 2 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 2u
# 2u - ultimate trust level
$ gpg2 --update-trustdbThe following key needs to be loaded prior to the jenkins CI service being started.
$ gpg2 --batch --import /path/to/.gnupg/jenkins.gpg
$ gpg2 --batch --import /path/to/.gnupg/releasr.gpg
$ gpg2 --import-ownertrust /path/to/.gnupg/owner-trust.txtA custom Unit File has been provided to ensure that preset pass phrases are referenced on boot. This file can be found at the following location.
Each time the server is rebooted the following commands will need to be ran. This is currently a manual process but will soon be automated.
# persist releasr gpg pass phrase
$ gpg2 --fingerprint --fingerprint <TR_KEY_FINGERPRINT>
$ /usr/libexec/gpg-preset-passphrase --preset --passphrase <RELEASR_PASS_PHRASE> <LONG_TR_SUB_KEY_FINGERPRINT>
# persist jenkins gpg pass phrase
$ gpg2 --fingerprint --fingerprint <JENKINS_KEY_FINGERPRINT>
$ /usr/libexec/gpg-preset-passphrase --preset --passphrase <JENKINS_PASS_PHRASE> <LONG_JENKINS_SUB_KEY_FINGERPRINT>
# or forget them
$ /usr/libexec/gpg-preset-passphrase --forget <LONG_TR_SUB_KEY_FINGERPRINT>
$ /usr/libexec/gpg-preset-passphrase --forget <LONG_JENKINS_SUB_KEY_FINGERPRINT>The following command may also be used to protect the pass phrase.
$ echo <RELEASR_PASS_PHRASE> | gpg --batch --pinentry-mode loopback --passphrase-fd 0 -d /path/to/.gnupg/releasr.gpg
$ echo <JENKINS_PASS_PHRASE> | gpg --batch --pinentry-mode loopback --passphrase-fd 0 -d /path/to/.gnupg/jenkins.gpgEach account as a reference to GPG and GPG Agent configuration files. These are pretty much identical for both root and jenkins and can be found at the following locations.
These are some of the common errors experienced during execution of gpg2 and pass / docker-credential-helpers.
This error indicates that the currently active GPG account was not able to access the pass backed credentials store. This potentially means the pass phrase has expired or the user doesnt currently have permission to access this path. This is likely to be experienced when running pass show docker-credential-helpers/aHR0cHM6Ly9pbmRleC5kb2NrZXIuaW8vdjEv with an expired or invalid access route.
pass is uninitialised
This error indicates that the currently active GPG account doesnt have access to the public key of the corresponding account: releasr or jenkins. This potentially means the corresponding GPG public key was not found via the users GPG account.
Error saving credentials: error storing credentials - err: exit status 1, out: `exit status 1: gpg: <KEY_FINGERPRINT>: skipped: No public key
This error indicates that the currently active GPG account doesnt have access to a locally signed signature for the corresponding account: releasr or jenkins. This potentially means the corresponding GPG public key will need to be locally signed before it can be used.
Error saving credentials: error storing credentials - err: exit status 1, out: `exit status 1: gpg: <KEY_FINGERPRINT>: There is no assurance this key belongs to the named user
- https://eklitzke.org/using-gpg-agent-effectively
- https://davesteele.github.io/gpg/2014/09/20/anatomy-of-a-gpg-key/
- docker/docker-credential-helpers#102
- https://medium.com/@jon_gille/decrypting-secrets-in-your-ci-cd-pipeline-c57da11e1794
- https://medium.com/@davidpiegza/using-pass-in-a-team-1aa7adf36592
- https://thoughtbot.com/blog/pgp-and-you
- https://github.com/rstacruz/cheatsheets/blob/master/gnupg.md
- https://lists.gnupg.org/pipermail/gnupg-users/2018-February/059918.html
- Author: @dataday
- Owner: Release Team
- Jira: https://jira.domain.co.uk/browse/TR