I need to give access to a private dependency. It can happen for continuous integration or deployment.
Here we use python and github, using the services CircleCI and Heroku. However, the principles applies everywhere.
See https://developer.github.com/v3/guides/managing-deploy-keys/
There are 4 ways of granting access to a private dependency, but deploy keys are a good compromise in term of security and ease of use for projects that do not require too many dependencies (in that case, prefer a machine user). In any case, do not use username/password of a developer account or oauth token as they do not provide privilege limitation.
Create a deploy key:
ssh-keygen -t rsa -b 4096 -C "myself@my_company.com"
Give the public part to gihub.
Give the private part to the service needing access. See below.
Whatever the service or the technology that I use, the goal is to access the git repo using ssh, using the deploy key.
Obviously, I do not want to put the deploy key in the repo. But most services (CI, deployment) provide a way to set protected environment variables that can be used at build time. The key can be encoded using base64:
cat deploy-key | base64
cat deploy-key.pub | base64
Most services also provide a way to tailor the build procedure. This is needed to configure ssh to use the deploy key.
Set the deploy key using env variables, encode with base64.
In config.yml, add a step:
echo $DEPLOY_KEY_PRIVATE | base64 --decode > ~/.ssh/deploy-key
chmod 400 ~/.ssh/deploy-key
echo $DEPLOY_KEY_PUBLIC | base64 --decode > ~/.ssh/deploy-key.pub
ssh-add ~/.ssh/deploy-key
# Run this to check which private key is used. If the checkout key is used,
# github replies "Hi my_org/my_package". If the deploy key is used as wished,
# github replies "Hi my_org/my_dependency".
#ssh -i ~/.ssh/deploy-key -T git@github.com || true
# Now pip connects to git+ssh using the deploy key
export GIT_SSH_COMMAND="ssh -i ~/.ssh/deploy-key"
pip install -r requirements.txt
requirements.txt can be something like:
# The purpose of this file is to install the private dependency *before*
# setup.py is run.
# Be sure ssh is configured to use a ssh key with read permission to the repo.
git+ssh://git@github.com/my_org/my_dependency@1.0.10
# Run setup.py. The private dependency is already installed with the good
# version so pip doesn't try to fetch it from PyPI.
--editable .
and setup.py does not care about the dependency beeing private:
from distutils.core import setup
setup(
name='my_package',
version='1.0',
packages=[
'my_package',
],
install_requires=[
# Beware, the following package is a private dependency.
# Python provides several way to install private dependencies, none
# are really satisfactory.
# 1. Use dependency_links / --process-dependency-links. Good luck with
# that!
# 2. Maintain a private package repository. Good luck with that!
# 3. Install the private dependency separately before setup.py is run.
# This is now the prefered way. Be sure that ssh is properly
# configured to use a ssh key with read permission to the github repo
# of the private dependency, then run:
# `pip install -r requirements.txt`
'my_dependency==1.0.10',
... # my normal dependencies
'unidecode==1.0.22',
'uwsgi==2.0.15',
'nose==1.3.7', # tests
'flake8==3.5.0', # style
],
)
For python, there is no need to write a custom buildpack. First, set the deploy key using env variables, encode with base64.
Then add the hook bin/pre_compile:
# This script configures ssh on Heroku to use the deploy key.
# This is needed to install private dependencies.
#
# Note that this does not work with Heroku review apps. Indeed review apps can
# inherits env variables from their parents, but they access their values after
# the build. You would need a way to pass the ssh key to this script another
# way.
#
# See also
# * https://stackoverflow.com/questions/21297755/heroku-python-dependencies-in-private-repos-without-storing-my-password#
# * https://github.com/bjeanes/ssh-private-key-buildpack
# Ensure we have an ssh folder
if [ ! -d ~/.ssh ]; then
mkdir -p ~/.ssh
chmod 700 ~/.ssh
fi
# Create the key files
cat $ENV_DIR/DEPLOY_KEY | base64 --decode > ~/.ssh/deploy-key
chmod 400 ~/.ssh/deploy-key
cat $ENV_DIR/DEPLOY_KEY | base64 --decode > ~/.ssh/deploy-key.pub
#ssh-add ~/.ssh/deploy-key
# If you want to disable host verification, you could use that.
#ssh -oStrictHostKeyChecking=no -T git@github.com 2>&1
# Run that if you want to check that ssh uses the correct key.
#ssh -i ~/.ssh/deploy-key -T git@github.com || true
# Configure ssh to use the correct deploy key when connecting to github.
# Disables host verification.
echo -e "Host github.com\n"\
" IdentityFile ~/.ssh/deploy-key\n"\
" IdentitiesOnly yes\n"\
" UserKnownHostsFile=/dev/null\n"\
" StrictHostKeyChecking no"\
>> ~/.ssh/config
# Unfortunately this does not seem to work.
#export GIT_SSH_COMMAND="ssh -i ~/.ssh/deploy-key"
# The vanilla python buildpack can now install all the dependencies in
# requirement.txt