jeremyyeo/README.md

## README.md

      
    Raw
  

              README.md
            
          
    Accessing private GitHub repositories (dbt packages) using GitHub Apps installation access tokens


This is written in the context of accessing private dbt packages for dbt projects.

There are 2 ways to access private GitHub repostories:

GitHub personal access tokens (PAT).

These are user specific and once created, can then used via git clone https://<pat>@github.com/user-org/private-repo.git.


GitHub Apps installation access tokens.

These are more involved and require creating a GitHub App, installing the GitHub App into the GitHub Organization/User, generating jwt tokens / installation access tokens.
Once an installation access token is generated, it can then be used via git clone https://x-access-token:<installation-access-token>@github.com/user-org/private-repo.git.
Installation access tokens only live for 1 hour¹ - so you will need some mechanism of regenerating them.


This gist covers the second method above.²
Create a GitHub App and install it into the GitHub user or organization.


Make sure you have a private repository.


Create a GitHub App.

You can choose to create a GitHub App owned by a user or organization - in this case, I'm creating the app in the organization that also owns my private repository above.

There are a lot of configuration options for these GitHub Apps (which is certainly out of scope for this write up) but the main one here is to allow the app to access repository contents.

If you want to follow how I've set up my app here, I basically have the following set (essentially a barebones GitHub App):

No Callback URL.
No Setup URL.
Webhook is not active - so no Webhook URL.


Take note of your GitHub Apps App ID and also be sure to "generate a private key" as the banner suggests. When you generate a private key, the browser should automatically download a pem key. The App ID and this private key will be used to generate a JWT token in a subsequent step.


Install the GitHub App created.

Here I'm installing my "dbt-app-for-repo-access" app into my organization and only giving it access to the private repo mentioned above in (1). Note that in (2), I created the app to be only installable into this organization - this is why you do not see any other users or organizations that I am apart of.

Generate an installation access token.

Now that our app has access to the private repository, we will need to generate an "installation access token" in multiple stages with the help of Python (GitHub's documentation uses Ruby + cURL).

Generate a JWT token.

# generate_jwt.py

import jwt # pip install pyjwt
import time
from cryptography.hazmat.primitives import serialization

# The App ID from (3) above.
GITHUB_APP_ID = 237118

# The path to the private key that we downloaded from (3) above.
pk = open("/Users/jeremy/dbt-app-for-repo-access.2022-09-12.private-key.pem", "r").read()
key = serialization.load_pem_private_key(pk.encode(), password=None)

issued_time = int(time.time()) - 60
expiry_time = int(time.time()) + (9 * 60)

payload_data = {
    # issued at time, 60 seconds in the past to allow for clock drift
    "iat": issued_time,
    # JWT expiration time (10 minute maximum)
    "exp": expiry_time,
    # GitHub App's identifier
    "iss": GITHUB_APP_ID,
}

jwt = jwt.encode(payload=payload_data, key=key, algorithm="RS256")
print(jwt)
$ python generate_jwt.py

eyJ0<TRUNCATED>Sp-Q


Use the JWT token to list the installations of the GitHub App.

# get_installations.py

import requests # pip install requests

# jwt from the previous step.
JWT = "eyJ0<TRUNCATED>Sp-Q"

header = {"Authorization": f"Bearer {JWT}", "Accept": "application/vnd.github+json"}

r = requests.get(url="https://api.github.com/app/installations", headers=header)
print(r.json())
$ python get_installations.py

[
    {
        'id': 29130729,
        'account': {
            'login': 'picturatechnica',
            'id': 56277636,
            ...
            'type': 'Organization',
            'site_admin': False
        },
        'repository_selection': 'all',
        ...
        'suspended_at': None
    }
]

The above response is truncated but what you're looking for is the Installation ID - here it is 29130729. If the app has multiple installations - there would be multiple objects in the returned array.

Use the Installation ID and JWT to retrieve an installation access token.

# get_installation_token.py

import requests

# jwt from above.
JWT = "eyJ0<TRUNCATED>Sp-Q"

# Installation ID from the previous step.
INSTALLATION_ID = 29130729

header = {"Authorization": f"Bearer {JWT}", "Accept": "application/vnd.github+json"}

r = requests.post(
    url=f"https://api.github.com/app/installations/{INSTALLATION_ID}/access_tokens",
    headers=header,
)
print(r.json())
$ python get_installation_token.py
{'token': 'ghs_<REDACTED>', 'expires_at': '2022-09-12T09:23:20Z', 'permissions': {'contents': 'read', 'metadata': 'read'}, 'repository_selection': 'all'}


Note here the "expires_at" will be 1 hour from when you requested the installation access token - there is no way to change this.


With our newly generated installation access token - use it to clone the private repository.

$ git clone https://x-access-token:ghs_<REDACTED>@github.com/picturatechnica/top-secret-package.git

Cloning into 'top-secret-package'...
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 6 (delta 0), reused 6 (delta 0), pack-reused 0
Receiving objects: 100% (6/6), done.

Use the installation access token to add a private dbt package to your project.

# packages.yml

packages:
  - git: "https://x-access-token:ghs_<REDACTED>@github.com/picturatechnica/top-secret-package.git"
    revision: main

Note that you should use dbt Cloud (or local) env vars³ to handle secrets here instead of what I have done above by simply pasting the access token straight into the file but this is for demonstration only.
Caveats

Finally, I should iterate again that installation access tokens are short lived (60 minutes) - so once that time has passed, you will have to go through the steps of regenerating it again plus also edit your environment variable and set it to a new value - this means that this method is not very viable unless you have a mechanism of rotating the tokens and setting it to the new value in your dbt project (likely this means only users who have dbt-core deployed on their own infrastructure will want to take this route).
Footnotes


"Installation tokens expire one hour from the time you create them" - https://docs.github.com/en/rest/apps/apps#create-an-installation-access-token-for-an-app ↩


Most of the instructions are from https://docs.github.com/en/developers/apps/building-github-apps/authenticating-with-github-apps ↩


https://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-environment-variables#handling-secrets ↩