Github became de-facto standard code storage platform for all types of organizations, from small startups to huge corporations. While a company evolves and grows it will face the need for keeping code organized, structured and secured. Of course, it can be done by a sysadmin person while you have one team and couple members. But what about 5 teams, 10 teams? At some point, it will become extremely hard to keep track on naming convention and access permissions for many users, teams, and repositories.
Luckily most of this problems can be solved by infrastructure management tool that has been around for a while: Terraform. Terraform provides an ability to manage infrastructure as a code on different platforms like AWS, Azure, Kubernetes and also Github.
The easiest way to integrate Terraform is to start using it from day 0. While is not always possible, I will cover a topic of existing organization migration later. For now let's create an organization with a couple teams, repositories, and projects. I will assume that you're familiar with Terraform basics. To learn about Terraform, follow the official Getting Started guide.
Due to provider limitation, we can't create organization using Terraform. For organization creation follow this guide. To generate a new personal access token, check this step-by-step tutorial. I recommend using the permission setup shown below.
Assuming we have organization and token created, we can continue to the provider configuration. Github provider allows to configure multiple parameters, such as token, organization name, organization base_url and insecure flag to skip SSL verification. For demo purposes, we only specify organization and token parameters linked to input variables.
variables.tf
variable "github_token" {
type = "string"
}
variable "github_organization" {
type = "string"
}
provider.tf
provider "github" {
token = "${var.github_token}"
organization = "${var.github_organization}"
}
Now we can create a simple organization configuration using available resource. Our demo organization will consist of a team with members and a repository with admin access for this team. Please use your own Github username for this demo.
main.tf
resource "github_team" "demo-team" {
name = "demo-team-1"
}
resource "github_team_membership" "member-1" {
team_id = "${github_team.demo-team.id}"
username = "<enter_your_username_here>"
}
resource "github_repository" "demo-app-repository" {
name = "demo-repository"
description = "This is Demo Repository"
private = false
auto_init = true
allow_merge_commit = false
}
resource "github_team_repository" "demo-app-repository-admin-access" {
team_id = "${github_team.demo-team.slug}"
repository = "${github_repository.demo-app-repository.name}"
permission = "admin"
}
To check and apply our configuration changes, we need to execute following terraform commands.
$ terraform init
Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "github" (1.3.0)...
...
* provider.github: version = "~> 1.3"
Terraform has been successfully initialized!
...
Now we can preview our changes before applying it to the infrastructure.
$ terraform plan
...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ github_repository.demo-app-repository
id: <computed>
allow_merge_commit: "false"
allow_rebase_merge: "true"
allow_squash_merge: "true"
archived: "false"
auto_init: "true"
default_branch: <computed>
description: "This is Demo Repository"
etag: <computed>
full_name: <computed>
git_clone_url: <computed>
html_url: <computed>
http_clone_url: <computed>
name: "demo-repository"
private: "false"
ssh_clone_url: <computed>
svn_url: <computed>
+ github_team.demo-team
id: <computed>
etag: <computed>
name: "demo-team-1"
privacy: "secret"
slug: <computed>
+ github_team_membership.member-1
id: <computed>
etag: <computed>
role: "member"
team_id: "${github_team.demo-team.id}"
username: "<your_username_will_be_here>"
+ github_team_repository.demo-app-repository-admin-access
id: <computed>
etag: <computed>
permission: "admin"
repository: "demo-repository"
team_id: "${github_team.demo-team.slug}"
Plan: 4 to add, 0 to change, 0 to destroy.
...
Now we can take a look at infrastructure changes that we are going to apply in the next step. Since we don't have anything pre-configured, all 4 resources are going to be created.
This step plays a crucial part in the deployment process since we can catch any inconsistencies or unnecessary changes before they will be pushed to an actual organization.
After validating that everything looks like we expected, we can move forward and deploy the changes to the organization. We'll be required to provide github_organization and github_token variables manually.
Otherwise, we can place these variables into testing.auto.tfvar so they will be loaded automatically.
Be careful and never commit your token into any public repository.
$ terraform apply
var.github_organization
Enter a value: <GITHUB_ORG_NAME>
var.github_token
Enter a value: <GITHUB_TOKEN>
...
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
...
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
Now we can go to the organization and see changes above applied:
The organization with one team, one repository, and one member
Team demo-team-1 with a user attached
Repository demo-repository with admin access for demo-team-1.
After making some changes, for example, downgrading demo-team-1 permissions to pull in demo-repository, we repeat plan and apply operations to update infrastructure.
main.tf
...
resource "github_team_repository" "demo-app-repository-admin-access" {
team_id = "${github_team.demo-team.slug}"
repository = "${github_repository.demo-app-repository.name}"
permission = "pull"
}
$ terraform plan
...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
~ github_team_repository.demo-app-repository-admin-access
permission: "admin" => "pull"
Plan: 0 to add, 1 to change, 0 to destroy.
...
$ terraform apply
...
github_team_repository.demo-app-repository-admin-access: Modifying... (ID: 3026096:demo-repository)
permission: "admin" => "pull"
github_team_repository.demo-app-repository-admin-access: Modifications complete after 1s (ID: 3026096:demo-repository)
Now new permission setting was applied to the repository.
As we can see Terraform provides a convenient way to store the whole Github organization configuration as a code. We can also version it and automate changes review and deployment with Jenkins and Github pull request hooks.
It's easy to commit building infrastructure-as-a-code from scratch, but can we move existing Github organization to Terraform resources? The answer is yes!
To import existing resource into Terraform first we need to create appropriate resource declaration in our configuration.
Let's add test-import github_repository resource into the configuration, but leave all fields blank. We need to create
main.tf
resource "github_repository" "test-import" {}
The next step is to import resource to the Terraform state with import command, where the first argument is a name of the resource and the second one is repository name in the organization. After the import is done, we can check resource configuration from the state.
$ terraform import github_repository.test-import test-import
github_repository.test-import: Importing from ID "test-import"...
github_repository.test-import: Import complete!
Imported github_repository (ID: test-import)
github_repository.test-import: Refreshing state... (ID: test-import)
Import successful!
...
$ terraform state show github_repository.test-import
id = test-import
allow_merge_commit = true
allow_rebase_merge = true
allow_squash_merge = true
archived = false
default_branch = master
description = Test Terraform Resource Import
etag = W/"16351eb92d2d64af974cdfe9c8250fa9"
full_name = demo-lo8AhK4i/test-import
git_clone_url = git://github.com/demo-lo8AhK4i/test-import.git
has_downloads = true
has_issues = true
has_wiki = true
homepage_url =
html_url = https://github.com/demo-lo8AhK4i/test-import
http_clone_url = https://github.com/demo-lo8AhK4i/test-import.git
name = test-import
private = false
ssh_clone_url = git@github.com:demo-lo8AhK4i/test-import.git
svn_url = https://github.com/demo-lo8AhK4i/test-import
topics.# = 0
Now we can move this configuration parameters to the configuration. The trick is to remove computed values from this configuration. Using github_repository resource documentation and terraform plan command, make sure that configuration copied correctly and won't cause any changes in the organization. The end result should be like this:
...
resource "github_repository" "test-import" {
allow_merge_commit = true
allow_rebase_merge = true
allow_squash_merge = true
archived = false
default_branch = "master"
description = "Test Terraform Resource Improt"
has_downloads = true
has_issues = true
has_wiki = true
homepage_url = ""
name = "test-import"
private = false
}
Be VERY cautions of some parameters, like auto_init for github_repository since it can cause full repository recreation and losing your data in this repo.
To avoid this issue always check your changes before applying configuration changes, repositories backend are highly recommended as well.
Here is a little bonus - a very nice article how to setup automated Git repositories backup with AWS S3 and Lambda.
Git Webhooks with AWS services
No matter how many users, repositories and teams are in an organization Terraform provides a good way to define, organize and version all kind of resources and permissions for Github organization and beyond, as well as recreate organization structure from scratch in any time.
All operations from creating new repositories to managing access rights can be automated with automated Jenkins pipeline and Github Pull Request, so there won't be any need to do any of this manually anymore.