Skip to content

Instantly share code, notes, and snippets.

@StevenACoffman
Forked from dmitshur/gist:6927554
Last active October 10, 2024 12:47
Show Gist options
  • Save StevenACoffman/866b06ed943394fbacb60a45db5982f2 to your computer and use it in GitHub Desktop.
Save StevenACoffman/866b06ed943394fbacb60a45db5982f2 to your computer and use it in GitHub Desktop.
How to `go get` private repos using SSH key auth instead of password auth.

Set GOPRIVATE to match your github organization

I keep fixing this up, but if it fails for you, check if these are better maintained https://tip.golang.org/cmd/go/#hdr-Configuration_for_downloading_non_public_code and https://golang.org/ref/mod#private-modules.

Cloning the repo using one of the below techniques should work correctly but you may still be getting an unrecognized import error.

As it stands for Go v1.13, I found in the doc that we should use the GOPRIVATE variable like so:

GOPRIVATE=github.com/ORGANISATION_OR_USER_NAME go get -u -f github.com/ORGANISATION_OR_USER_NAME/REPO_NAME

The 'go env -w' command (see 'go help env') can be used to set these variables for future go command invocations.

How to go get private repos using SSH key auth instead of password auth.

$ git config --global url."git@github.com:".insteadOf "https://github.com/"
$ git config --global url."git@gitlab.com:".insteadOf "https://gitlab.com/"
$ cat ~/.gitconfig
[url "git@github.com:"]
	insteadOf = https://github.com/
[url "git@gitlab.com:"]
	insteadOf = https://gitlab.com/
$ go get -u -f github.com/private/repo && echo Success!
Success!

In Git ssh://git@host/resource, git+ssh://git@host/resource, ssh+git://git@host/resource and git@host:resource are all the same protocol.

💡 NOTE: go get uses HTTPS when cloning a repository. So configuring insteadOf rewriting rules will cause problems with go get -u github.com/private/repo later on, since a check at update time to verify that the local package's remote repository matches that of its custom import. This change was introduced in Go 1.4.

The workaround for that rewriting check would be to use go get -u -f github.com/private/repo:

From the go get options here:

The -f flag, valid only when -u is set, forces get -u not to verify that each package has been checked out from the source control repository implied by its import path. This can be useful if the source is a local fork of the original.

Sources:

How to go get private repos using HTTPS via Personal Access Tokens

An alternative to using git@github.com is to generate a personal access token on your GitHub account (or for a service account), grant it repo access, and then use the following instead:

# Github
git config --global url."https://oauth2:${GITHUB_TOKEN}@github.com/".insteadOf "https://github.com/"

# Gitlab personal access token
git config --global url."https://gitlab-ci-token:${GITLAB_PERSONAL_ACCESS_TOKEN}@gitlab.com/".insteadOf "https://gitlab.com/"

# BitBucket is a weird place. You need user slug rather than username
git config --global url."https://user%40bitbucket.com:${BITBUCKET_ACCESS_TOKEN}@bitbucket.com/".insteadOf "https://bitbucket.com/"

You could also just store credentials in a credential helper like the ~/.git-credentials file.

Both should still work with go get -u, and also works with Docker builds.

Git Credential Manager (GCM) alternative

Git Credential Manager (GCM) is another way to store your credentials securely and connect to GitHub over HTTPS. With GCM, you don't have to manually create and store a personal access token, as GCM manages authentication on your behalf, including 2FA (two-factor authentication).

  1. Install Git using Homebrew:
brew install git
  1. Install GCM using Homebrew:
brew install --cask git-credential-manager

For macOS, you don't need to run git config because GCM automatically configures Git for you.

The next time you clone an HTTPS URL that requires authentication, Git will prompt you to log in using a browser window. You may first be asked to authorize an OAuth app. If your account or organization requires two-factor auth, you'll also need to complete the 2FA challenge.

Once you've authenticated successfully, your credentials are stored in the macOS keychain and will be used every time you clone an HTTPS URL. Git will not require you to type your credentials in the command line again unless you change your credentials.

For more information or to report issues with GCM, see the official GCM docs at "Git Credential Manager."

@pablodz
Copy link

pablodz commented Sep 19, 2021

Nice

@Shellbye
Copy link

And helpful

@risyadaji
Copy link

risyadaji commented Aug 24, 2022

it works!

@StevenACoffman
Copy link
Author

current known configuration for MFA is here config available here: golang/go#49515 (comment)

@shuchow
Copy link

shuchow commented Sep 30, 2022

Thanks for writing this up. Question for you. - let's say I have a project that uses a module that's in a private repo. I set my GOPRIVATE as you describe.

At the git config step (git config --global url."git@github.com:".insteadOf "https://github.com/"), can't I replace the --global flag with --local and make that rewrite specific only to my project?

@StevenACoffman
Copy link
Author

@shuchow It depends on your use case. Inside the context of that configured git repository, it will work fine. However, if you are ever trying to use the Go tools to reference the private module outside of the context of that git configured repository, it will fail. For instance, when you might go install <modulename> a binary from that private repository, or if you have another git repository that is not configured.

For me, I always want ssh, so git config --global has less potential to be problematic.

@scottstensland
Copy link

for me I was incorrectly using a delimiter of character : instead of the comma when defining env var

export GOPRIVATE=github.com/foo/bar_private_repoAAA,github.com/foo/bar_private_repoBBB

@hmit208
Copy link

hmit208 commented Dec 21, 2022

After change url to git@gitlab.com I can clone the repo but cannot go get
Got this error:

	remote: 
	remote: ========================================================================
	remote: 
	remote: ERROR: The project you were looking for could not be found or you don't have permission to view it.
	
	remote: 
	remote: ========================================================================
	remote: 
	fatal: Could not read from remote repository.
	
	Please make sure you have the correct access rights
	and the repository exists.

Does anyone have the same error like that?

@StevenACoffman
Copy link
Author

@hmit208 did you do this first:

# set GITLAB_PERSONAL_ACCESS_TOKEN environment variable to Gitlab personal access token
git config --global url."https://gitlab-ci-token:${GITLAB_PERSONAL_ACCESS_TOKEN}@gitlab.com/".insteadOf "https://gitlab.com/"

@pplanel
Copy link

pplanel commented Jan 6, 2023

After change url to git@gitlab.com I can clone the repo but cannot go get Got this error:

	remote: 
	remote: ========================================================================
	remote: 
	remote: ERROR: The project you were looking for could not be found or you don't have permission to view it.
	
	remote: 
	remote: ========================================================================
	remote: 
	fatal: Could not read from remote repository.
	
	Please make sure you have the correct access rights
	and the repository exists.

Does anyone have the same error like that?

You are using sub-groups in Gitlab? If that's the case you should follow this issue:
https://gitlab.com/gitlab-org/gitlab/-/issues/36354

@AdamsNimbus
Copy link

Extra thing to try (that worked for me) - ssh-add ~/.ssh/github

@Schachte
Copy link

Schachte commented May 7, 2023

This is great, thanks a lot for publishing this!

@StevenACoffman
Copy link
Author

Also a good related article about getting private Go modules to work with Github actions:
https://aran.dev/posts/github-actions-go-private-modules/

@thw0rted
Copy link

thw0rted commented Aug 25, 2023

I have found that go get private-repo.com/path/myModule still tries to use HTTPS but go get private-repo.com/path/myModule.git respects the insteadOf configuration correctly (note the ".git" at the end). Lots of older writeups say it worked without the ".git" so I'm wondering if it was a change within the last few Go versions...

@rkonfj
Copy link

rkonfj commented Sep 23, 2024

thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment