Proxying puppetforge files with Artifactory

README.md

Proxying puppetforge requests with Artifactory and Nginx

We have two goals:

• Ensure that we have local deterministic cache of puppet modules pulled from the forge
• Force all server requests to go through artifactory where possible

Artifactory configuration

We're going to use a remote repo type of generic pointing to forgeapi.puppetlabs.com. The json in this gist should provide the relevant information.

nginx configuration

This is where things get KIND of hairy: puppet module makes calls to some endpoints to find a module with pagination query params

• /modules
• /users
• /releases

Because artifactory doesn't really understand the pagination stuff, what ends up happening is the cached releases call, forces an infinite loop:

Debug: HTTP GET https://puppetforge.mydomain.com/v3/releases?module=puppetlabs-stdlib&limit=20&offset=20

over and over and over again because Artifactory is serving a cached copy of that with the same pagination data. Ideally what would happen here is that Artifactory would build the FINAL releases,modules,users from iterating through all pagination.

What this means is that for now we need to just send all of those requests to the forge directly. However we CAN at least proxy and cache the actual tarballs of the modules cleanly so for requests to v3/files we're going to send those directly to artifactory.

If we needed to recover those files for whatever reason, we could though doing it via puppet module install would likely fail if forgeapi.puppetlabs.com was offline for any reason.

It's not a perfect solution but it helps a bit.

making it better

It may be possible to write a user plugin for Artifactory that handles those specific requests more cleanly to build final consolidated files with no pagination data listing everything. What would be nicer is if Puppet made a v4 api that offered releases.gz, modules.gz and users.gz to make proxying and caching the forge for this reason much easier.

in action

$ bundle exec puppet module install --debug --module_repository=https://puppetforge.mydomain.com -i .tmp puppetlabs-stdlib
Debug: Runtime environment: puppet_version=3.7.4, ruby_version=2.1.5, run_mode=user, default_encoding=UTF-8
Notice: Preparing to install into /home/jvincent/.tmp ...
Notice: Downloading from https://puppetforge.mydomain.com ...
Debug: HTTP GET https://puppetforge.mydomain.com/v3/releases?module=puppetlabs-stdlib
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://puppetforge.mydomain.com/v3/releases?module=puppetlabs-stdlib&limit=20&offset=20
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://puppetforge.mydomain.com/v3/releases?module=puppetlabs-stdlib&limit=20&offset=40
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Info: Resolving dependencies ...
Info: Preparing to install ...
Debug: HTTP GET https://puppetforge.mydomain.com/v3/files/puppetlabs-stdlib-4.13.1.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Executing 'gzip -dc /home/jvincent/.puppet/var/puppet-module/cache/puppetlabs-stdlib20161013-9639-1ihx3s7 | tar xof -'
Debug: Executing 'find . -type d -exec chmod 755 {} +'
Debug: Executing 'find . -type f -exec chmod a-wst {} +'
Debug: Executing 'chown -R 1000:1000 .'
Notice: Installing -- do not interrupt ...
/home/jvincent/.tmp
└── puppetlabs-stdlib (v4.13.1)

nginx.conf

server {
	listen 443 ssl;
	server_name puppetforge.mydomain.com;
	if ($http_x_forwarded_proto = '') {
		set $http_x_forwarded_proto $scheme;
	}
	client_max_body_size 0;
	chunked_transfer_encoding on;
  # since artifactory doesn't handle the api requests very well (limit + offset)
  # we send this directly to the forge.
  # there are implications for reachability here but minor
  # the goal is to have a single system being responsible for reaching out
	location / {
	        proxy_pass https://forgeapi.puppetlabs.com;
	}

  # this is where we store the ACTUAL module tarballs
  # by proxying to artifactory
	location /v3/files {
		proxy_read_timeout 900;
		proxy_pass_header Server;
		proxy_cookie_path ~*^/.* /;
		proxy_pass http://localhost:8081/artifactory/puppetforge-upstream/v3/files;
		proxy_set_header X-JFrog-Art-API <dedicated artifactory user api key>;
		proxy_set_header X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host:$server_port/artifactory;
		proxy_set_header X-Forwarded-Port $server_port;
		proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
		proxy_set_header Host $http_host;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	}
}

repository config.json

{
  "key" : "puppetforge-upstream",
  "packageType" : "generic",
  "description" : "",
  "notes" : "",
  "includesPattern" : "**/*",
  "excludesPattern" : "",
  "repoLayoutRef" : "simple-default",
  "enableNuGetSupport" : false,
  "enableGemsSupport" : false,
  "enableNpmSupport" : false,
  "enableBowerSupport" : false,
  "enableCocoaPodsSupport" : false,
  "enableDebianSupport" : false,
  "debianTrivialLayout" : false,
  "enablePypiSupport" : false,
  "enableDockerSupport" : false,
  "dockerApiVersion" : "V2",
  "forceNugetAuthentication" : false,
  "enableVagrantSupport" : false,
  "enableGitLfsSupport" : false,
  "enableDistRepoSupport" : false,
  "url" : "https://forgeapi.puppetlabs.com",
  "username" : "",
  "password" : "",
  "handleReleases" : true,
  "handleSnapshots" : true,
  "suppressPomConsistencyChecks" : true,
  "remoteRepoChecksumPolicyType" : "",
  "hardFail" : false,
  "offline" : false,
  "blackedOut" : false,
  "storeArtifactsLocally" : true,
  "socketTimeoutMillis" : 15000,
  "localAddress" : "",
  "retrievalCachePeriodSecs" : 600,
  "assumedOfflinePeriodSecs" : 300,
  "missedRetrievalCachePeriodSecs" : 1800,
  "unusedArtifactsCleanupPeriodHours" : 0,
  "fetchJarsEagerly" : false,
  "fetchSourcesEagerly" : false,
  "shareConfiguration" : false,
  "synchronizeProperties" : false,
  "maxUniqueSnapshots" : 0,
  "maxUniqueTags" : 0,
  "propertySets" : [ "artifactory" ],
  "archiveBrowsingEnabled" : false,
  "listRemoteFolderItems" : true,
  "rejectInvalidJars" : false,
  "allowAnyHostAuth" : false,
  "enableCookieManagement" : false,
  "enableTokenAuthentication" : false,
  "propagateQueryParams" : true,
  "blockMismatchingMimeTypes" : true,
  "mismatchingMimeTypesOverrideList" : "",
  "contentSynchronisation" : {
    "enabled" : false,
    "statistics" : {
      "enabled" : false
    },
    "properties" : {
      "enabled" : false
    },
    "source" : {
      "originAbsenceDetection" : false
    }
  },
  "xrayIndex" : false,
  "xrayMinimumBlockedSeverity" : "",
  "blockXrayUnscannedArtifacts" : false,
  "rclass" : "remote"
}

Hello @lusis, I'm trying out latest Artifactory version and it seems that it works out of the box with puppet module install
OTOH I have problems making it work with librarian-puppet, does your solution work with Librarian? Did you try? Thanks!