Reproducing the r10k vs g10k benchmarks (2026 re-run of the g10k README benchmark)

r10k-g10k-benchmark-repro.md

Reproducing the r10k vs g10k benchmarks (2026 re-run)

Steps to reproduce the numbers from the blog post revisiting the g10k "Why fork?" benchmark.

Everything lives under one scratch directory; nothing touches ~/.r10k, your system Ruby, or any real Puppet infrastructure.

Measured setup: Apple M2 Pro, macOS, Ruby 3.4.8, r10k 5.0.3 (built from a checkout of main @ 68979c87, run via bundle exec, default settings, shellgit provider), g10k v0.9.10 (darwin-arm64 release binary, default settings). Installing r10k with gem install r10k should behave the same minus a little bundler startup overhead on every run.

Prerequisites

• Ruby >= 3.1
• git
• a g10k release binary from https://github.com/voxpupuli/g10k/releases
• gh CLI (only used to download g10k below; grab the zip manually if you prefer)

Common setup

BENCH=$(mktemp -d)/bench && mkdir -p $BENCH

# r10k from source (skip the rugged/dev gem groups; rugged needs libgit2
# and isn't used -- shellgit is the default provider)
git clone https://github.com/puppetlabs/r10k $BENCH/r10k-src
export BUNDLE_WITHOUT="extra:development"
export BUNDLE_GEMFILE=$BENCH/r10k-src/Gemfile
bundle install
bundle exec r10k version

# g10k release binary (pick your platform's asset)
gh release download v0.9.10 --repo voxpupuli/g10k \
  --pattern '*darwin-arm64*' --dir $BENCH
(cd $BENCH && unzip g10k-*.zip)

# the Puppetfile from the original 2016 benchmark: 4 git + 25 forge modules
curl -sL https://raw.githubusercontent.com/xorpaul/g10k-environment/benchmark/Puppetfile \
  -o $BENCH/Puppetfile

Benchmark 1: single Puppetfile

mkdir -p $BENCH/b1/r10k $BENCH/b1/g10k
cp $BENCH/Puppetfile $BENCH/b1/r10k/ && cp $BENCH/Puppetfile $BENCH/b1/g10k/

# r10k config. default_ref is needed because the Puppetfile's 'aws' module
# has no :ref; r10k 4+ refuses to guess (g10k silently assumes master).
cat > $BENCH/b1/r10k.yaml <<EOF
cachedir: '$BENCH/b1/r10k-cache'
git:
  default_ref: 'master'
EOF

Timed runs. Cold = wipe cache and module dir first; warm = just run again:

# r10k
rm -rf $BENCH/b1/r10k-cache $BENCH/b1/r10k/external_modules   # cold only
(cd $BENCH/b1/r10k && time bundle exec r10k puppetfile install -c $BENCH/b1/r10k.yaml)

# g10k
rm -rf $BENCH/b1/g10k-cache $BENCH/b1/g10k/external_modules   # cold only
(cd $BENCH/b1/g10k && time $BENCH/g10k -puppetfile -cachedir $BENCH/b1/g10k-cache -quiet)

Sanity check: both external_modules/ dirs contain 29 modules.

Results (3 runs per cell):

tool	cold cache	warm cache
r10k 5.0.3	4.9s, 4.4s, 4.7s	1.2s, 1.1s, 1.2s
g10k 0.9.10	3.5s, 3.5s, 2.9s	0.5s, 0.5s, 0.4s

Benchmark 2: 50 environments

A local bare control repo with 50 branches, each containing the same Puppetfile, deployed as a full environment sync.

B2=$BENCH/b2 && mkdir -p $B2/r10k-envs $B2/g10k-envs

# control repo: one commit, 50 branches
mkdir $B2/work && cd $B2/work && git init -b production
cp $BENCH/Puppetfile . && git add . && git commit -m bench
for i in $(seq -w 2 50); do git branch env_$i; done
git clone --bare $B2/work $B2/control-repo.git

cat > $B2/r10k.yaml <<EOF
cachedir: '$B2/r10k-cache'
git:
  default_ref: 'master'
sources:
  bench:
    remote: '$B2/control-repo.git'
    basedir: '$B2/r10k-envs'
EOF

cat > $B2/g10k.yaml <<EOF
:cachedir: '$B2/g10k-cache'

sources:
  bench:
    remote: '$B2/control-repo.git'
    basedir: '$B2/g10k-envs'
EOF

Timed runs:

# r10k
rm -rf $B2/r10k-cache $B2/r10k-envs && mkdir -p $B2/r10k-envs   # cold only
time bundle exec r10k deploy environment -c $B2/r10k.yaml --modules

# g10k
rm -rf $B2/g10k-cache $B2/g10k-envs && mkdir -p $B2/g10k-envs   # cold only
time $BENCH/g10k -config $B2/g10k.yaml -quiet

Sanity check: both *-envs/ dirs contain 50 environments, each with 29 modules under external_modules/.

Results (2 cold runs per tool -- each cold r10k run makes ~700 Forge :latest version queries, so be kind to the Forge API; 3 warm runs):

tool	cold cache	warm cache (no-op)
r10k 5.0.3	84.3s, 83.4s	23.0s, 23.0s, 23.1s
g10k 0.9.10	48.5s, 48.0s	1.7s, 1.6s, 1.7s

Notes

• Defaults everywhere: r10k pool_size 4; g10k 50 resolve / 20 extract workers.
• All 29 modules from the 2016 Puppetfile still resolve, though many are deprecated on the Forge.
• Times are wall-clock, no discards. Compare ratios, not absolute numbers, when relating these to the 2016 figures.