Skip to content

Instantly share code, notes, and snippets.

@timbirk
Last active August 14, 2018 14:55
Show Gist options
  • Save timbirk/0308d8128927ec0f66011d7726980a9e to your computer and use it in GitHub Desktop.
Save timbirk/0308d8128927ec0f66011d7726980a9e to your computer and use it in GitHub Desktop.
A ruby script to generate a complete R10k ready Puppetfile from a librarian puppet style file.
#!/usr/bin/env ruby
require 'librarian/puppet'
lockfile = Librarian::Puppet::Lockfile.new(
Librarian::Puppet::Environment.new, 'Puppetfile.lock'
)
puppet_modules = {}
lockfile.load(File.read(lockfile.path)).manifests.each do |mod|
mod_type = mod.source.class.name.split('::').last.downcase.to_sym
puppet_modules[mod_type] = {} unless puppet_modules.key?(mod_type)
puppet_modules[mod_type][mod.name.to_sym] = {}
case mod_type
when :forge
puppet_modules[mod_type][mod.name.to_sym][:version] = mod.version.to_s
when :git
puppet_modules[mod_type][mod.name.to_sym][:git] = mod.source.uri
puppet_modules[mod_type][mod.name.to_sym][:ref] = mod.source.ref
end
end
puppetfile = []
File.readlines('Puppetfile').each do |line|
break if line =~ /#{Regexp.escape("# Puppetfile.lock dependencies")}/
puppetfile << line
end
puppetfile.pop if puppetfile.last == "\n"
puppetfile << "\n# Puppetfile.lock dependencies"
puppet_modules.each do |mod_type, sources|
sources = sources.sort_by { |k, _v| k }
puppetfile << "\n# #{mod_type} modules"
sources.each do |mod, data|
case mod_type
when :forge
next if puppetfile.grep(/#{Regexp.escape(mod)}/).any?
puppetfile << "mod '#{mod}', '#{data[:version]}'"
when :git
next if puppetfile.grep(/#{Regexp.escape(data.to_h[:git])}/).any?
puppetfile << "mod '#{mod}',"
puppetfile << " :git => '#{data[:git]}',"
puppetfile << " :ref => '#{data[:ref]}'"
end
end
end
puts puppetfile
@timbirk
Copy link
Author

timbirk commented Mar 4, 2018

For a long time there's been debate and unhappy faces when it comes to using librarian-puppet vs r10k vs g10k.

The main reason for librarian-puppet is it's awesome (but slow) dependency resolution. This script uses the librarian-puppet gem to parse and effectively reverse engineer your Puppetfile.lock to a complete Puppetfile with versions for forge module dependencies.

@timbirk
Copy link
Author

timbirk commented Mar 4, 2018

For example, a Puppetfile like this:

#!/usr/bin/env ruby
#^syntax detection

forge "https://forgeapi.puppetlabs.com"

mod 'puppetlabs-stdlib'
mod 'puppetlabs-git'
mod 'puppetlabs-haproxy'
mod 'puppetlabs-java'
mod 'theforeman-foreman'

Gives the following output:

#!/usr/bin/env ruby
#^syntax detection

forge "https://forgeapi.puppetlabs.com"

mod 'puppetlabs-stdlib'
mod 'puppetlabs-git'
mod 'puppetlabs-haproxy'
mod 'puppetlabs-java'
mod 'theforeman-foreman'

# Puppetfile.lock dependencies

# forge modules
mod 'puppet-archive', '2.3.0'
mod 'puppet-extlib', '2.0.1'
mod 'puppetlabs-concat', '2.2.1'
mod 'puppetlabs-apache', '2.3.1'
mod 'puppetlabs-apt', '2.4.0'
mod 'puppetlabs-postgresql', '5.3.0'

@timbirk
Copy link
Author

timbirk commented Mar 4, 2018

You have to run a librarian-Puppet install first to generate the .lock file. To replace your Puppetfile with the output of this do something like:

ruby lp2r10k.rb > Puppetfile.tmp ; mv Puppetfile.tmp Puppetfile

@timbirk
Copy link
Author

timbirk commented Mar 4, 2018

Performance improvements with the above complete Puppetfile and various tools:

With regular Puppetfile, no .lock file and librarian-puppet:
librarian-puppet install --clean  15.78s user 9.11s system 21% cpu 1:53.87 total

With complete Puppetfile and librarian-puppet:
librarian-puppet install --clean  16.14s user 9.59s system 38% cpu 1:06.24 total

With complete Puppetfile and r10k:
r10k puppetfile install --puppetfile Puppetfile --moduledir modules  1.50s user 1.58s system 24% cpu 12.628 total

With complete Puppetfile and g10k:
g10k -usemove -puppetfile -moduledir modules  0.37s user 1.57s system 44% cpu 4.399 total

@madAndroid
Copy link

Impressive!! From 2 minutes down to 5 seconds?? that's an amazing improvement

@timbirk
Copy link
Author

timbirk commented Mar 5, 2018

That is on a MacBook Pro with a Core i7. Of course. Mileage may vary when it comes to EC2 instances which generally have less powerful CPU’s and are prone to context switching due to a lower number of cores. Taking advantage of g10k’s concurrency (downloads, gz extracts) should still see an impressive improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment