recently a friend asked me on to provide a basic howto for puppet on howto
- install a puppet server
- populate the server with modules (from the forge)
- populate the server with our manifests (from git)
- connect a node to the server, and
- configure nginx on that node to serve a blog
I've always found the Puppetlabs documentation to be good, but not very beginner friendly. And beginner, in this context doesn't mean new to Systems Administration, or even Automation, or Configuration Management. Just new to Puppet.
To familiarize your with the basic syntax and lingo, I will be using puppet to get you bootstrapped. I will always try to explain new concepts I'm introducing, linking to their documentation, or their blog posts where those explain them better :)
We'll be using Vagrant for this Guide. You can follow along each section of this guide, by switching branches in this accompanying repository
Note that our Vagrant Boxes already come pre-installed with Puppet, so we don't have to bootstrap them completely!
Let's get started, then, by preparing our vagrant box:
~ % git clone https://github.com/igalic/getting-started-w-puppet.git
~ % cd getting-started-w-puppet
~/getting-s.. % ./bootstrap.sh # this will install the vagrant-hostmanager plugin
~/getting-s.. % vagrant up puppet
...
~/getting-s.. % git checkout s00-add-some-modules
~/getting-s.. % vagrant ssh puppetPuppet is a great configuration management tool, but what makes it truly powerful are the myriad of modules for all kinds of software out there. Many of those modules are published to the forge, others can be found on GitHub, or BitBucket. To manage those modules' installation, we will use a tool called r10k
package { 'r10k':
ensure => 'present',
provider => 'gem',
}And, since we'll need git for most of our work, we'll install that too:
package { 'git':
ensure => 'present',
}Note that we're installing git with no provider attribute. As such, it will default to the system default provider. In Ubuntu's case, this is apt.
You can install git and r10k by running puppet apply:
vagrant@puppet:/vagrant/manifests$ sudo puppet apply -v install.pp
Notice: Compiled catalog for puppet.acme in environment production in 0.18 seconds
Info: Applying configuration version '1426538544'
Notice: /Stage[main]/Main/Package[git]/ensure: ensure changed 'purged' to 'present'
Notice: /Stage[main]/Main/Package[r10k]/ensure: created
Notice: Finished catalog run in 23.99 seconds
vagrant@puppet:/vagrant/manifests$We can ignore any Warning:s for now. We'll fix those soon enough. Because first of all, we have to configure r10k.
r10k is configured via a yaml file in /etc/r10k.yaml, the most important bits are set in the :sources: hash, which sets the target directory where the remote repository is checked out:
:cachedir: /var/cache/r10k
:sources:
puppet:
basedir: /etc/puppet/environments
remote: git://github.com/igalic/getting-started-w-puppet-environment.gitwe can generate this easily in puppet by putting all of this in a Puppet hash, and then writing out:
$r10k = {
'cachedir' => '/var/cache/r10k',
'sources' => {
'puppet' => {
'basedir' => '/etc/puppet/environments',
'remote' => 'git://github.com/igalic/getting-started-w-puppet-environment.git',
}
}
}
file { '/etc/r10k.yaml':
ensure => present,
# cachedir and sources must be ruby symbols, so transform them:
content => inline_template("<%= cfg = {} ; @r10k.keys.each { |k| cfg[k.to_sym] = @r10k[k] } ; cfg.to_yaml %>\n%>"),
}Let's configure r10k then:
vagrant@puppet:/vagrant/manifests$ sudo puppet apply -v config.pp
Notice: Compiled catalog for puppet.acme in environment production in 0.13 seconds
Info: Applying configuration version '1426541932'
Notice: /Stage[main]/Main/File[/etc/r10k.yaml]/ensure: created
Notice: Finished catalog run in 0.02 seconds
vagrant@puppet:/vagrant/manifests$our /etc/r10k.yaml should now look something like this:
---
!ruby/sym cachedir: /var/cache/r10k
!ruby/sym sources:
puppet:
basedir: /etc/puppet/environments
remote: "git://github.com/igalic/getting-started-w-puppet-environment.git"
that's a funny way of writing ruby symbols, but it sure works! We can check, by deploying our environment:
vagrant@puppet:~$ sudo r10k deploy environment
vagrant@puppet:~$this command checks out every branch of our remote into a sub-directory of /etc/puppet/environments. So for now, it should just contain production:
vagrant@puppet:/etc/puppet/environments$ cd production/
vagrant@puppet:/etc/puppet/environments/production$ ls
Puppetfile
vagrant@puppet:/etc/puppet/environments/production$but it's the Puppetfile that contains the magic. This is how we tell r10k which modules we want to have in this environment:
# vim: set ft=ruby:
forge 'http://forge.puppetlabs.com'
# pretty much indispensible, also, almost all modules depend on it ;)
mod 'puppetlabs/concat' , '1.2.0'
mod 'puppetlabs/inifile' , '1.2.0'
mod 'puppetlabs/stdlib' , '4.5.1'
# we're on ubuntu, so add apt:
mod 'puppetlabs/apt' , '1.7.0'
# for our puppetserver:
mod 'camptocamp/puppetserver' , '0.7.0'
mod 'postgresql' , :git => 'git://github.com/puppetlabs/puppetlabs-postgresql'
mod 'puppetdb' , :git => 'git://github.com/puppetlabs/puppetlabs-puppetdb'
# we'll use these for deploying our blog, and serving it:
mod 'jfryman/nginx' , '0.2.2'
mod 'puppetlabs/vcsrepo' , '1.2.0'
mod 'nodejs' , :git => 'git://github.com/puppetlabs/puppetlabs-nodejs'
mod 'wintersmith' , :git => 'git://github.com/Brainsware/puppet-wintersmith'we can deploy all of those modules by running sudo r10k deploy environment -p.
Please note that for our very simple start here we haven't declared any custom, in-house modules. When r10k needs access to your private repositories, you'll have to deploy a (read-only!) ssh-key in the root-user's home directory. And, when accessing these git repositories via ssh, it's good to know them beforehand. Because we used the git:// protocol, I didn't see the necessity to do this. However, it's very easily done, with the built-in sshkey type, and GitHub's well-documented ssh-keys, well, the fingerprints, anyway:
sshkey { 'github.com':
key => 'AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==',
type => 'ssh-rsa',
}you get the idea…
Now, with all our modules in place, it's time to tackle the configuration of the Puppetserver.
We'' start this section by checking out the branch for this section:
~/src/me/getting-started-w-puppet (git)-[s00-add-some-modules] % git checkout s01-install-puppetserver
Switched to branch 's01-install-puppetserver'
~/src/me/getting-started-w-puppet (git)-[s01-install-puppetserver] % with all modules in place, we will install, but more importantly configure Puppetserver and PuppetDB. Puppetserver is a high-performance rewrite of Puppetmaster. It runs on the JVM, is written in Clojure, and only contains a tiny Ruby-core, which is still responsible for executing our code. Puppetserver is based on the same, proven technology as PuppetDB, which is used to store reports about our puppet runs, and other useful information, such as exported resources.
We can get all of those components installed with puppet, by including them:
include puppetdb
include puppetdb::master::config
include puppetserverhowever, before we can actually use an include, puppet apply must be able to find our modules. So, let's finally fix our puppet.conf. We'll do that with ini_settings. We'll also make sure we really have the latest version of puppet installed:
Ini_setting {
path => '/etc/puppet/puppet.conf',
}
ini_setting {
'environmentpath':
section => 'main',
value => '$confdir/environments';
'pluginsync':
section => 'main',
value => true;
'templatedir':
ensure => 'absent',
section => 'main';
}
ini_setting {
'ca':
section => 'master',
value => true;
'reports':
section => 'master',
value => 'puppetdb';
'storeconfigs':
section => 'master',
value => true;
}note the use of the Capitalised Ini_setting, which defines a resource's default.
We also see the semicolon syntax of condensing multiple resource declarations of the same kind.
To apply this configuration, we have to tell puppet apply where to find modules:
vagrant@puppet:/vagrant/manifests$ sudo puppet apply --modulepath=/etc/puppet/environments/production/modules -v puppetconf.pp
Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
(at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
Info: Loading facts
Info: Loading facts
Info: Loading facts
Notice: Compiled catalog for puppet.acme in environment production in 0.05 seconds
Info: Applying configuration version '1426607644'
Notice: /Stage[main]/Main/Ini_setting[remove templatedir]/ensure: removed
Notice: /Stage[main]/Main/Ini_setting[ca]/ensure: created
Notice: /Stage[main]/Main/Ini_setting[storeconfigs]/ensure: created
Notice: /Stage[main]/Main/Ini_setting[environmentpath]/ensure: created
Notice: /Stage[main]/Main/Ini_setting[reports]/ensure: created
Notice: /Stage[main]/Main/Ini_setting[pluginsync]/ensure: created
Notice: Finished catalog run in 0.02 seconds
vagrant@puppet:/vagrant/manifests$if we run the same command without --modulepath, it should not only succeed, but it shouldn't change anything:
vagrant@puppet:/vagrant/manifests$ sudo puppet apply -v puppetconf.pp
Info: Loading facts
Info: Loading facts
Info: Loading facts
Notice: Compiled catalog for puppet.acme in environment production in 0.05 seconds
Info: Applying configuration version '1426607692'
Notice: Finished catalog run in 0.02 seconds
vagrant@puppet:/vagrant/manifests$Success! We're ready to deploy Puppetserver and Puppetdb!
Now, unfortunately, that is slightly more complicated, because…
i'm pretty and sure that no one creating packaging sits down and asks themselves: how do we make this more hostile to automation, BUTT…
— The Wrath of PB™ (@hirojin) March 18, 2015The new PostgreSQL Debian packages require you to explicitly create a (even if it's the) main cluster with pg_createcluster. Rather than doing that, I "simply" created the required directories, and had puppet's postgresql module do the rest:
package { 'postgresql':
ensure => installed,
}
# Generate the locale(s) we'll be using:
exec { '/usr/sbin/locale-gen C.UTF-8':
unless => '/usr/bin/locale -a | /bin/grep -q C.UTF-8'
} ->
exec { '/usr/bin/pg_createcluster --locale C.UTF-8 9.3 main':
creates => '/var/lib/postgresql/9.3/main',
require => Package[postgresql],
}After running this with puppet apply -v postgresql.pp, we're ready to setup puppetserver (including puppetdb, and its postgresql dependencies):
class { 'postgresql::globals':
locale => 'C',
encoding => 'UTF-8',
before => Class[::postgresql::server]
}
include postgresql::server
include postgresql::server::contrib
include puppetserver
class { 'puppetdb':
manage_dbserver => false,
ssl_listen_address => '0.0.0.0',
require => [
Class[::postgresql::server],
Class[::puppetserver],
]
}
# PuppetDB logs always say they want this:
postgresql::server::extension { 'pg_trgm':
database => 'puppetdb',
require => Class[::postgresql::server::contrib],
}
# Tell puppetdb that our puppetmaster is actually puppetserver!
class { 'puppetdb::master::config':
puppet_service_name => 'puppetserver',
}We can run this monstrosity with:
vagrant@puppet:/vagrant$ sudo puppet apply -v manifests/puppetserver.pp
...
I'll spare you the output here. I recommand being patient, while watching ps -cafe, or (re)reading Dave Barry's How to Install Software.
You might also have to run this a couple of times until it "stabilizes". You might have noticed the different uses of include vs declaring a class. If you found this confusing, you can read about the differences.
Once it's "done", we should verify that puppetserver is actually running. Hint, it's probably not. Let's fix it:
root@puppet:~# sudo rm -rf /var/lib/puppet/ssl/
root@puppet:~# sudo service puppetserver start
root@puppet:~#
PuppetDB will probably also suffer from a case of bad SSL config (as if there is such a case as a good SSL config…), which we'll also fix manually:
vagrant@puppet:~$ sudo puppetdb ssl-setup -f
PEM files in /etc/puppetdb/ssl already exists, checking integrity.
Warning: /etc/puppetdb/ssl/ca.pem does not match the file used by Puppet (/var/lib/puppet/ssl/certs/ca.pem)
Warning: /etc/puppetdb/ssl/private.pem does not match the file used by Puppet (/var/lib/puppet/ssl/private_keys/puppet.acme.pem)
Warning: /etc/puppetdb/ssl/public.pem does not match the file used by Puppet (/var/lib/puppet/ssl/certs/puppet.acme.pem)
Overwriting existing PEM files due to -f flag
Copying files: /var/lib/puppet/ssl/certs/ca.pem, /var/lib/puppet/ssl/private_keys/puppet.acme.pem and /var/lib/puppet/ssl/certs/puppet.acme.pem to /etc/puppetdb/ssl
Setting ssl-host in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-port in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-key in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-cert in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-ca-cert in /etc/puppetdb/conf.d/jetty.ini already correct.
vagrant@puppet:~$
and then service puppetdb restart it. Once it comes up, the puppet apply -v puppetserver.pp run should complete successfully.
Which means, we're ready for a puppet agent --test for the very first time.
Note that some vagrant boxes that come with chef or puppet disable it, so you may be required to puppet agent --enable it first.
Our first puppet agent -t run will do a pluginsync. This might sound unusual, since this machine already has the plugins, it's serving them! However, when we consider that puppet agent and puppetserver are different entities it makes slightly more sense.
vagrant@puppet:~$ sudo puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for puppet.acme
Info: Applying configuration version '1426668907'
Notice: Finished catalog run in 0.16 seconds
vagrant@puppet:~$
and we should have some reports:
vagrant@puppet:~$ curl http://localhost:8080/v4/nodes ; echo
[ {
"catalog-environment" : "production",
"catalog-timestamp" : "2015-03-18T08:55:08.134Z",
"certname" : "puppet.acme",
"deactivated" : null,
"facts-environment" : "production",
"facts-timestamp" : "2015-03-18T08:55:07.334Z",
"report-environment" : "production",
"report-timestamp" : "2015-03-18T08:55:03.407Z"
} ]
vagrant@puppet:~$
wh000t! This means we're ready to…
This should be a separate blog-post.