support discussion

r10k-4.0.md

7 Things to know about r10k 4 in PE 2023.4

(From the PE documentation at: https://www.puppet.com/docs/pe/2023.4/upgrading_pe#upgrade_cautions-r10k-4-upgrade)

1. Starting in PE 2023.4, if you use Code Manager or r10k, with SSH protocol for remote Git repository access, you must set up SSH host key verification.

To manage the known_hosts file and enable host key verification for Code Manager or r10k, you must define the puppet_enterprise::profile::master::r10k_known_hosts paramet er with an array of hashes specifying "name", "type", and "key" with your hostname, key type, and public key, respectively.

This is the error message you will see if running code deploy:

$ puppet code deploy --all --wait                                                                                                                                             
                                           
Deploying all environments.
2023/10/11 03:53:54 ERROR - [POST /deploys][500] Errors while collecting a list of environments to deploy (exit code: 1).                                                     
"ERROR\t -\u003e Unable to determine current branches for Git source 'puppet' (/etc/puppetlabs/code-staging/environments)\nOriginal exception:\ninvalid or unknown remote ssh
hostkey at /opt/puppetlabs/server/data/code-manager/git/git@github.com-justinstoller-control-repo.git\n"

Note the key portion "Original exception: invalid or unknown remote ssh hostkey"

To resolve this we must know the public keys of the computer hosting the repository, to do so:

$ ssh-keyscan github.com                                                               
                                                                                                                                                                              
# github.com:22 SSH-2.0-babeld-dd067d10
github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOx
SDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6Xyv
xvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/Wn
wH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=
# github.com:22 SSH-2.0-babeld-dd067d10
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
# github.com:22 SSH-2.0-babeld-dc5ec9be                                                                                                                                       
github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl

Each host key outputs a comment followed by a line containing the information we need. A line we can use:

github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl                                                                                   

The line follows the form of <hostname> <keytype> <keycontent>

To turn this into input for PE, add the puppet_enterprise::profile::master::r10k_known_hosts parameter. The parameter is a list of host key definitions:

[ <host-key-definition-1>, <host-key-definition-2>, <host-key-defnition-3> ]

Where host-key-definition can be:

{
    "name": "<hostname>",
    "title": "<unique identifier",
    "type": "<keytype>",
    "key": "<keycontent>",
    "host_aliases": [ "<host-aliases>", ...],
    "ensure": "<ensure-value>"
}

"name", "type", and "key" are required.

For example:

[
    {
        "name":"github.com",
        "type":"ssh-ed25519",
        "key":"AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl"
    }
]

If a key should be removed from the known hosts file the key definition must have the "ensure" key set to "absent", eg:

{
    "name": "example.com",
    "type": "ssh-rsa",
    "key": "AAAAC3NzaC1lZDI1...",
    "ensure": "absent"
}

After running the puppet agent we should see:

$ ls -lah /opt/puppetlabs/server/data/puppetserver/.ssh/
total 4.0K
drwx------  2 pe-puppet pe-puppet  25 Oct 10 22:58 .
drwxr-xr-x 16 pe-puppet pe-puppet 239 Oct 10 22:58 ..
-rw-r--r--  1 pe-puppet pe-puppet 265 Oct 11 04:25 known_hosts

$ cat /opt/puppetlabs/server/data/puppetserver/.ssh/known_hosts

# HEADER: This file was autogenerated at 2023-10-11 04:25:04 +0000
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl

Should an issue arise one can manually do what the agent is doing, eg:

$ mkdir /opt/puppetlabs/server/data/puppetserver/.ssh

$ ssh-keyscan github.com | grep -v '# ' > /opt/puppetlabs/server/data/puppetserver/.ssh/known_hosts

$ chown -R pe-puppet:pe-puppet /opt/puppetlabs/server/data/puppetserver/.ssh

You can verify the content of the known_hosts file by copying it from Puppet Server's .ssh directory to your user's .ssh directory.

2. Starting in version 4.0, r10k no longer accesses repos on local file systems not owned by the pe-puppet user. If you use r10k on the local file system, ensure that your control repo and module repos are owned by the pe-puppet user.

3. Git-based modules no longer have a default reference such as master. Now, when pointing to Git-based modules, you must specify the branches or revisions. Alternatively, you can use the global default_ref setting to manually set your default reference.

For example, this Puppetfile is not a best practice, and is no longer supported by default:

mod 'apache', git: 'git@github.com:puppetlabs/puppetlabs-apache.git'

It should be replaced with a better practice of:

mod 'apache', git: 'git@github.com:puppetlabs/puppetlabs-apache.git', ref: '8f44e53d18315deff406c21dafc4a064913ea51f'

4. By default, the exclude_spec setting is now set to true so that spec directories are automatically deleted from Git-based sources.

5. If you use your own Ruby install, note that r10k 4.0 drops support for Ruby versions 2.3, 2.4, and 2.5.

6. The purge_whitelist setting has been removed. Instead, you can use purge_allowlist when manually configuring r10k 4.0.

7. The protected method basedir is removed from the Puppetfile @librarian instance. Protected methods should no longer be used in Puppetfile. However, if you require a temp orary workaround, you can use the environment_name accessor.