Upgrade Symbiosis to LetsEncrypt v2 API

README.md

Bytemark have now published an update!

Since writing this gist, Bytemark have published an update to use the LetsEncrypt v2 API, so an apt-get update/upgrade should work (or rather it should have already worked :)).

Upgrade Symbiosis to LetsEncrypt v2 API (not using the Bytemark update)

This is effectively a monkey-patch to overwrite existing letsencrypt functionality in Symbiosis to use the new v2 API, as the existing v1 API is being discontinued in June 2020.

This depends on a newer version of the [acme-client gem](https://github.com/unixcharles/acme-client].

Installation

First install a newer version of acme-client as a gem. This sits alongside (but doesn't overwrite) the existing Debian packaged version of ruby-acme-client.

$ sudo gem install acme-client

Now drop in the new letsencrypt.rb to `/usr/local/lib/site_ruby/symbiosis/ssl'.

$ sudo mkdir -p /usr/local/lib/site_ruby/symbiosis/ssl
$ sudo wget https://gist.githubusercontent.com/patch0/1be74e58fe5957aa27cfefd56bc813e2/raw/f3a3c7e5bcacb394f16d4499ea0c77f2e3ae6e4d/letsencrypt.rb > /usr/local/lib/site_ruby/symbiosis/ssl/letsencrypt.rb

That should be it.

Usage

This is a drop-in bit of code, and the symbiosis-ssl script should continue to work as before, except using v2 endpoint. If you run it with verbose output (use the -v flag), you should see output mentioning the new endpoint:

        Requesting verification for example.net from https://acme-v02.api.letsencrypt.org/directory
        Successfully verified example.net
        Requesting verification for www.example.net from https://acme-v02.api.letsencrypt.org/directory
        Successfully verified www.example.net

letsencrypt.rb

# frozen_string_literal: true

require 'symbiosis/ssl'
require 'symbiosis/ssl/selfsigned'
require 'symbiosis/host'
require 'symbiosis/utils'
require 'time'
# Make sure you install acme-client gem
gem 'acme-client', '~> 2.0.5'
require 'acme-client'

begin
  require 'symbiosis/domain/http'
rescue LoadError
  # Do nothing
end

module Symbiosis
  class SSL
    class LetsEncrypt < Symbiosis::SSL::SelfSigned
      ENDPOINT = 'https://acme-v02.api.letsencrypt.org/directory'

      def initialize(domain, directory = nil)
        super
        @config = { email: nil, endpoint: nil, rsa_key_size: nil, docroot: nil, account_key: nil }
        @acme_certificates = nil
      end

      #
      # Returns the client instance.
      #
      def client
        @client ||= Acme::Client.new(private_key: account_key,
                                     directory: endpoint,
                                     bad_nonce_retry: 5)
      end

      def acme_order
        @acme_order ||= client.new_order(identifiers: @names)
      end

      #
      # Returns the account key.  If one has not been set, it generates and
      # writes it to the configuration directory.
      #
      def account_key
        if @config[:account_key].is_a?(OpenSSL::PKey::RSA)
          return @config[:account_key]
        end

        if config[:account_key].is_a? String
          account_key = OpenSSL::PKey::RSA.new(config[:account_key])
        else
          account_key = OpenSSL::PKey::RSA.new(rsa_key_size)
          set_param('account_key', account_key.to_pem, config_dirs.first, mode: 0o600)
        end

        @config[:account_key] = account_key
      end

      #
      # Returns the document root for the HTTP01 challenge
      #
      def docroot
        if config[:docroot].is_a?(String) && File.directory?(config[:docroot])
          return config[:docroot]
        end

        #
        # If symbiosis-http is installed, we use htdocs dir, otherwise default to public/htdocs.
        #
        @config[:docroot] = if domain.respond_to?(:htdocs_dir)
                              domain.htdocs_dir
                            else
                              File.join(domain.directory, 'public', 'htdocs')
                            end

        @config[:docroot]
      end

      #
      # Returns the account's email address, defaulting to root@fqdn if nothing set.
      #
      def email
        unless config[:email].is_a?(String)
          @config[:email] = 'root@' + Symbiosis::Host.fqdn
        end

        if @config[:email] =~ %r{([^\.@%!/\|\s][^@%!/\|\s]*@[a-z0-9\.-]+)}i
          @config[:email] = Regexp.last_match(1)
        else
          if $VERBOSE
            puts "\tAddress #{@config[:email].inspect} looks wrong.  Using default"
          end
          @config[:email] = 'root@' + Symbiosis::Host.fqdn
        end

        config[:email]
      end

      #
      # Returns the default endpoint, defaulting to the live endpoint
      #
      def endpoint
        return config[:endpoint] if config[:endpoint].is_a?(String)

        @config[:endpoint] = ENDPOINT
      end

      #
      # Register the account RSA kay with the letsencrypt server
      #
      def register
        #
        # Send the key to the server.
        #
        account = client.new_account(contact: 'mailto:' + email, terms_of_service_agreed: true)

        true
      end

      #
      # Tests to see if we're registered by doing a pre-emptive authorize
      # request.
      #
      def registered?
        true
      rescue Acme::Client::Error::AccountDoesNotExist
        false
      end

      #
      # This does the authorization.  Returns true if the verification succeeds.
      #
      def verify_name(name)
        #
        # Set up the authorisation for the http01 challenge
        authorisation = acme_order.authorizations.find{|a| a.domain == name }
        challenge     = authorisation.http
        challenge_directory = File.join(docroot, File.dirname(challenge.filename))

        mkdir_p(challenge_directory)

        set_param(File.basename(challenge.filename), challenge.file_content, challenge_directory)

        vs = nil # Record the verify status

        if challenge.request_validation
          if $VERBOSE
            puts "\tRequesting verification for #{name} from #{endpoint}"
          end

          60.times do
            vs = challenge.status
            break unless vs == 'pending'

            sleep(1)
            challenge.reload
          end
        end

        unless $DEBUG
          set_param(File.basename(challenge.filename), false, challenge_directory)
        end

        if vs == 'valid'
          puts "\tSuccessfully verified #{name}" if $VERBOSE
          return true
        else
          if $VERBOSE
            puts "\t!! Unable to verify #{name} (status: #{vs})"
            puts "\t!! Check http://#{name}/#{challenge.filename} works."
          end
          return false
        end
      end

      def acme_certificates(request = self.request)
        return @acme_certificates if @acme_certificates

        unless request.is_a?(OpenSSL::X509::Request)
          raise ArgumentError, 'Invalid certificate request'
        end

        acme_order.finalize(csr: request)

        while acme_order.status == 'processing'
          sleep(1)
          acme_order.reload
        end

        certs = acme_order.certificate.split(/(?=-----BEGIN CERTIFICATE-----)/)
        x509_certs = certs.map{|c| OpenSSL::X509::Certificate.new(c) }

        @acme_certificates = x509_certs
      end

      #
      # Returns the signed X509 certificate for the request.
      #
      def certificate(request = self.request)
        acme_certificates(request).first
      end

      #
      # Returns the CA bundle as an array
      #
      def bundle(request = self.request)
        acme_certificates(request)[1..-1]
      end
    end

    PROVIDERS << LetsEncrypt
  end
end

The @directory not initialized message is a debug one relating to the acme gem, and nothing to worry about. It means that the variable @directory was being used before it was properly initialized, and in all likelihood is harmless. I don't know why you had problems with your key @hairydogltd.

Yes, Bytemark put out an update (hello @andrewladlow!) the only difference is that they have implemented their own library to access the LetsEncrypt API, whereas this solution uses the acme-client gem.

To uninstall this version fully, you can remove the extra lib as @fortybelowzero said, and also uninstall that gem.

$ sudo rm -rf /usr/local/lib/site_ruby/symbiosis/ssl
$ sudo gem uninstall acme-client

@patch0 - FYI i got the same "!! Failed: No account exists with the provided key" when i set up a new test virtual host on a server and asked it to request a letsencrypt certificate - i got the warning but no error when it was trying to request a certificate for a virtualhost that's already existed a while, so possibly something related to symbiosis-ssl creating a separate letsencrypt account key for each virtualhost and something's not happening quite rightly in that instance?

@andrewladlow's patch works fine tho 🥳, so all good (but thanks @patch0 for tackling it -- i was fully expecting to need to migrate vm's to sympl or manually run certbot)