Upgrade Symbiosis to LetsEncrypt v2 API

README.md

Bytemark have now published an update!

Since writing this gist, Bytemark have published an update to use the LetsEncrypt v2 API, so an apt-get update/upgrade should work (or rather it should have already worked :)).

Upgrade Symbiosis to LetsEncrypt v2 API (not using the Bytemark update)

This is effectively a monkey-patch to overwrite existing letsencrypt functionality in Symbiosis to use the new v2 API, as the existing v1 API is being discontinued in June 2020.

This depends on a newer version of the [acme-client gem](https://github.com/unixcharles/acme-client].

Installation

First install a newer version of acme-client as a gem. This sits alongside (but doesn't overwrite) the existing Debian packaged version of ruby-acme-client.

$ sudo gem install acme-client

Now drop in the new letsencrypt.rb to `/usr/local/lib/site_ruby/symbiosis/ssl'.

$ sudo mkdir -p /usr/local/lib/site_ruby/symbiosis/ssl
$ sudo wget https://gist.githubusercontent.com/patch0/1be74e58fe5957aa27cfefd56bc813e2/raw/f3a3c7e5bcacb394f16d4499ea0c77f2e3ae6e4d/letsencrypt.rb > /usr/local/lib/site_ruby/symbiosis/ssl/letsencrypt.rb

That should be it.

Usage

This is a drop-in bit of code, and the symbiosis-ssl script should continue to work as before, except using v2 endpoint. If you run it with verbose output (use the -v flag), you should see output mentioning the new endpoint:

        Requesting verification for example.net from https://acme-v02.api.letsencrypt.org/directory
        Successfully verified example.net
        Requesting verification for www.example.net from https://acme-v02.api.letsencrypt.org/directory
        Successfully verified www.example.net

letsencrypt.rb

# frozen_string_literal: true

require 'symbiosis/ssl'
require 'symbiosis/ssl/selfsigned'
require 'symbiosis/host'
require 'symbiosis/utils'
require 'time'
# Make sure you install acme-client gem
gem 'acme-client', '~> 2.0.5'
require 'acme-client'

begin
  require 'symbiosis/domain/http'
rescue LoadError
  # Do nothing
end

module Symbiosis
  class SSL
    class LetsEncrypt < Symbiosis::SSL::SelfSigned
      ENDPOINT = 'https://acme-v02.api.letsencrypt.org/directory'

      def initialize(domain, directory = nil)
        super
        @config = { email: nil, endpoint: nil, rsa_key_size: nil, docroot: nil, account_key: nil }
        @acme_certificates = nil
      end

      #
      # Returns the client instance.
      #
      def client
        @client ||= Acme::Client.new(private_key: account_key,
                                     directory: endpoint,
                                     bad_nonce_retry: 5)
      end

      def acme_order
        @acme_order ||= client.new_order(identifiers: @names)
      end

      #
      # Returns the account key.  If one has not been set, it generates and
      # writes it to the configuration directory.
      #
      def account_key
        if @config[:account_key].is_a?(OpenSSL::PKey::RSA)
          return @config[:account_key]
        end

        if config[:account_key].is_a? String
          account_key = OpenSSL::PKey::RSA.new(config[:account_key])
        else
          account_key = OpenSSL::PKey::RSA.new(rsa_key_size)
          set_param('account_key', account_key.to_pem, config_dirs.first, mode: 0o600)
        end

        @config[:account_key] = account_key
      end

      #
      # Returns the document root for the HTTP01 challenge
      #
      def docroot
        if config[:docroot].is_a?(String) && File.directory?(config[:docroot])
          return config[:docroot]
        end

        #
        # If symbiosis-http is installed, we use htdocs dir, otherwise default to public/htdocs.
        #
        @config[:docroot] = if domain.respond_to?(:htdocs_dir)
                              domain.htdocs_dir
                            else
                              File.join(domain.directory, 'public', 'htdocs')
                            end

        @config[:docroot]
      end

      #
      # Returns the account's email address, defaulting to root@fqdn if nothing set.
      #
      def email
        unless config[:email].is_a?(String)
          @config[:email] = 'root@' + Symbiosis::Host.fqdn
        end

        if @config[:email] =~ %r{([^\.@%!/\|\s][^@%!/\|\s]*@[a-z0-9\.-]+)}i
          @config[:email] = Regexp.last_match(1)
        else
          if $VERBOSE
            puts "\tAddress #{@config[:email].inspect} looks wrong.  Using default"
          end
          @config[:email] = 'root@' + Symbiosis::Host.fqdn
        end

        config[:email]
      end

      #
      # Returns the default endpoint, defaulting to the live endpoint
      #
      def endpoint
        return config[:endpoint] if config[:endpoint].is_a?(String)

        @config[:endpoint] = ENDPOINT
      end

      #
      # Register the account RSA kay with the letsencrypt server
      #
      def register
        #
        # Send the key to the server.
        #
        account = client.new_account(contact: 'mailto:' + email, terms_of_service_agreed: true)

        true
      end

      #
      # Tests to see if we're registered by doing a pre-emptive authorize
      # request.
      #
      def registered?
        true
      rescue Acme::Client::Error::AccountDoesNotExist
        false
      end

      #
      # This does the authorization.  Returns true if the verification succeeds.
      #
      def verify_name(name)
        #
        # Set up the authorisation for the http01 challenge
        authorisation = acme_order.authorizations.find{|a| a.domain == name }
        challenge     = authorisation.http
        challenge_directory = File.join(docroot, File.dirname(challenge.filename))

        mkdir_p(challenge_directory)

        set_param(File.basename(challenge.filename), challenge.file_content, challenge_directory)

        vs = nil # Record the verify status

        if challenge.request_validation
          if $VERBOSE
            puts "\tRequesting verification for #{name} from #{endpoint}"
          end

          60.times do
            vs = challenge.status
            break unless vs == 'pending'

            sleep(1)
            challenge.reload
          end
        end

        unless $DEBUG
          set_param(File.basename(challenge.filename), false, challenge_directory)
        end

        if vs == 'valid'
          puts "\tSuccessfully verified #{name}" if $VERBOSE
          return true
        else
          if $VERBOSE
            puts "\t!! Unable to verify #{name} (status: #{vs})"
            puts "\t!! Check http://#{name}/#{challenge.filename} works."
          end
          return false
        end
      end

      def acme_certificates(request = self.request)
        return @acme_certificates if @acme_certificates

        unless request.is_a?(OpenSSL::X509::Request)
          raise ArgumentError, 'Invalid certificate request'
        end

        acme_order.finalize(csr: request)

        while acme_order.status == 'processing'
          sleep(1)
          acme_order.reload
        end

        certs = acme_order.certificate.split(/(?=-----BEGIN CERTIFICATE-----)/)
        x509_certs = certs.map{|c| OpenSSL::X509::Certificate.new(c) }

        @acme_certificates = x509_certs
      end

      #
      # Returns the signed X509 certificate for the request.
      #
      def certificate(request = self.request)
        acme_certificates(request).first
      end

      #
      # Returns the CA bundle as an array
      #
      def bundle(request = self.request)
        acme_certificates(request)[1..-1]
      end
    end

    PROVIDERS << LetsEncrypt
  end
end

Although this is now using the acme-v02.ppapi it isn't working:
No valid certificate sets found.
Fetching a new certificate from LetsEncrypt.
/var/lib/gems/2.1.0/gems/acme-client-2.0.5/lib/acme/client/resources/directory.rb:71: warning: instance variable @Directory not initialized
!! Failed: No account exists with the provided key

But if I replace the account.key file with one I have used in the past, it works.

What else should I have changed?

And what does ":71: warning: instance variable @Directory not initialized" mean?

I'm getting the same "warning: instance variable @Directory not initialized" - i don't really know Ruby, but (as a wild guess) it sounds like which-ever part of Symbiosis-ssl is initializing this class isn't passing the directory parameter to the constructor - wondering if it's an issue with the version of Symbiosis? (The version of the server i just tried this on was Debian Stretch).

Glad it's not just me!

@hairydogltd - FYI, digging around i just noticed someone at bytemark pushed an update to symbiosis on github about a month ago that adds support for acme-v2 - it's definitely in the stretch and jessie branches (there is a buster branch, it's not been added to it as a patch but i'm guessing it's abandoned given they never made buster available).

You might find if you remove Patricks' patch ( ie sudo rm /usr/local/lib/site_ruby/symbiosis/ssl/letsencrypt.rb -- it didn't overwrite the default classes ) then symbiosis-ssl then works (did for me on stretch).

(If i run grep 'symbiosis-httpd' /var/log/dpkg.lo* i can see symbiosis was updated on the server on the 4th February 2020, and when i run symbiosis-ssl --verbose it is saying it's using acme-v2 and successfully requested a new cert :-)

(there's a possibility if using an older debian like jessie the symbiosis patch may not have applied as i've discovered we have a jessie server thats not applying updates as the jessie-backports repo no-longer exists - i'm more likely to replace that server with a newer debian and migrate the sites rather than try to work out how to patch it tho given it's old)

The @directory not initialized message is a debug one relating to the acme gem, and nothing to worry about. It means that the variable @directory was being used before it was properly initialized, and in all likelihood is harmless. I don't know why you had problems with your key @hairydogltd.

Yes, Bytemark put out an update (hello @andrewladlow!) the only difference is that they have implemented their own library to access the LetsEncrypt API, whereas this solution uses the acme-client gem.

To uninstall this version fully, you can remove the extra lib as @fortybelowzero said, and also uninstall that gem.

$ sudo rm -rf /usr/local/lib/site_ruby/symbiosis/ssl
$ sudo gem uninstall acme-client

@patch0 - FYI i got the same "!! Failed: No account exists with the provided key" when i set up a new test virtual host on a server and asked it to request a letsencrypt certificate - i got the warning but no error when it was trying to request a certificate for a virtualhost that's already existed a while, so possibly something related to symbiosis-ssl creating a separate letsencrypt account key for each virtualhost and something's not happening quite rightly in that instance?

@andrewladlow's patch works fine tho 🥳, so all good (but thanks @patch0 for tackling it -- i was fully expecting to need to migrate vm's to sympl or manually run certbot)