Skip to content

Instantly share code, notes, and snippets.

@rmarchei
Last active June 16, 2022 04:11
Show Gist options
  • Star 11 You must be signed in to star a gist
  • Fork 8 You must be signed in to fork a gist
  • Save rmarchei/98489c05f0898abe612eec916508f2bf to your computer and use it in GitHub Desktop.
Save rmarchei/98489c05f0898abe612eec916508f2bf to your computer and use it in GitHub Desktop.
route53 hook for dehydrated - python2 / python3 + boto2 version. Tested on Ubuntu 16.04
#!/usr/bin/env python
# How to use:
#
# Ubuntu 16.04: apt install -y python-boto OR apt install -y python3-boto
#
# Specify the default profile on aws/boto profile files or use the optional AWS_PROFILE env var:
# AWS_PROFILE=example ./dehydrated -c -d example.com -t dns-01 -k /etc/dehydrated/hooks/route53.py
#
# Manually specify hosted zone:
# HOSTED_ZONE=example.com AWS_PROFILE=example ./dehydrated -c -d example.com -t dns-01 -k /etc/dehydrated/hooks/route53.py
#
# More info about dehaydrated and dns challenge: https://github.com/lukas2511/dehydrated/wiki/Examples-for-DNS-01-hooks
# Using AWS Profiles: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-multiple-profiles
import os
import sys
from boto.route53 import *
from time import sleep
def route53_dns(domain, txt_challenge, action='upsert'):
conn = connection.Route53Connection()
action = action.upper()
if 'HOSTED_ZONE' in os.environ:
hosted_zone = os.environ['HOSTED_ZONE']
if not domain.endswith(hosted_zone):
raise Exception("Incorrect hosted zone for domain {0}".format(domain))
zone = conn.get_hosted_zone_by_name("{0}.".format(hosted_zone))
zone_id = zone['GetHostedZoneResponse']['HostedZone']['Id'].replace('/hostedzone/', '')
else:
zones = conn.get_all_hosted_zones()
for zone in zones['ListHostedZonesResponse']['HostedZones']:
if "{0}.".format(domain).endswith(zone['Name']):
zone_id = zone['Id'].replace('/hostedzone/', '')
break
else:
raise Exception("Hosted zone not found for domain {0}".format(domain))
name = u'_acme-challenge.{0}.'.format(domain)
record_set = conn.get_all_rrsets(zone_id, name=name, type='TXT')
challenges = [u'"{0}"'.format(txt_challenge)]
for r in record_set:
if r.name == name and r.type == 'TXT':
challenges += r.resource_records
change_set = record.ResourceRecordSets(conn, zone_id)
change = change_set.add_change("{0}".format(action), '_acme-challenge.{0}'.format(domain), type='TXT', ttl=60)
for c in set(challenges):
change.add_value(c)
try:
response = change_set.commit()
except Exception as e:
if action == "DELETE":
pass
else:
print(e.message, e.args)
if action in ('CREATE', 'UPSERT'):
# wait for DNS update
timeout = 300
sleep_time = 5
time_elapsed = 0
st = status.Status(conn, response['ChangeResourceRecordSetsResponse']['ChangeInfo'])
while st.update() != 'INSYNC' and time_elapsed <= timeout:
print("Waiting for DNS change to complete... (Elapsed {0} seconds)".format(time_elapsed))
sleep(sleep_time)
time_elapsed += sleep_time
if st.update() != 'INSYNC' and time_elapsed > timeout:
raise Exception("Timed out while waiting for DNS record to be ready. Waited {0} seconds".format(time_elapsed))
print("DNS change completed")
if __name__ == "__main__":
hook = sys.argv[1]
if len(sys.argv) > 2:
domain = sys.argv[2]
txt_challenge = sys.argv[4]
else:
domain = None
txt_challenge = None
if hook == "deploy_challenge":
action = 'upsert'
elif hook == "clean_challenge":
action = 'delete'
else:
sys.exit(0)
print("hook: {0}".format(hook))
print("domain: {0}".format(domain))
print("txt_challenge: {0}".format(txt_challenge))
route53_dns(domain, txt_challenge, action)
@clofresh
Copy link

clofresh commented Feb 9, 2017

Patch to avoid an exception when receiving an exit_hook

--- route53.py	2017-02-09 11:00:12.000000000 -0800
+++ route53-patched.py	2017-02-09 10:59:57.000000000 -0800
@@ -49,8 +49,12 @@

 if __name__ == "__main__":
     hook = sys.argv[1]
-    domain = sys.argv[2]
-    txt_challenge = sys.argv[4]
+    if len(sys.argv) > 2:
+        domain = sys.argv[2]
+        txt_challenge = sys.argv[4]
+    else:
+        domain = None
+        txt_challenge = None

     print("hook: {0}".format(hook))
     print("domain: {0}".format(domain))

@rmarchei
Copy link
Author

rmarchei commented Mar 2, 2017

Thanks! gist updated

@mirath
Copy link

mirath commented Feb 19, 2018

Patch that fixes a bug and adds several features:

  • BUG: dehydrated now passes an invalid hook when it runs, to force hook maintainers to ignore unknown hooks. This patch fixes this
  • FEATURE: If the target AWS account has many Zones with the same postfix (eg. example.com and foo.example.com) the script would always return the shortest one (so it would return example.com for a record for baz.foo.example.com). This patch makes it so it returns the one with the longest postfix
  • FEATURE: The script waits 30 fixed seconds for the Route53 record to be up, and then assumes that it is up and exits, but sometimes records take more than 30s to be ready in Route53. This patch adds logic that polls the Route53 API every 5s for the status of the change and doesn't exit until either AWS reports the change as "INSYNC" or a timeout (of 5 minutes) occurs. It also provides output while it waits so the operator knows what the script is doing and how much time has it been waiting.
33a34,35
>       candidate_zones = []
>       domain_dot = "{0}.".format(domain)
35,38c37,40
<         if "{0}.".format(domain).endswith(zone['Name']):
<           zone_id = zone['Id'].replace('/hostedzone/', '')
<           break
<       else:
---
>         if domain_dot.endswith(zone['Name']):
>           candidate_zones.append((domain_dot.find(zone['Name']), zone['Id'].replace('/hostedzone/', '')))
> 
>       if len(candidate_zones) == 0:
41,44c43,49
<     change_set = record.ResourceRecordSets(conn, zone_id)
<     change = change_set.add_change("{0}".format(action.upper()), '_acme-challenge.{0}'.format(domain), type='TXT', ttl=60)
<     change.add_value('"{0}"'.format(txt_challenge))
<     change_set.commit()
---
>       candidate_zones.sort()
>       zone_id = candidate_zones[0][1]
> 
>       change_set = record.ResourceRecordSets(conn, zone_id)
>       change = change_set.add_change("{0}".format(action.upper()), '_acme-challenge.{0}'.format(domain), type='TXT', ttl=60)
>       change.add_value('"{0}"'.format(txt_challenge))
>       response = change_set.commit()
48c53,65
<       sleep(30)
---
>       timeout = 300
>       sleep_time = 5
>       time_elapsed = 0
>       st = status.Status(conn, response['ChangeResourceRecordSetsResponse']['ChangeInfo'])
>       while st.update() != 'INSYNC' and time_elapsed <= timeout:
>         print("Waiting for DNS change to complete... (Elapsed {0} seconds)".format(time_elapsed))
>         sleep(sleep_time)
>         time_elapsed += sleep_time
> 
>       if st.update() != 'INSYNC' and time_elapsed > timeout:
>         raise Exception("Timed out while waiting for DNS record to be ready. Waited {0} seconds".format(time_elapsed))
> 
>       print("DNS change completed")
52c69,70
<     if len(sys.argv) > 2:
---
> 
>     if hook == "deploy_challenge":
54a73,89
>         action = 'upsert'
>     elif hook == "clean_challenge":
>         domain = sys.argv[2]
>         txt_challenge = sys.argv[4]
>         action = 'delete'
>     elif hook == "startup_hook":
>         print("Ignoring startup_hook")
>         exit(0)
>     elif hook == "exit_hook":
>         print("Ignoring exit_hook")
>         exit(0)
>     elif hook == "deploy_cert":
>         print("Ignoring deploy_cert hook")
>         exit(0)
>     elif hook == "unchanged_cert":
>         print("Ignoring unchanged_cert hook")
>         exit(0)
56,57c91,92
<         domain = None
<         txt_challenge = None
---
>         print("Ignoring unknown hook %s", hook)
>         exit(0)
63,66c98
<     if hook == "deploy_challenge":
<         route53_dns(domain, txt_challenge, 'upsert')
<     elif hook == "clean_challenge":
<         route53_dns(domain, txt_challenge, 'delete')
---
>     route53_dns(domain, txt_challenge, action)

I leave the complete code here, as the diff is not easy to analize

#!/usr/bin/env python

# How to use:
#
# Ubuntu 16.04: apt install -y python-boto OR apt install -y python3-boto
#
# Specify the default profile on aws/boto profile files or use the optional AWS_PROFILE env var:
# AWS_PROFILE=example ./dehydrated -c -d example.com -t dns-01 -k /etc/dehydrated/hooks/route53.py
#
# Manually specify hosted zone:
# HOSTED_ZONE=example.com AWS_PROFILE=example ./dehydrated -c -d example.com -t dns-01 -k /etc/dehydrated/hooks/route53.py
#
# More info about dehaydrated and dns challenge: https://github.com/lukas2511/dehydrated/wiki/Examples-for-DNS-01-hooks
# Using AWS Profiles: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-multiple-profiles

import os
import sys
from boto.route53 import *
from time import sleep


def route53_dns(domain, txt_challenge, action='upsert'):

    conn = connection.Route53Connection()

    if 'HOSTED_ZONE' in os.environ:
        hosted_zone = os.environ['HOSTED_ZONE']
        if not domain.endswith(hosted_zone):
            raise Exception("Incorrect hosted zone for domain {0}".format(domain))
        zone = conn.get_hosted_zone_by_name("{0}.".format(hosted_zone))
        zone_id = zone['GetHostedZoneResponse']['HostedZone']['Id'].replace('/hostedzone/', '')
    else:
        zones = conn.get_all_hosted_zones()
        candidate_zones = []
        domain_dot = "{0}.".format(domain)
        for zone in zones['ListHostedZonesResponse']['HostedZones']:
            if domain_dot.endswith(zone['Name']):
                candidate_zones.append((domain_dot.find(zone['Name']), zone['Id'].replace('/hostedzone/', '')))

        if len(candidate_zones) == 0:
            raise Exception("Hosted zone not found for domain {0}".format(domain))

        candidate_zones.sort()
        zone_id = candidate_zones[0][1]

        change_set = record.ResourceRecordSets(conn, zone_id)
        change = change_set.add_change("{0}".format(action.upper()), '_acme-challenge.{0}'.format(domain), type='TXT', ttl=60)
        change.add_value('"{0}"'.format(txt_challenge))
        response = change_set.commit()

    if action.upper() == 'UPSERT':
        # wait for DNS update
        timeout = 300
        sleep_time = 5
        time_elapsed = 0
        st = status.Status(conn, response['ChangeResourceRecordSetsResponse']['ChangeInfo'])
        while st.update() != 'INSYNC' and time_elapsed <= timeout:
            print("Waiting for DNS change to complete... (Elapsed {0} seconds)".format(time_elapsed))
            sleep(sleep_time)
            time_elapsed += sleep_time

        if st.update() != 'INSYNC' and time_elapsed > timeout:
            raise Exception("Timed out while waiting for DNS record to be ready. Waited {0} seconds".format(time_elapsed))

        print("DNS change completed")

if __name__ == "__main__":
    hook = sys.argv[1]

    if hook == "deploy_challenge":
        domain = sys.argv[2]
        txt_challenge = sys.argv[4]
        action = 'upsert'
    elif hook == "clean_challenge":
        domain = sys.argv[2]
        txt_challenge = sys.argv[4]
        action = 'delete'
    elif hook == "startup_hook":
        print("Ignoring startup_hook")
        exit(0)
    elif hook == "exit_hook":
        print("Ignoring exit_hook")
        exit(0)
    elif hook == "deploy_cert":
        print("Ignoring deploy_cert hook")
        exit(0)
    elif hook == "unchanged_cert":
        print("Ignoring unchanged_cert hook")
        exit(0)
    else:
        print("Ignoring unknown hook %s", hook)
        exit(0)

    print("hook: {0}".format(hook))
    print("domain: {0}".format(domain))
    print("txt_challenge: {0}".format(txt_challenge))

    route53_dns(domain, txt_challenge, action)

@Tea23
Copy link

Tea23 commented Mar 25, 2018

@mirath Your updates to the script are amazing and it's been working well for a while!

However Let's Encrypt just introduced support for wildcard certs and it fails in these instances. When requesting a cert for *.domain.com and domain.com, the hook asks for two challenges for domain.com. Before responding to a challenge, it deletes the first one and the response fails as the valid challenge no longer exists.

I don't fully understand boto and I've been scratching my head for a few days trying to come up with a way to fix it but I'm at a loss. Any input you might have would be much appreciated.

@rmarchei
Copy link
Author

rmarchei commented Apr 11, 2018

I don't have notifications enabled and I was missing those comments. I'll take a look at the script with latest dehydrated version today!

@Tea23 are you using this script on linux/unix or windows?

@Tea23
Copy link

Tea23 commented Apr 12, 2018

@rmarchei I'm running this on Ubuntu 16.04

@mirath
Copy link

mirath commented Apr 30, 2018

@Tea23 @rmarchei

I tried to reproduce the error you mentioned but I wasn't able to. I created a certificate only for *.example.com and it worked, then I added the example.com domain and it still worked.

Letsencrypt has cached those challenges, so I'll wait a bit and try generating *.example.com and example.com at the same time. Also note that I'm using the staging environment.

Do you still have issues with the script or have you been able to find a workaround? Could you give us more information, like the exact error message with sensitive information cesored?

@gytisgreitai
Copy link

@mirath your changes break the script when HOSTED_ZONE is set, the ident on line change_set = record.ResourceRecordSets(conn, zone_id) is wrong, it should not be under if

@mirath
Copy link

mirath commented May 4, 2018

@gytisgreitai you are right. I'll fix it, though if you have a patch feel free to upload it to the gist

@Tea23
Copy link

Tea23 commented May 19, 2018

@mirath

This is attempting to produce a cert for *.scoutlink.net and scoutlink.net.

Ignoring startup_hook
Processing *.scoutlink.net with alternative names: scoutlink.net 
 + Checking domain name(s) of existing cert... unchanged.
 + Checking expire date of existing cert...
 + Valid till Jun 15 23:55:16 2018 GMT (Less than 30 days). Renewing!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 2 authorizations URLs from the CA
 + Handling authorization for scoutlink.net
 + Handling authorization for scoutlink.net
 + 2 pending challenge(s)
 + Deploying challenge tokens...
hook: deploy_challenge
domain: scoutlink.net
txt_challenge: 047GnpkfdgScW4-u7_zljXXOxs1snEUDrPAMzyiBy-Q
Waiting for DNS change to complete... (Elapsed 0 seconds)
Waiting for DNS change to complete... (Elapsed 5 seconds)
Waiting for DNS change to complete... (Elapsed 10 seconds)
Waiting for DNS change to complete... (Elapsed 15 seconds)
Waiting for DNS change to complete... (Elapsed 20 seconds)
Waiting for DNS change to complete... (Elapsed 25 seconds)
Waiting for DNS change to complete... (Elapsed 30 seconds)
Waiting for DNS change to complete... (Elapsed 35 seconds)
Waiting for DNS change to complete... (Elapsed 40 seconds)
DNS change completed
hook: deploy_challenge
domain: scoutlink.net
txt_challenge: YaqTDtzLjFKncHxrfFAI1iJrlyt7svFZIwkaD3jhl0g
Waiting for DNS change to complete... (Elapsed 0 seconds)
Waiting for DNS change to complete... (Elapsed 5 seconds)
Waiting for DNS change to complete... (Elapsed 10 seconds)
Waiting for DNS change to complete... (Elapsed 15 seconds)
Waiting for DNS change to complete... (Elapsed 20 seconds)
Waiting for DNS change to complete... (Elapsed 25 seconds)
Waiting for DNS change to complete... (Elapsed 30 seconds)
Waiting for DNS change to complete... (Elapsed 35 seconds)
DNS change completed
 + Responding to challenge for scoutlink.net authorization...
hook: clean_challenge
domain: scoutlink.net
txt_challenge: 047GnpkfdgScW4-u7_zljXXOxs1snEUDrPAMzyiBy-Q
Traceback (most recent call last):
  File "/etc/dehydrated/hook.py", line 88, in <module>
    route53_dns(domain, txt_challenge, action)
  File "/etc/dehydrated/hook.py", line 38, in route53_dns
    response = change_set.commit()
  File "/usr/lib/python2.7/dist-packages/boto/route53/record.py", line 168, in commit
    return self.connection.change_rrsets(self.hosted_zone_id, self.to_xml())
  File "/usr/lib/python2.7/dist-packages/boto/route53/connection.py", line 475, in change_rrsets
    body)
boto.route53.exception.DNSServerError: DNSServerError: 400 Bad Request
<?xml version="1.0"?>
<ErrorResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><Error><Type>Sender</Type><Code>InvalidChangeBatch</Code><Message>Tried to delete resource record set [name='_acme-challenge.scoutlink.net.', type='TXT'] but the values provided do not match the current values</Message></Error><RequestId>e56f9833-5b74-11e8-bd24-0713c2fe4f99</RequestId></ErrorResponse>

@niall-byrne
Copy link

The indentation in the above code seems to be off, this worked for me:

#!/usr/bin/env python

# How to use:
#
# Ubuntu 16.04: apt install -y python-boto OR apt install -y python3-boto
#
# Specify the default profile on aws/boto profile files or use the optional AWS_PROFILE env var:
# AWS_PROFILE=example ./dehydrated -c -d example.com -t dns-01 -k /etc/dehydrated/hooks/route53.py
#
# Manually specify hosted zone:
# HOSTED_ZONE=example.com AWS_PROFILE=example ./dehydrated -c -d example.com -t dns-01 -k /etc/dehydrated/hooks/route53.py
#
# More info about dehaydrated and dns challenge: https://github.com/lukas2511/dehydrated/wiki/Examples-for-DNS-01-hooks
# Using AWS Profiles: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-multiple-profiles

import os
import sys
from boto.route53 import *
from time import sleep


def route53_dns(domain, txt_challenge, action='upsert'):

    conn = connection.Route53Connection()

    if 'HOSTED_ZONE' in os.environ:
        hosted_zone = os.environ['HOSTED_ZONE']
        if not domain.endswith(hosted_zone):
            raise Exception("Incorrect hosted zone for domain {0}".format(domain))
        zone = conn.get_hosted_zone_by_name("{0}.".format(hosted_zone))
        zone_id = zone['GetHostedZoneResponse']['HostedZone']['Id'].replace('/hostedzone/', '')
    else:
        zones = conn.get_all_hosted_zones()
        candidate_zones = []
        domain_dot = "{0}.".format(domain)
        for zone in zones['ListHostedZonesResponse']['HostedZones']:
            if domain_dot.endswith(zone['Name']):
                candidate_zones.append((domain_dot.find(zone['Name']), zone['Id'].replace('/hostedzone/', '')))

        if len(candidate_zones) == 0:
            raise Exception("Hosted zone not found for domain {0}".format(domain))

        candidate_zones.sort()
        zone_id = candidate_zones[0][1]

    change_set = record.ResourceRecordSets(conn, zone_id)
    change = change_set.add_change("{0}".format(action.upper()), '_acme-challenge.{0}'.format(domain), type='TXT', ttl=60)
    change.add_value('"{0}"'.format(txt_challenge))
    response = change_set.commit()

    if action.upper() == 'UPSERT':
        # wait for DNS update
        timeout = 300
        sleep_time = 5
        time_elapsed = 0
        st = status.Status(conn, response['ChangeResourceRecordSetsResponse']['ChangeInfo'])
        while st.update() != 'INSYNC' and time_elapsed <= timeout:
            print("Waiting for DNS change to complete... (Elapsed {0} seconds)".format(time_elapsed))
            sleep(sleep_time)
            time_elapsed += sleep_time

        if st.update() != 'INSYNC' and time_elapsed > timeout:
            raise Exception("Timed out while waiting for DNS record to be ready. Waited {0} seconds".format(time_elapsed))

        print("DNS change completed")

if __name__ == "__main__":
    hook = sys.argv[1]

    if hook == "deploy_challenge":
        domain = sys.argv[2]
        txt_challenge = sys.argv[4]
        action = 'upsert'
    elif hook == "clean_challenge":
        domain = sys.argv[2]
        txt_challenge = sys.argv[4]
        action = 'delete'
    elif hook == "startup_hook":
        print("Ignoring startup_hook")
        exit(0)
    elif hook == "exit_hook":
        print("Ignoring exit_hook")
        exit(0)
    elif hook == "deploy_cert":
        print("Ignoring deploy_cert hook")
        exit(0)
    elif hook == "unchanged_cert":
        print("Ignoring unchanged_cert hook")
        exit(0)
    else:
        print("Ignoring unknown hook %s", hook)
        exit(0)

    print("hook: {0}".format(hook))
    print("domain: {0}".format(domain))
    print("txt_challenge: {0}".format(txt_challenge))

    route53_dns(domain, txt_challenge, action)```

@mirath
Copy link

mirath commented May 26, 2018

@niall-byrne Thanks!!!

@nh2
Copy link

nh2 commented May 27, 2018

Is any of the versions above able to use wildcards? I'm still getting the error but the values provided do not match the current values as mentioned above (https://gist.github.com/rmarchei/98489c05f0898abe612eec916508f2bf#gistcomment-2594451).

@nh2
Copy link

nh2 commented May 27, 2018

I've written a fork of this gist to:

  • Implement wildcard support
  • Allow HOOK_CHAIN="yes" for much faster batch confirmation (only need to wait for DNS propagation once, not once per domain)

https://gist.github.com/nh2/f744ac591e95f0c25b501db00cf7c71a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment