Ansible AWSRetry Decorator

ansible_awsretry_decorator.md

Some example usages of AWSRetry (implementation of CloudRetry)

• exponential_backoff
• jittered_backoff

Though current interest is around an ongoing issue with AWS throttling api calls... AWSRetry deals more generally with eventual consistency issue when working with aws resources... delete returns but all references are not yet freed, create returns but the resource isn't yet fully realized, and so on.

Also seems like you should be able to tweak, at least, the parameters to the backoff function from your ansible code. Exposing a dictionary of retry params would be useful while we are testing this out.

Lastly, how do we make this sustainable? We'll want to make minimum touches to these modules so upgrading ansible versions isn't a big pain.

Exponential versus Jittered Backoff

AWS Architecture blog backoff post is really interesting. Our primary usecase is to get past throttoling issues. However, if you have high concurrency you want more than longer and longer delays... you want smooth out access over time versus have bunches of accesses periodically.

From ec2_asg

backoff_params = dict(tries=10, delay=3, backoff=1.5)

@AWSRetry.backoff(**backoff_params)

From aws_api_gateway

retry_params = {"tries": 10, "delay": 5, "backoff": 1.2}

@AWSRetry.backoff(**retry_params)

cloudformation_facts... Saving the backoff wrapper

The backoff decorator does return a function which you can, in turn, use to wrap existing functions. Also a pythonic approach to overwriting existing functions with the backoff_wrapper. For quick hacks and testing of the AWSRetry this is less invasive than decorating every method and changing code in an existing method.

I did notice some of these methods are catching all exceptions, which seems to be in conflict with

            backoff_wrapper = AWSRetry.jittered_backoff(retries=10, delay=3, max_delay=30)
            self.client.describe_stacks = backoff_wrapper(self.client.describe_stacks)
            self.client.list_stack_resources = backoff_wrapper(self.client.list_stack_resources)
            self.client.describe_stack_events = backoff_wrapper(self.client.describe_stack_events)
            self.client.get_stack_policy = backoff_wrapper(self.client.get_stack_policy)
            self.client.get_template = backoff_wrapper(self.client.get_template)

Guidelines for AWS Modules

Has an interesting example of wrapping boto3 paged output (paginators/markers).

@AWSRetry.exponential_backoff(tries=5, delay=5)
def describe_some_resource_with_backoff(client, **kwargs):
     paginator = client.get_paginator('describe_some_resource')
     return paginator.paginate(**kwargs).build_full_result()['SomeResource']


def describe_some_resource(client, module):
    filters = ansible_dict_to_boto3_filter_list(module.params['filters'])
    try:
        return describe_some_resource_with_backoff(client, Filters=filters)
    except botocore.exceptions.ClientError as e:
        module.fail_json_aws(e, msg="Could not describe some resource")