Skip to content

Instantly share code, notes, and snippets.

@johnib
Created April 28, 2017 13:25
Show Gist options
  • Save johnib/f15827f960fa21ee87be02589a5312ea to your computer and use it in GitHub Desktop.
Save johnib/f15827f960fa21ee87be02589a5312ea to your computer and use it in GitHub Desktop.
@retry(wait_exponential_multiplier=250,
wait_exponential_max=4500,
stop_max_attempt_number=8,
retry_on_result=lambda failures_count: failures_count > 0)
def put():
global non_delivered_tweets
logger.info("Executing Firehose put_batch command on {} tweets".format(len(non_delivered_tweets)))
response = firehose.put_record_batch(DeliveryStreamName=firehose_stream_name, Records=non_delivered_tweets)
failures_count = response["FailedPutCount"]
failures_list = []
if failures_count > 0:
for index, request_response in enumerate(response["RequestResponses"]):
if "ErrorCode" in request_response:
failures_list.append(non_delivered_tweets[index])
non_delivered_tweets = failures_list
return failures_count
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment