KeenIO - move streamed files from default folder structure to alternate folder structure

move_keenio_files.py

#!/usr/bin/python3

import boto3
import argparse
import time
import re

NAME_OF_BUCKET = "name-of-s3-bucket"
ACCOUNT_ID = "account-id-from-keen-io-should-be-large-hex-string"

# FROM: s3://NAME_OF_BUCKET/ACCOUNT_ID/2018-12-04T19:20:00.000Z/event_name/ACCOUNT_ID-event_name-2018-12-04T19:20:00.000Z.json.gz
# TO:   s3://NAME_OF_BUCKET/ACCOUNT_ID/event_name/2018-12-05T11:30:00.000Z/ACCOUNT_ID-event_name-2018-12-05T11:30:00.000Z.json.gz

s3 = boto3.resource('s3')
bucket = s3.Bucket(NAME_OF_BUCKET)

p = re.compile("(.*)\/(.*)\/(.*)\/(.*)")

i = 0
for object in bucket.objects.filter(Prefix=(ACCOUNT_ID+'/201')):
    matches = p.match(object.key).groups()
    from_filename = object.key
    to_filename = "/".join((matches[0], matches[2], matches[1], matches[3]))
    backup_filename = "/".join(("old", matches[1], matches[2], matches[3]))
    s3.Object(NAME_OF_BUCKET, to_filename).copy_from(CopySource=(NAME_OF_BUCKET + '/' + from_filename))
    s3.Object(NAME_OF_BUCKET, backup_filename).copy_from(CopySource=(NAME_OF_BUCKET + '/' + from_filename))
    s3.Object(NAME_OF_BUCKET, from_filename).delete()
    print(from_filename + " -> " + to_filename)
    i = i+1

print(i)

move_keenio_files.README.md

According to documentation in https://keen.io/docs/streams/amazon-s3/, we were using the default format. For some applications (AWS Athena mostly), we began using the alternate format, and we needed to move the files in the old places to the new folder structure.

This is a very simple script provided as is. Comments are welcome :)