Skip to content

Instantly share code, notes, and snippets.

@jastang
Last active August 30, 2018 20:04
Show Gist options
  • Save jastang/0397e1446dc0e27ce450838f49b77173 to your computer and use it in GitHub Desktop.
Save jastang/0397e1446dc0e27ce450838f49b77173 to your computer and use it in GitHub Desktop.
Lambda function for metadata validation
def lambda_handler(event, context):
records = event['Records']
for record in records:
b = record['s3']['bucket']['name']
s3_file = record['s3']['object']['key']
# Load the metadata validation configuration for this client
conf = json.loads(s3.meta.client.get_object(Bucket=b, Key=VALIDATION_CONFIG)['Body'].read(),
object_pairs_hook=OrderedDict)
object_summary = s3.ObjectSummary(b, s3_file)
# See if this newly uploaded object passes metadata validations and can be catalogued by Glue
validator = Validator(conf=conf, s3_object=object_summary)
objects_to_move = validator.validate()
for s3o in objects_to_move:
# Move the data to an auto-generated S3 partition
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment