Skip to content

Instantly share code, notes, and snippets.

David Howell davoscollective

Block or report user

Report or block davoscollective

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View PySpark_Vectorized_UDFs.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@davoscollective
davoscollective / time_del.py
Created Apr 20, 2018 — forked from zigg/time_del.py
Time various methods of removing a possibly-present item from a dict
View time_del.py
#!/usr/bin/python
import time
def new_d():
return {
1: 2, 3: 4, 5: 6, 7: 8, 9: 10,
11: 12, 13: 14, 15: 16, 17: 18, 19: 20
}
@davoscollective
davoscollective / json_or_xml.py
Created Apr 23, 2018
Python class with init method to parse json / xml to test valid formats. Example of @Property usage and some EAFP style exception handling.
View json_or_xml.py
from __future__ import print_function
import xml.etree.cElementTree as ET
import json
class ReportDocument(object):
"""
ReportDocument is an abstraction of a JSON, XML or neither file type.
"""
def __init__(self, report_input_file=None, *args, **kwargs):
self._is_json_report_file=False
@davoscollective
davoscollective / kmscrypter.py
Last active May 24, 2018
Python command line tool plus library to use KMS for symmetric encryption/decryption of secrets file
View kmscrypter.py
"""
A utility to encrypt/decrypt a file using an AWS KMS key.
Useful for storing credentials or other sensitive data.
You will need a KMS key created in AWS, and have permission to use it. The script defaults to
a key with alias "alias/botoUsage".
This script intentionally doesn't support providing key/secret key credentials directly to boto3.
boto3 uses a cascading approach to searching for credentials for connecting to AWS, and this script
needs to run in one of those contexts i.e. running under an IAM role or with AWS Key / Secret key
@davoscollective
davoscollective / crypter.py
Last active May 24, 2018
Python command line tool plus library to use cryptography.Fernet for symmetric encryption/decryption of secrets file
View crypter.py
"""
A utility to encrypt/decrypt a file using a fernet key.
Useful for storing credentials or other sensitive data.
You will need a fernet.key file, which you can also create with this utility.
-------
By leveraging the magical click library, this can be used at the command line. Because the click interface
then calls another function internally, it can also be used by importing that crypter function into your python
@davoscollective
davoscollective / fizzbuzz.scala
Created May 31, 2018
fizzbuzz variants in SQL, Scala and Python, displayed as tables in Apache Zeppelin
View fizzbuzz.scala
//idiomatic scala version
val fizzbuzz: (Int => String) = (i) => (i % 3, i % 5) match {
case (0, 0) => "fizzbuzz"
case (0, _) => "fizz"
case (_, 0) => "buzz"
case _ => ""
}
1 to 100 map fizzbuzz
@davoscollective
davoscollective / download_zeppelin_from_s3.py
Last active Jun 26, 2018
Download Zeppelin Notebooks from S3 and organise into folders
View download_zeppelin_from_s3.py
import boto3
from pprint import pprint
import json
from pathlib import Path
region = 'ap-southeast-2'
s3client = boto3.client('s3', region_name=region)
paginator = s3client.get_paginator('list_objects')
@davoscollective
davoscollective / logging_example.py
Created Nov 20, 2018
Python Logging with a decent format
View logging_example.py
import logging
log = logging.getLogger()
log.setLevel(logging.INFO)
log_handler = logging.StreamHandler()
log_formatter = logging.Formatter(
"[%(levelname)s] %(asctime)s.%(msecs)03d line:%(lineno)d %(funcName)s %(message)s",
style="%",
datefmt="%Y-%m-%dT%H:%M:%S")
@davoscollective
davoscollective / dynamodb_migrate_region.py
Created Jul 17, 2019
A python 3 script to migrate a dynamodb table from one region to another with pagination
View dynamodb_migrate_region.py
#!/usr/bin/env python3
"""
A paginated version of
https://itnext.io/replicating-amazon-dynamodb-tables-in-different-region-1299b3eca2ae
Differences
* Doesn't create the table for you, we use terraform for that.
* Uses batch_writer context which handles batching, buffering, retries etc
* Pages the source table for very large table migrations
You can’t perform that action at this time.