Skip to content

Instantly share code, notes, and snippets.

David Howell davoscollective

Block or report user

Report or block davoscollective

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@davoscollective
davoscollective / logging_example.py
Created Nov 20, 2018
Python Logging with a decent format
View logging_example.py
import logging
log = logging.getLogger()
log.setLevel(logging.INFO)
log_handler = logging.StreamHandler()
log_formatter = logging.Formatter(
"[%(levelname)s] %(asctime)s.%(msecs)03d line:%(lineno)d %(funcName)s %(message)s",
style="%",
datefmt="%Y-%m-%dT%H:%M:%S")
@davoscollective
davoscollective / download_zeppelin_from_s3.py
Last active Jun 26, 2018
Download Zeppelin Notebooks from S3 and organise into folders
View download_zeppelin_from_s3.py
import boto3
from pprint import pprint
import json
from pathlib import Path
region = 'ap-southeast-2'
s3client = boto3.client('s3', region_name=region)
paginator = s3client.get_paginator('list_objects')
@davoscollective
davoscollective / fizzbuzz.scala
Created May 31, 2018
fizzbuzz variants in SQL, Scala and Python, displayed as tables in Apache Zeppelin
View fizzbuzz.scala
//idiomatic scala version
val fizzbuzz: (Int => String) = (i) => (i % 3, i % 5) match {
case (0, 0) => "fizzbuzz"
case (0, _) => "fizz"
case (_, 0) => "buzz"
case _ => ""
}
1 to 100 map fizzbuzz
@davoscollective
davoscollective / crypter.py
Last active May 24, 2018
Python command line tool plus library to use cryptography.Fernet for symmetric encryption/decryption of secrets file
View crypter.py
"""
A utility to encrypt/decrypt a file using a fernet key.
Useful for storing credentials or other sensitive data.
You will need a fernet.key file, which you can also create with this utility.
-------
By leveraging the magical click library, this can be used at the command line. Because the click interface
then calls another function internally, it can also be used by importing that crypter function into your python
@davoscollective
davoscollective / kmscrypter.py
Last active May 24, 2018
Python command line tool plus library to use KMS for symmetric encryption/decryption of secrets file
View kmscrypter.py
"""
A utility to encrypt/decrypt a file using an AWS KMS key.
Useful for storing credentials or other sensitive data.
You will need a KMS key created in AWS, and have permission to use it. The script defaults to
a key with alias "alias/botoUsage".
This script intentionally doesn't support providing key/secret key credentials directly to boto3.
boto3 uses a cascading approach to searching for credentials for connecting to AWS, and this script
needs to run in one of those contexts i.e. running under an IAM role or with AWS Key / Secret key
@davoscollective
davoscollective / json_or_xml.py
Created Apr 23, 2018
Python class with init method to parse json / xml to test valid formats. Example of @Property usage and some EAFP style exception handling.
View json_or_xml.py
from __future__ import print_function
import xml.etree.cElementTree as ET
import json
class ReportDocument(object):
"""
ReportDocument is an abstraction of a JSON, XML or neither file type.
"""
def __init__(self, report_input_file=None, *args, **kwargs):
self._is_json_report_file=False
@davoscollective
davoscollective / time_del.py
Created Apr 20, 2018 — forked from zigg/time_del.py
Time various methods of removing a possibly-present item from a dict
View time_del.py
#!/usr/bin/python
import time
def new_d():
return {
1: 2, 3: 4, 5: 6, 7: 8, 9: 10,
11: 12, 13: 14, 15: 16, 17: 18, 19: 20
}
View PySpark_Vectorized_UDFs.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@davoscollective
davoscollective / PySpark_createDataFrame_with_Arrow.ipynb
Created Apr 19, 2018 — forked from BryanCutler/PySpark_createDataFrame_with_Arrow.ipynb
How to create a Spark DataFrame from Pandas or NumPy with Arrow
View PySpark_createDataFrame_with_Arrow.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@davoscollective
davoscollective / lds.py
Last active Apr 18, 2018
List versus generator for a code interview question - find largest sum of digits for some range of exponential expressions
View lds.py
"""
lds - large digit sum
"""
from itertools import product
from timeit import timeit
import tracemalloc
import matplotlib.pyplot as plt
You can’t perform that action at this time.