Skip to content

Instantly share code, notes, and snippets.

View ian-whitestone's full-sized avatar
Exit code 143

Ian Whitestone ian-whitestone

Exit code 143
View GitHub Profile
ian-whitestone /
Created August 29, 2018 13:02
Setting up jupyter lab on a ubuntu instance

Setting Up Jupyter Lab on an EC2

  1. Install Jupyter Lab

conda install -c conda-forge jupyterlab

  1. Create certs

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout mycert.pem -out mycert.pem

ian-whitestone /
Created September 14, 2018 18:30
Pandas missing values summary
# Source:
import pandas as pd
# Number of missing in each column
missing = pd.DataFrame(data.isnull().sum()).rename(columns = {0: 'total'})
# Create a percentage missing
missing['percent'] = missing['total'] / len(data)
ian-whitestone /
Created September 24, 2018 01:18
Generating fake data to compare dask and spark for reading avro files into a dataframe
"""Generate a bunch of fake avro data and upload to s3
Running in python 3.7. Installed the following:
- pip install Faker
- pip install fastavro
- pip install boto3
- pip install graphviz
- brew install graphviz
import sys
import dask.bag as db
def gt(x):
return x > 3
def even(x):
return x % 2 == 0
ian-whitestone /
Created March 16, 2020 02:17
Script for running executing python jobs in docker
# User defined constants
# Input Arguments
ian-whitestone /
Last active March 17, 2020 01:01
Script for sending failure notifications in slack
Trigger slack notifications
import argparse
import logging
import os
from slack.web.client import WebClient
LOGGER = logging.getLogger(__name__)
import pickle
def proxy_request(url):
proxy_response =
data={'url': url}
if not proxy_response.ok:
ian-whitestone / App.js
Created May 30, 2020 18:52
Learning react useState hooks by doing...
import React from 'react';
import Container from '@material-ui/core/Container';
import Typography from '@material-ui/core/Typography';
import Box from '@material-ui/core/Box';
import { makeStyles } from '@material-ui/core/styles';
import Link from '@material-ui/core/Link';
import SvgIcon from '@material-ui/core/SvgIcon';
import Button from '@material-ui/core/Button';
ian-whitestone /
Created June 28, 2020 20:23
A simple progress bar that just shows the total number of "things" processed and the time elapsed
A simple progress bar that just shows the total number of "things" processed
and the time elapsed
import time
from datetime import timedelta
from rich.progress import Progress, ProgressColumn, Task, TaskID, Text, TextColumn