Skip to content

Instantly share code, notes, and snippets.

Avatar
🐍
Exit code 143

Ian Whitestone ian-whitestone

🐍
Exit code 143
View GitHub Profile
@ian-whitestone
ian-whitestone / main.py
Last active May 3, 2021
Code for the "Testing SQL" blog post: http://ianwhitestone.work/testing-sql
View main.py
import os
from jinja2 import Environment, meta, Template
import pandas as pd
from pandas.testing import assert_frame_equal
import pandas.io.sql as psql
import psycopg2
# All these constants would likely live in separate files
CONNECTION = psycopg2.connect(
View simulations_for_post.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@ian-whitestone
ian-whitestone / a_a_sim.py
Last active Feb 16, 2021
Bayesian conversion test A/A simulations
View a_a_sim.py
import numpy as np
def bayesian_rate_comparison(
control_successes,
control_samples,
test_successes,
test_samples,
prior_a=0,
prior_b=0
):
View simulations.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@ian-whitestone
ian-whitestone / helpers.py
Created Dec 21, 2020
Code for the Dask cluster on demand blog post
View helpers.py
import logging
import time
from dataclasses import dataclass
from typing import Any, Optional, Tuple
import requests
import yaml
from distributed import Client
from distributed.security import Security
from googleapiclient import discovery
@ian-whitestone
ian-whitestone / Dockerfile
Created Sep 29, 2020
Code snippets for single node Dask cluster on GCP blog post
View Dockerfile
FROM python:3.7.6-buster
# Set the working directory
RUN mkdir /opt/app
WORKDIR /opt/app
# Copy poetry files into docker image
COPY pyproject.toml .
COPY poetry.lock .
@ian-whitestone
ian-whitestone / parse_unixtime.py
Last active Aug 5, 2020
Parse a unix timestamp with proper handling for milliseconds
View parse_unixtime.py
from __future__ import unicode_literals
import pytest
from starscream.pipeline.stage import TransformStage
from pyspark.sql import functions as F, types as T
from starscream.contract import Contract
from starscream.utils.dataframe import as_dicts, from_dicts
@ian-whitestone
ian-whitestone / reusable_stage.py
Last active Jul 29, 2020
Test starscream stages locally with `dev console --local`
View reusable_stage.py
from __future__ import unicode_literals
import pytest
import starscream.reusable_stages as RS
from starscream.contract import Contract
from starscream.utils.dataframe import as_dicts, from_dicts
import pyspark.sql.types as T
@ian-whitestone
ian-whitestone / sample_code.py
Created Jul 29, 2020
Test a generic starscream stage
View sample_code.py
from __future__ import unicode_literals
import pytest
from starscream.pipeline.stage import TransformStage
from pyspark.sql import functions as F, types as T
@ian-whitestone
ian-whitestone / stack_trace.txt
Last active Jul 2, 2020
Dask + pydantic pickling error
View stack_trace.txt
---------------------------------------------------------------------------
PicklingError Traceback (most recent call last)
<ipython-input-67-59601d7f9a60> in <module>
15
16 bag = db.from_sequence(data)
---> 17 bag.map(validate_data).compute()
~/Library/Caches/pypoetry/virtualenvs/domi-IWOYYLRr-py3.7/lib/python3.7/site-packages/dask/base.py in compute(self, **kwargs)
164 dask.base.compute
165 """