Skip to content

Instantly share code, notes, and snippets.

Avatar
🐍
Linting

Ian Whitestone ian-whitestone

🐍
Linting
View GitHub Profile
@ian-whitestone
ian-whitestone / Dockerfile
Created Sep 29, 2020
Code snippets for single node Dask cluster on GCP blog post
View Dockerfile
FROM python:3.7.6-buster
# Set the working directory
RUN mkdir /opt/app
WORKDIR /opt/app
# Copy poetry files into docker image
COPY pyproject.toml .
COPY poetry.lock .
@ian-whitestone
ian-whitestone / parse_unixtime.py
Last active Aug 5, 2020
Parse a unix timestamp with proper handling for milliseconds
View parse_unixtime.py
from __future__ import unicode_literals
import pytest
from starscream.pipeline.stage import TransformStage
from pyspark.sql import functions as F, types as T
from starscream.contract import Contract
from starscream.utils.dataframe import as_dicts, from_dicts
@ian-whitestone
ian-whitestone / reusable_stage.py
Last active Jul 29, 2020
Test starscream stages locally with `dev console --local`
View reusable_stage.py
from __future__ import unicode_literals
import pytest
import starscream.reusable_stages as RS
from starscream.contract import Contract
from starscream.utils.dataframe import as_dicts, from_dicts
import pyspark.sql.types as T
@ian-whitestone
ian-whitestone / sample_code.py
Created Jul 29, 2020
Test a generic starscream stage
View sample_code.py
from __future__ import unicode_literals
import pytest
from starscream.pipeline.stage import TransformStage
from pyspark.sql import functions as F, types as T
@ian-whitestone
ian-whitestone / stack_trace.txt
Last active Jul 2, 2020
Dask + pydantic pickling error
View stack_trace.txt
---------------------------------------------------------------------------
PicklingError Traceback (most recent call last)
<ipython-input-67-59601d7f9a60> in <module>
15
16 bag = db.from_sequence(data)
---> 17 bag.map(validate_data).compute()
~/Library/Caches/pypoetry/virtualenvs/domi-IWOYYLRr-py3.7/lib/python3.7/site-packages/dask/base.py in compute(self, **kwargs)
164 dask.base.compute
165 """
@ian-whitestone
ian-whitestone / progress_bar.py
Created Jun 28, 2020
A simple progress bar that just shows the total number of "things" processed and the time elapsed
View progress_bar.py
"""
A simple progress bar that just shows the total number of "things" processed
and the time elapsed
"""
import time
from datetime import timedelta
from rich.progress import Progress, ProgressColumn, Task, TaskID, Text, TextColumn
@ian-whitestone
ian-whitestone / App.js
Created May 30, 2020
Learning react useState hooks by doing...
View App.js
// https://github.com/mui-org/material-ui/tree/master/examples/create-react-app
import React from 'react';
import Container from '@material-ui/core/Container';
import Typography from '@material-ui/core/Typography';
import Box from '@material-ui/core/Box';
import { makeStyles } from '@material-ui/core/styles';
import Link from '@material-ui/core/Link';
import SvgIcon from '@material-ui/core/SvgIcon';
import Button from '@material-ui/core/Button';
View redfin_custom_proxy.py
import pickle
PROXY_URL = 'https://z1h4spb3u7.execute-api.us-west-1.amazonaws.com/proxy_us_west_1'
def proxy_request(url):
proxy_response = requests.post(
PROXY_URL,
data={'url': url}
)
if not proxy_response.ok:
@ian-whitestone
ian-whitestone / run_docker.sh
Created Mar 16, 2020
Script for running executing python jobs in docker
View run_docker.sh
# User defined constants
HOME="/home/ianwhitestone/"
LOGS_DIR="logs"
MASTER_LOG_FILENAME="master.txt"
SECRETS_FILE="$HOME/.secrets"
# Input Arguments
MODULE=$1
FUNCTION=$2
@ian-whitestone
ian-whitestone / notify.py
Last active Mar 17, 2020
Script for sending failure notifications in slack
View notify.py
"""
Trigger slack notifications
"""
import argparse
import logging
import os
from slack.web.client import WebClient
LOGGER = logging.getLogger(__name__)
You can’t perform that action at this time.