Skip to content

Instantly share code, notes, and snippets.

View scottpham's full-sized avatar

Scott Pham scottpham

View GitHub Profile
@scottpham
scottpham / census.py
Last active March 8, 2021 18:37
How to download census data in Python
import censusdata
import pandas as pd
# https://towardsdatascience.com/accessing-census-data-with-python-3e2f2b56e20d
# https://jtleider.github.io/censusdata/
# search for the right table
sample = censusdata.search('acs1', 2019, 'concept', 'total population')
sample[0]
@scottpham
scottpham / apply_groupby.py
Created April 20, 2021 22:41
How to use apply with groupby in pandas
# grooop
grouped = with_region.groupby('level_comp_region')
# How many workers are in which
res = (
grouped
.apply( lambda grp: pd.Series({
"old_min": grp["old_min"].iloc[0],
"old_max": grp["old_max"].iloc[0],
"employees here": len(grp)
@scottpham
scottpham / stats.py
Last active May 24, 2021 22:15
How to hypothesis test in Python
import scipy.stats as stats
import numpy as np
import scipy.stats.distributions as dist
import statsmodels.api as sm
# calc by hand
va = prop_sub * (1 - prop_sub) # variance
se = np.sqrt(va * (1 / total_high + 1/total_mod)) # grouped standard error
@scottpham
scottpham / instructions.md
Created August 5, 2022 21:41
How to install image magick for pdf plumber

The Magic Wand Library doesn't work yet (as of 08/05/2022) with ImageMagick7. Download 6 via homebrew:

brew install imagemagick@6

It's not enough just to link. Find the executables by searching brew --prefix. It'll be in Cellar.

Grab the main directory of the whole package and write to zshrc

export MAGICK_HOME=/opt/homebrew/Cellar/imagemagick@6/6.9.12-60

@scottpham
scottpham / snippets.py
Created November 15, 2022 21:07
snippets for mining project
# helper for counting basic sums
def basic_summary(frame):
return frame.pipe( lambda f: pd.Series({
"cases": f["CASE NUMBER"].nunique(),
"companies": f["COMPANY NAME"].nunique(),
"mines": f["MINE_ID"].nunique(),
"permits": f["PERMIT_NUMBER"].nunique()
}))