Skip to content

Instantly share code, notes, and snippets.

@austospumanto
austospumanto / gist:6857f9bdbc5b3c8fb2c3a1f0d1bdae12
Created Apr 30, 2021 — forked from christofluethi/gist:646ae60d797a46a706a5
Convert m4a to mp3 on OS X command line using ffmpeg
View gist:6857f9bdbc5b3c8fb2c3a1f0d1bdae12
brew update
brew link yasm
brew link x264
brew link lame
brew link xvid
brew install ffmpeg
ffmpeg wiki:
https://trac.ffmpeg.org/wiki/Encode/MP3
@austospumanto
austospumanto / pd_processit.py
Last active Aug 31, 2020
Pandas groupby-apply using processit
View pd_processit.py
"""
System/Runtime Requirements:
>=Python3.7
Linux / Mac
>=2 CPU Cores
Must pip install to use `pd_processit`:
pickle5, tqdm, numpy, pandas
To use this file, have processit.py in same folder as pd_processit.py (this file).
@austospumanto
austospumanto / profiling.py
Last active Nov 20, 2019
Memory & Time Profiling Decorators
View profiling.py
"""
################
# Installation #
################
memory_profiler:
pip install memory-profiler
pympler:
pip install pympler
@austospumanto
austospumanto / base.py
Last active Aug 28, 2019
"sqlalchemy/dialects/mssql/base.py" for `sqlacodegen` w/ Azure SQL Data Warehouse
View base.py
"""
Modified file (<VIRTUALENV>/lib/python3.7/site-packages/sqlalchemy/dialects/mssql/base.py)
for sqlalchemy==1.3.7 to get it to work with sqlacodegen==2.1.0
for an Azure SQL Data Warehouse server using the following get_engine function:
```
def get_engine(username=None, password=None, hostname=None, database=None) -> Engine:
from skykick_ds.runconfig import RunConfig
username = username or RunConfig.sql_username
@austospumanto
austospumanto / diskcached.py
Created Aug 3, 2019
Diskcached: Like functools.lru_cache, but pickled to disk
View diskcached.py
"""
from .diskcached import clear_all, diskcached
@diskcached()
def cached_fn(x):
print(x)
return x + 2
@austospumanto
austospumanto / timeout.py
Last active Jul 23, 2019
Timeout on a synchronous Python block using context manager
View timeout.py
"""
NOTE: THIS IS NOT MY CODE. I'M JUST SAVING IT HERE.
---------------------------------------------------
Easily put time restrictions on things
Note: Requires Python 3.x
Usage as a context manager:
```
with timeout(10):
something_that_should_not_exceed_ten_seconds()
@austospumanto
austospumanto / pdutils.py
Created Jul 23, 2019
Some Python Pandas Utility Functions (Serialization, Data Type Casting)
View pdutils.py
"""
pip install \
pandas \
sqlalchemy \
pyarrow
"""
import json
import os
import time
@austospumanto
austospumanto / correlation.py
Created Jul 23, 2019
Use Numba to Quickly Calculate Pearson and Cramer's Correlation Coefficients
View correlation.py
"""
pip install \
numpy \
pandas \
numba
"""
import time
from dataclasses import dataclass
from functools import wraps
@austospumanto
austospumanto / gbq_utils.py
Created Jul 23, 2019
Python Google BigQuery Utility Functions
View gbq_utils.py
"""
pip install \
bigquery-schema-generator \
google-api-python-client \
pandas-gbq \
pandas
"""
import logging
import os
View SharedMemoryManager.py
"""
System/Runtime Requirements:
Python3.7
Linux / Mac
Must pip install:
numpy, pandas
Must build from source:
shared_memory (https://github.com/SleepProgger/py_shared_memory)