Skip to content

Instantly share code, notes, and snippets.

@lockefox
lockefox / python_logging_blog.md
Last active August 31, 2019 03:14
Everything You Never Wanted To Know About Python Logging (1)

Everything You Never Wanted To Know About Python Logging

print() can't remain the only tool for debugging/reporting in programs. Python offers an extremely powerful suite of reporting tools with the logging library. But be warned, logging isn't merely "print() on steroids", and mishandling this library can become a major pain point.

In this guide we will discuss the underlying model behind logging and how to get the most out of the reporting in our Python projects.

The Life of a Logging Message

TODO: grafix

@lockefox
lockefox / config_utils.py
Created November 19, 2018 23:19
Old code removed from ProsperCommon and pytest-prosper because it's weird and bad
'''utilities.py: worker functions for CREST calls'''
from os import path
import logging
from datetime import datetime
from prosper.common.prosper_config import get_config, get_local_config_filepath
DEFAULT_LOGGER = logging.getLogger('NULL')
DEFAULT_LOGGER.addHandler(logging.NullHandler())
@lockefox
lockefox / eve_online_map_experiments.ipynb
Created October 31, 2017 00:52
Searching for patterns in EVE Online map statistics
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@lockefox
lockefox / coin_demo_doge.ipynb
Created October 3, 2017 19:37
CryptoCompare history [DOGE] + fbprophet forecasting
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@lockefox
lockefox / stock_demo.ipynb
Created September 29, 2017 20:05
Pandas-Datareader + fbprophet forecasting
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@lockefox
lockefox / coin_demo.ipynb
Last active September 29, 2017 20:13
CryptoCompare history + fbprophet forecasting
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
data_columns = ['killID', 'solarSystemID', 'killTime', 'moonID']
pivot_column = 'items'
index_column = 'killID'
## Make source DataFrame ##
raw_df = pd.DataFrame(raw_data)
## Rotate list-column into its own frame ##
pivot_df = pd.DataFrame(list(raw_df[pivot_column]))
Data Scenario 200 records 2000 records 50000 records
Pivot Dicts - Pandas 0.015s 0.041s 0.804s
Pivot Dicts - Raw Python 0.028s 0.217s 5.781s
Pivot Lists - Pandas (concat) 0.834s 9.672 2069.694s
Pivot Lists - Pandas (melt) 0.029s 0.190s 17.454s
Pivot Lists - Raw Python 0.043s 0.213s 5.502s
data_columns = ['killID', 'solarSystemID', 'killTime', 'moonID']
pivot_column = 'items'
index_column = 'killID'
## Make source DataFrame ##
raw_df = pd.DataFrame(raw_data)
## Row-by-row rotate and append desired data ##
result_df = None
for row in raw_df.itertuples():
## Split out JSON-blob columns into their own tables ##
# Rotates out key/value pairs into their own columns
position_df = pd.DataFrame(list(kill_df['position']))
victim_df = pd.DataFrame(list(kill_df['victim']))
zkb_df = pd.DataFrame(list(kill_df['zkb']))
## Merge pivoted data back into source frame ##
# row-index has not changed, append-horizontally
combined_df = pd.concat([kill_df, position_df, victim_df, zkb_df], axis=1)