print() can't remain the only tool for debugging/reporting in programs. Python offers an extremely powerful suite of reporting tools with the logging library. But be warned, logging isn't merely "print() on steroids", and mishandling this library can become a major pain point.
In this guide we will discuss the underlying model behind logging and how to get the most out of the reporting in our Python projects.
TODO: grafix
| '''utilities.py: worker functions for CREST calls''' | |
| from os import path | |
| import logging | |
| from datetime import datetime | |
| from prosper.common.prosper_config import get_config, get_local_config_filepath | |
| DEFAULT_LOGGER = logging.getLogger('NULL') | |
| DEFAULT_LOGGER.addHandler(logging.NullHandler()) |
| data_columns = ['killID', 'solarSystemID', 'killTime', 'moonID'] | |
| pivot_column = 'items' | |
| index_column = 'killID' | |
| ## Make source DataFrame ## | |
| raw_df = pd.DataFrame(raw_data) | |
| ## Rotate list-column into its own frame ## | |
| pivot_df = pd.DataFrame(list(raw_df[pivot_column])) |
| Data Scenario | 200 records | 2000 records | 50000 records | |
|---|---|---|---|---|
| Pivot Dicts - Pandas | 0.015s | 0.041s | 0.804s | |
| Pivot Dicts - Raw Python | 0.028s | 0.217s | 5.781s | |
| Pivot Lists - Pandas (concat) | 0.834s | 9.672 | 2069.694s | |
| Pivot Lists - Pandas (melt) | 0.029s | 0.190s | 17.454s | |
| Pivot Lists - Raw Python | 0.043s | 0.213s | 5.502s |
| data_columns = ['killID', 'solarSystemID', 'killTime', 'moonID'] | |
| pivot_column = 'items' | |
| index_column = 'killID' | |
| ## Make source DataFrame ## | |
| raw_df = pd.DataFrame(raw_data) | |
| ## Row-by-row rotate and append desired data ## | |
| result_df = None | |
| for row in raw_df.itertuples(): |
| ## Split out JSON-blob columns into their own tables ## | |
| # Rotates out key/value pairs into their own columns | |
| position_df = pd.DataFrame(list(kill_df['position'])) | |
| victim_df = pd.DataFrame(list(kill_df['victim'])) | |
| zkb_df = pd.DataFrame(list(kill_df['zkb'])) | |
| ## Merge pivoted data back into source frame ## | |
| # row-index has not changed, append-horizontally | |
| combined_df = pd.concat([kill_df, position_df, victim_df, zkb_df], axis=1) |