DGrady

Query Oracle databases with Python and SQLAlchemy

N.B. SQLAlchemy now incorporates all of this information in its documentation; I’m leaving this post here, but recommend referring to SQLAlchemy instead of these instructions.

Install requirements

1. We’ll assume you already have SQLAlchemy and Pandas installed; these are included by default in many Python distributions.
2. Install the cx_Oracle package in your Python environment, using either pip or conda, for example:

DGrady

DGrady

"""
Problem: provide two-way communication with a subprocess in Python.

See also: 
- https://kevinmccarthy.org/2016/07/25/streaming-subprocess-stdin-and-stdout-with-asyncio-in-python/
- http://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/
"""

import asyncio
import sys

DGrady

import numpy as np
import pandas as pd


def frequency_histogram(
    data: pd.DataFrame,
    n_bins=20,
    bins=None,
    log_bins=False,
    normalize=False,

DGrady

# To reproduce a random sample, we need a fixed seed.

"{:_}".format(np.random.randint(np.iinfo(np.uint32).max))

DGrady

"""
The schemas that Spark produces for DataFrames are typically
nested, and these nested schemas are quite difficult to work with
interactively. In many cases, it's possible to flatten a schema
into a single level of column names.
"""

import typing as T

import cytoolz.curried as tz

DGrady

DGrady

Pretty printing delimited text files at the command line

Sometimes, you’d like to look at delimited files on the command line:

cat test.csv

DGrady

// Automatically hide the toolbar and header in Jupyter Notebook 4.1.0
// This should go in ~/.jupyter/custom/custom.js
require(
    ['base/js/namespace', 'base/js/events'],
    function(Jupyter, events) {
        events.on("notebook_loaded.Notebook", function () {
            Jupyter.toolbar.actions.call('jupyter-notebook:toggle-toolbar')
            Jupyter.toolbar.actions.call('jupyter-notebook:toggle-header')
        })
    }

DGrady

import pandas as pd

def describe_population(df: pd.DataFrame) -> pd.DataFrame:
    """
    Report the populated and uniqueness counts for each column of the input.
    
    The ratio columns are given as percents.
    """

    N = len(df)