ericmjl

#!/bin/bash

# Script for installing tmux on systems where you don't have root access.
# tmux will be installed in $HOME/local/bin.
# It's assumed that wget and a C/C++ compiler are installed.

# exit on error
set -e

TMUX_VERSION=2.6

ericmjl

UPDATE: I have baked the ideas in this file inside a Python CLI tool called pyds-cli. Please find it here: https://github.com/ericmjl/pyds-cli

How to organize your Python data science project

Having done a number of data projects over the years, and having seen a number of them up on GitHub, I've come to see that there's a wide range in terms of how "readable" a project is. I'd like to share some practices that I have come to adopt in my projects, which I hope will bring some organization to your projects.

Disclaimer: I'm hoping nobody takes this to be "the definitive guide" to organizing a data project; rather, I hope you, the reader, find useful tips that you can adapt to your own projects.

Disclaimer 2: What I’m writing below is primarily geared towards Python language users. Some ideas may be transferable to other languages; others may not be so. Please feel free to remix whatever you see here!

ericmjl

import networkx as nx
import numpy as np

def generate_graph():
    num_nodes = np.random.randint(low=3, high=20)
    G = nx.erdos_renyi_graph(n=num_nodes, p=0.3)
    for n in G.nodes():
        value = np.random.randint(low=1, high=20)
        G.node[n]['value'] = value
    return G

ericmjl

from pathlib import Path
import os

# We will use the example of creating a .directory under home.

home = Path.home()
dirname = home / '.dir'

if not dirname.exists():
    os.mkdir(dirname)

ericmjl

from sklearn.metrics import explained_variance_score

def var_explained(preds, actual):
    """
    Implementation taken directly from the formula on this page:
    http://scikit-learn.org/stable/modules/model_evaluation.html#explained-variance-score
    """
    return 1 - ((preds - actual).var() / actual.var())

y_pred = np.array([3, -0.5, 2, 7])

ericmjl

import datashader as ds
import holoviews as hv
from holoviews.operation.datashader import datashade
hv.extension('bokeh')

def scatter(dim1, dim2):
    def _scatter(data):
        return hv.Scatter(data, kdims=[dim1], vdims=[dim2], extents=(-10, -10, 10, 10))
    return _scatter

ericmjl

ericmjl

name: dl-workshop
channels:
- defaults
- conda-forge
- ericmjl
dependencies:
- python=3.7
- jupyter
- jupyterlab
- conda

ericmjl

ericmjl