ericmjl

UPDATE: I have baked the ideas in this file inside a Python CLI tool called pyds-cli. Please find it here: https://github.com/ericmjl/pyds-cli

How to organize your Python data science project

Having done a number of data projects over the years, and having seen a number of them up on GitHub, I've come to see that there's a wide range in terms of how "readable" a project is. I'd like to share some practices that I have come to adopt in my projects, which I hope will bring some organization to your projects.

Disclaimer: I'm hoping nobody takes this to be "the definitive guide" to organizing a data project; rather, I hope you, the reader, find useful tips that you can adapt to your own projects.

Disclaimer 2: What I’m writing below is primarily geared towards Python language users. Some ideas may be transferable to other languages; others may not be so. Please feel free to remix whatever you see here!

ericmjl

<script defer src="https://pyscript.net/unstable/pyscript.js"></script> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.1.3/dist/js/bootstrap.bundle.min.js" integrity="sha384-ka7Sk0Gln4gmtz2MlQnikT1wXgYsOg+OMhuP+IlRH9sENBO0LRn5q+8nbTov4+1p" crossorigin="anonymous"></script>

PyScript Pre-populated REPL

ericmjl

import datashader as ds
import holoviews as hv
from holoviews.operation.datashader import datashade
hv.extension('bokeh')

def scatter(dim1, dim2):
    def _scatter(data):
        return hv.Scatter(data, kdims=[dim1], vdims=[dim2], extents=(-10, -10, 10, 10))
    return _scatter

ericmjl

"""
Author: Eric J. Ma  

Purpose: To merge PDFs together in an automated fashion.
"""


import os

from PyPDF2 import PdfFileReader, PdfFileMerger

ericmjl

Data Science Programming September 2020 Newsletter

Hello fellow datanistas!

Welcome to the September edition of the programming-oriented data science newsletter. I hope you've all been staying safe amid the COVID-19 outbreak.

There's no special theme this month, just a smattering of cool tools and articles that I think will improve your productivity!

Setting up VSCode for Python Development like RStudio

ericmjl

ericmjl

# Taken from https://github.com/ericmjl/dotfiles/blob/master/install_functions.sh

function install_anaconda {
  bash anaconda.sh -b -p $HOME/anaconda
  rm anaconda.sh
  export PATH=$HOME/anaconda/bin:$PATH

  # Install basic data science stack into default environment
  conda install --yes pandas scipy numpy matplotlib seaborn jupyter ipykernel nodejs

ericmjl

ericmjl

$ docker run --gpus all -i -t jax:latest /bin/bash
(base) [docker@e697aef58065 ~]$ ls
anaconda  cuda-repo-rhel8-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm
(base) [docker@e697aef58065 ~]$ which python
~/anaconda/bin/python
(base) [docker@e697aef58065 ~]$ conda activate mouse-hmm
(mouse-hmm) [docker@e697aef58065 ~]$ python
Python 3.7.6 | packaged by conda-forge | (default, Jun  1 2020, 18:57:50) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

ericmjl

@pytest.fixture
def path_graph():
    """Return a path graaph of length three."""
    G = nx.path_graph(3, create_using=nx.DiGraph)
    G.graph["name"] = "path"
    nx.freeze(G)
    return G

@pytest.fixture