modeling.ipynb

pipeline_recap.md

Data Science Pipeline

When working on a project, we probably go through the pipeline multiple times.

First pass is our MVP

1. Acquire: whatever SQL query gives us workable data
2. Prepare: drop nulls, data split
3. Explore: visualize the target against independent variables
4. Model: baseline, LinearRegression, LassoLars compare performance with rmse on validate

NB. not worried about scaling or automated feature engineering

Second pass:

• focus on modeling, lets scale our data in prep and then try out models on the scaled data

Third pass:

• let's look at null values more, impute instead of drop
• rerun models to see if performance changes

fourth pass

• let's do more exploration, lets visualize more variable interactions

....

Eventually

1. Acquire: fancy SQL query that joins and stuff
2. Prepare: handle nulls, handle outliers, scale data, split data
3. Explore: multiple visualizations of independent variable interactions as well as drivers of the target and statistical tests
4. Modeling: try out multiple different model types with different hyperparameters