suriyadeepan/hyp_gen_data_science_pipeline.md

## hyp_gen_data_science_pipeline.md

      
    Raw
  

              hyp_gen_data_science_pipeline.md
            
          
Stage
Description


1
Hypothesis Generation
Study the business problem. Build a conceptual model by developing a deeper understanding of the problem and domain. Generate Hypotheses.


2
Data Collection
Go out in the wild and collect data based on the generated hypotheses.


3
Study the variables
Identify potential predictors using data visualization


4
Data Preparation
Clean the data. Fill in missing data points. Scale, normalize and transform data as necessary.


5
Bivariate/Multivariate Analysis
Test the hypotheses you've generated earlier. Choose predictors based on correlation with target.


6
Data Transformation
Perform non-linear transformations (log) on variables to fish out non-linear relationships with the target variable - log-linear, linear-log, log-log, etc.


7
Feature Engineering
Engineer new features guided by your data intuition


8
Model Evaluation
Choose a list of appropriate models and rank them by evaluating against the validation set


9
Hyperparameter Search
Find the optimal hyperparameters for the models. Re-evaluate and re-rank models


10
Ensembling
Stabilize the final model using ensemble methods like averaging, voting and stacking.


11
Model Explanation
Extract insights from the model by using visualization or explanatory tools like SHAP values.
	Stage	Description
1	Hypothesis Generation	Study the business problem. Build a conceptual model by developing a deeper understanding of the problem and domain. Generate Hypotheses.
2	Data Collection	Go out in the wild and collect data based on the generated hypotheses.
3	Study the variables	Identify potential predictors using data visualization
4	Data Preparation	Clean the data. Fill in missing data points. Scale, normalize and transform data as necessary.
5	Bivariate/Multivariate Analysis	Test the hypotheses you've generated earlier. Choose predictors based on correlation with target.
6	Data Transformation	Perform non-linear transformations (log) on variables to fish out non-linear relationships with the target variable - log-linear, linear-log, log-log, etc.
7	Feature Engineering	Engineer new features guided by your data intuition
8	Model Evaluation	Choose a list of appropriate models and rank them by evaluating against the validation set
9	Hyperparameter Search	Find the optimal hyperparameters for the models. Re-evaluate and re-rank models
10	Ensembling	Stabilize the final model using ensemble methods like averaging, voting and stacking.
11	Model Explanation	Extract insights from the model by using visualization or explanatory tools like SHAP values.