zhiyzuo

zhiyzuo

A not-so-complete guide to macOS setup for MSCI 3250: Analyzing Data for Business Intelligence

---

A side note: if you are not familiar with terminal, you can find a lot of related resources by googling. I found a video and a web page that may be helpful:

• https://youtu.be/-Vl4rpZVA6I
• https://macpaw.com/how-to/use-terminal-on-mac

---

zhiyzuo

# Add Homebrew `/usr/local/bin` and User `~/bin` to the `$PATH`
#PATH=$HOME/bin:$PATH
export PATH=/usr/local/bin:$PATH

if [ -f $(brew --prefix)/etc/bash_completion ]; then
  . $(brew --prefix)/etc/bash_completion
fi

export PATH="/usr/local/opt/sqlite/bin:$PATH"
export PATH="/usr/local/opt/openssl/bin:$PATH"

zhiyzuo

import pandas as pd
import statsmodels.formula.api as smf
from statsmodels.stats.outliers_influence import variance_inflation_factor

def get_vif(data):
    d = {data.columns.values[i]: variance_inflation_factor(data.values, i)\
         for i in range(data.shape[1])}
    vif_series = pd.Series(d)
    return vif_series

zhiyzuo

import pandas as pd

def get_coef_table(lin_reg):
    ''' lin_reg is a fitted statsmodels regression model
    Return a dataframe containing coefficients, pvalues, and the confidence intervals
    '''
    err_series = lin_reg.params - lin_reg.conf_int()[0]
    coef_df = pd.DataFrame({'coef': lin_reg.params.values[1:],
                            'ci_err': err_series.values[1:],
                            'pvalue': lin_reg.pvalues.round(4).values[1:],

zhiyzuo

import numpy as np
import scipy as sp
import pandas as pd

import seaborn as sns
from matplotlib import pyplot as plt
%matplotlib inline

zhiyzuo

import numpy as np
from scipy import stats

def pearsonr_ci(x,y,alpha=0.05):
    ''' calculate Pearson correlation along with the confidence interval using scipy and numpy

    Parameters
    ----------
    x, y : iterable object such as a list or np.array
      Input for correlation calculation

zhiyzuo

import numpy as np
import scipy as sp

def jsd(p, q, base=np.e):
    '''
        Implementation of pairwise `jsd` based on  
        https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence
    '''
    ## convert to np.array
    p, q = np.asarray(p), np.asarray(q)

zhiyzuo

This gist will be a place for some interesting data source I found but have no time to collect/think about yet.

__author__ = "Zhiya Zuo"
__email__ = "zhiyazuo@gmail.com"

---

Toy/Experimental datasets

This section lists datasets to play with or experiment with (e.g., test/evaluate/validate/learn new algorithms.)

zhiyzuo

Installing Rattle on MacOS 10.11 (or above)

This article was moved to https://zhiyzuo.github.io/installation-rattle/ !