Udacity Data Analyst Nanodegree notes

udacity-data-analyst-notes.md

Pandas

Import Pandas

import pandas as pd

Read CSV into dataframe

df = pd.read_csv('filename.csv')

Get dataframe shape - rows/columns

df.shape

Get dataframe size, including NaNs

df.size

Get dataframe size, excluding NaNs

df.count

Get dataframe summary

df.info()

Get dataframe statistics

df.describe()

Parse column as datetime

df['column_name'] = pd.to_datetime(df['column_name'])

Get unique column values

df['column_name'].unique()

Get column duplicates

pd.concat(group for _, group in df.groupby("column_name") if len(group) > 1)

Count dataframe duplicate rows

df[df.duplicated()].shape[0]
df.duplicated().sum()

Count dataframe rows with missing data

df.shape[0] - df.dropna().shape[0]

Count non-null unique values

df['column_name'].dropna().unique().size

Drop dataframe columns

df.drop('column_name', axis=1, inplace=True)

Group by two colums then again by one

df_18.groupby(['col1', 'col2']).size().groupby('col2').max()

Rename columns using Lambda function

df_08.rename(columns=lambda x: x + "_2008", inplace=True)

Bootstrap sampling for mean distribution

means = []
for _ in range(10000):
    bootstrap = FULL_DATA.sample(SAMPLE_SIZE, replace = True)
    bootstrap_mean = bootstrap[bootstrap['QUERY_PARAM'] == 'QUERY_VALUE']['COLUMN'].mean()
    means.append(bootstrap_mean)

Confidence interval of 95%

np.percentile(sample, 2.5), np.percentile(sample, 97.5)

Ordinary least squared - estimating unknown parameters in linear regression model

import statsmodels.api as sm;
df['intercept'] = 1
linear_model = sm.OLS(df['DEPENDENT_COLUMN'], df[['intercept', 'COLUMN_A', 'COLUMN_B', ...]])
regression_results = linear_model.fit()
regression_results.summary()

Adding dummy variables to a linear model

dummies = pd.get_dummies(df['CATEGORICAL_COLUMN'])
df_with_dummies = df.join(dummies)
model = sm.OLS(df_with_dummies['DEPENDENT_COLUMN'], df_with_dummies[['intercept', 'COLUMN_A', 'COLUMN_B', ...]])
regression = model.fit()
regression.summary()

Seaborn Pairplot

import seaborn as sns
sns.pairplot(df[['COLUMN_A', 'COLUMN_B', 'COLUMN_C']]);

Variance Inflation Factor (VIF)

from patsy import dmatrices
y, X = dmatrices('price ~ area + bedrooms + bathrooms', df, return_type = 'dataframe')
vif = pd.DataFrame()
vif["VIF Factor"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
vif["features"] = X.columns
vif