Skip to content

Instantly share code, notes, and snippets.

@pb111

pb111/XGBoost with Python and Scikit-Learn.ipynb

Created July 8, 2019 07:03
Show Gist options
  • Save pb111/cc341409081dffa5e9eaf60d79562a03 to your computer and use it in GitHub Desktop.
Save pb111/cc341409081dffa5e9eaf60d79562a03 to your computer and use it in GitHub Desktop.
Download ZIP
XGBoost with Python and Scikit-Learn
Display the source blob
Display the rendered blob
Raw
XGBoost with Python and Scikit-Learn.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# XGBoost with Python and Scikit-Learn \n",
"\n",
"\n",
"**XGBoost** is an acronym for **Extreme Gradient Boosting**. It is a powerful machine learning algorithm that can be used to solve classification and regression problems. In this project, I implement XGBoost with Python and Scikit-Learn to solve a classification problem. The problem is to classify the customers from two different channels as Horeca (Hotel/Retail/Café) customers or Retail channel (nominal) customers.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Table of Contents\n",
"\n",
"\n",
"\n",
"1.\tIntroduction to XGBoost algorithm\n",
"2.\tXGBoost algorithm intuition\n",
"3.\tThe problem statement\n",
"4.\tDataset description\n",
"5.\tImport libraries\n",
"6.\tImport dataset\n",
"7.\tExploratory data analysis\n",
"8.\tDeclare feature vector and target variable\n",
"9.\tSplit data into separate training and test set\n",
"10.\tTrain the XGBoost classifier\n",
"11.\tMake predictions with XGBoost classifier\n",
"12.\tCheck accuracy score\n",
"13.\tk-fold Cross Validation using XGBoost\n",
"14.\tFeature importance with XGBoost\n",
"15.\tResults and conclusion\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Introduction to XGBoost algorithm\n",
"\n",
"\n",
"**XGBoost** stands for **Extreme Gradient Boosting**. XGBoost is a powerful machine learning algorithm that is dominating the world of applied machine learning and Kaggle competitions. It is an implementation of gradient boosted trees designed for speed and accuracy.\n",
"\n",
"\n",
"**XGBoost (Extreme Gradient Boosting)** is an advanced implementation of the gradient boosting algorithm. It has proved to be a highly effective machine learning algorithm extensively used in machine learning competitions. XGBoost has high predictive power and is almost 10 times faster than other gradient boosting techniques. It also includes a variety of regularization parameters which reduces overfitting and improves overall performance. Hence, it is also known as **regularized boosting** technique.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. XGBoost algorithm intuition\n",
"\n",
"\n",
"XGBoost (Extreme Gradient Boosting) belongs to a family of boosting algorithms. It uses the gradient boosting (GBM) framework at its core. So, first of all we should know about gradient boosting.\n",
"\n",
"\n",
"### Gradient boosting\n",
"\n",
"Gradient boosting is a supervised machine learning algorithm, which tries to predict a target variable by combining the estimates of a set of simpler, weaker models. In boosting, the trees are built in a sequential manner such that each subsequent tree aims to reduce the errors of the previous tree. The misclassified labels are given higher weights. Each tree learns from its predecessors and tries to reduce the residual errors. So, the tree next in sequence will learn from the previous tree residuals.\n",
"\n",
"\n",
"### XGBoost\n",
"\n",
"In XGBoost, we try to fit a model on the gradient of the loss function generated from the previous step. So, in XGBoost we modified our gradient boosting algorithm so that it works with any differentiable loss function.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. The problem statement\n",
"\n",
"In this project, I try to solve a classification problem. The problem is to classify the customers from two different channels as Horeca (Hotel/Retail/Café) customers or Retail channel (nominal) customers. I implement XGBoost with Python and Scikit-Learn to solve the classification problem. \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Dataset description\n",
"\n",
"\n",
"I have used the `Wholesale customers data set` for this project, downloaded from the UCI Machine learning repository. \n",
"This dataset can be found at the following url-\n",
"\n",
"\n",
"https://archive.ics.uci.edu/ml/datasets/Wholesale+customers\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import warnings\n",
"\n",
"warnings.filterwarnings('ignore')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Import dataset"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Import dataset\n",
"\n",
"data = 'C:/datasets/Wholesale customers data.csv'\n",
"\n",
"df = pd.read_csv(data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Exploratory Data Analysis\n",
"\n",
"\n",
"I will start off by checking the shape of the dataset."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(440, 8)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are 440 instances and 8 attributes in the dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Preview dataset"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Channel</th>\n",
" <th>Region</th>\n",
" <th>Fresh</th>\n",
" <th>Milk</th>\n",
" <th>Grocery</th>\n",
" <th>Frozen</th>\n",
" <th>Detergents_Paper</th>\n",
" <th>Delicassen</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>12669</td>\n",
" <td>9656</td>\n",
" <td>7561</td>\n",
" <td>214</td>\n",
" <td>2674</td>\n",
" <td>1338</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>7057</td>\n",
" <td>9810</td>\n",
" <td>9568</td>\n",
" <td>1762</td>\n",
" <td>3293</td>\n",
" <td>1776</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>6353</td>\n",
" <td>8808</td>\n",
" <td>7684</td>\n",
" <td>2405</td>\n",
" <td>3516</td>\n",
" <td>7844</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" <td>3</td>\n",
" <td>13265</td>\n",
" <td>1196</td>\n",
" <td>4221</td>\n",
" <td>6404</td>\n",
" <td>507</td>\n",
" <td>1788</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>22615</td>\n",
" <td>5410</td>\n",
" <td>7198</td>\n",
" <td>3915</td>\n",
" <td>1777</td>\n",
" <td>5185</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Channel Region Fresh Milk Grocery Frozen Detergents_Paper Delicassen\n",
"0 2 3 12669 9656 7561 214 2674 1338\n",
"1 2 3 7057 9810 9568 1762 3293 1776\n",
"2 2 3 6353 8808 7684 2405 3516 7844\n",
"3 1 3 13265 1196 4221 6404 507 1788\n",
"4 2 3 22615 5410 7198 3915 1777 5185"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that `Channel` variable contains values as `1` and `2`. These two values classify the customers from two different channels as 1 for Horeca (Hotel/Retail/Café) customers and 2 for Retail channel (nominal) customers."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### View summary of dataframe"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 440 entries, 0 to 439\n",
"Data columns (total 8 columns):\n",
"Channel 440 non-null int64\n",
"Region 440 non-null int64\n",
"Fresh 440 non-null int64\n",
"Milk 440 non-null int64\n",
"Grocery 440 non-null int64\n",
"Frozen 440 non-null int64\n",
"Detergents_Paper 440 non-null int64\n",
"Delicassen 440 non-null int64\n",
"dtypes: int64(8)\n",
"memory usage: 27.6 KB\n"
]
}
],
"source": [
"df.info()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are only numerical variables in the dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### View summary statistics of dataframe"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Channel</th>\n",
" <th>Region</th>\n",
" <th>Fresh</th>\n",
" <th>Milk</th>\n",
" <th>Grocery</th>\n",
" <th>Frozen</th>\n",
" <th>Detergents_Paper</th>\n",
" <th>Delicassen</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>440.000000</td>\n",
" <td>440.000000</td>\n",
" <td>440.000000</td>\n",
" <td>440.000000</td>\n",
" <td>440.000000</td>\n",
" <td>440.000000</td>\n",
" <td>440.000000</td>\n",
" <td>440.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>1.322727</td>\n",
" <td>2.543182</td>\n",
" <td>12000.297727</td>\n",
" <td>5796.265909</td>\n",
" <td>7951.277273</td>\n",
" <td>3071.931818</td>\n",
" <td>2881.493182</td>\n",
" <td>1524.870455</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>0.468052</td>\n",
" <td>0.774272</td>\n",
" <td>12647.328865</td>\n",
" <td>7380.377175</td>\n",
" <td>9503.162829</td>\n",
" <td>4854.673333</td>\n",
" <td>4767.854448</td>\n",
" <td>2820.105937</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>1.000000</td>\n",
" <td>1.000000</td>\n",
" <td>3.000000</td>\n",
" <td>55.000000</td>\n",
" <td>3.000000</td>\n",
" <td>25.000000</td>\n",
" <td>3.000000</td>\n",
" <td>3.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>1.000000</td>\n",
" <td>2.000000</td>\n",
" <td>3127.750000</td>\n",
" <td>1533.000000</td>\n",
" <td>2153.000000</td>\n",
" <td>742.250000</td>\n",
" <td>256.750000</td>\n",
" <td>408.250000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>1.000000</td>\n",
" <td>3.000000</td>\n",
" <td>8504.000000</td>\n",
" <td>3627.000000</td>\n",
" <td>4755.500000</td>\n",
" <td>1526.000000</td>\n",
" <td>816.500000</td>\n",
" <td>965.500000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>2.000000</td>\n",
" <td>3.000000</td>\n",
" <td>16933.750000</td>\n",
" <td>7190.250000</td>\n",
" <td>10655.750000</td>\n",
" <td>3554.250000</td>\n",
" <td>3922.000000</td>\n",
" <td>1820.250000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>2.000000</td>\n",
" <td>3.000000</td>\n",
" <td>112151.000000</td>\n",
" <td>73498.000000</td>\n",
" <td>92780.000000</td>\n",
" <td>60869.000000</td>\n",
" <td>40827.000000</td>\n",
" <td>47943.000000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Channel Region Fresh Milk Grocery \\\n",
"count 440.000000 440.000000 440.000000 440.000000 440.000000 \n",
"mean 1.322727 2.543182 12000.297727 5796.265909 7951.277273 \n",
"std 0.468052 0.774272 12647.328865 7380.377175 9503.162829 \n",
"min 1.000000 1.000000 3.000000 55.000000 3.000000 \n",
"25% 1.000000 2.000000 3127.750000 1533.000000 2153.000000 \n",
"50% 1.000000 3.000000 8504.000000 3627.000000 4755.500000 \n",
"75% 2.000000 3.000000 16933.750000 7190.250000 10655.750000 \n",
"max 2.000000 3.000000 112151.000000 73498.000000 92780.000000 \n",
"\n",
" Frozen Detergents_Paper Delicassen \n",
"count 440.000000 440.000000 440.000000 \n",
"mean 3071.931818 2881.493182 1524.870455 \n",
"std 4854.673333 4767.854448 2820.105937 \n",
"min 25.000000 3.000000 3.000000 \n",
"25% 742.250000 256.750000 408.250000 \n",
"50% 1526.000000 816.500000 965.500000 \n",
"75% 3554.250000 3922.000000 1820.250000 \n",
"max 60869.000000 40827.000000 47943.000000 "
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Check for missing values"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Channel 0\n",
"Region 0\n",
"Fresh 0\n",
"Milk 0\n",
"Grocery 0\n",
"Frozen 0\n",
"Detergents_Paper 0\n",
"Delicassen 0\n",
"dtype: int64"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are no missing values in the dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. Declare feature vector and target variable"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"X = df.drop('Channel', axis=1)\n",
"\n",
"y = df['Channel']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### let's take a look at feature vector(X) and target variable(y)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Region</th>\n",
" <th>Fresh</th>\n",
" <th>Milk</th>\n",
" <th>Grocery</th>\n",
" <th>Frozen</th>\n",
" <th>Detergents_Paper</th>\n",
" <th>Delicassen</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>3</td>\n",
" <td>12669</td>\n",
" <td>9656</td>\n",
" <td>7561</td>\n",
" <td>214</td>\n",
" <td>2674</td>\n",
" <td>1338</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" <td>7057</td>\n",
" <td>9810</td>\n",
" <td>9568</td>\n",
" <td>1762</td>\n",
" <td>3293</td>\n",
" <td>1776</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>3</td>\n",
" <td>6353</td>\n",
" <td>8808</td>\n",
" <td>7684</td>\n",
" <td>2405</td>\n",
" <td>3516</td>\n",
" <td>7844</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>3</td>\n",
" <td>13265</td>\n",
" <td>1196</td>\n",
" <td>4221</td>\n",
" <td>6404</td>\n",
" <td>507</td>\n",
" <td>1788</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>3</td>\n",
" <td>22615</td>\n",
" <td>5410</td>\n",
" <td>7198</td>\n",
" <td>3915</td>\n",
" <td>1777</td>\n",
" <td>5185</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Region Fresh Milk Grocery Frozen Detergents_Paper Delicassen\n",
"0 3 12669 9656 7561 214 2674 1338\n",
"1 3 7057 9810 9568 1762 3293 1776\n",
"2 3 6353 8808 7684 2405 3516 7844\n",
"3 3 13265 1196 4221 6404 507 1788\n",
"4 3 22615 5410 7198 3915 1777 5185"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X.head()"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 2\n",
"1 2\n",
"2 2\n",
"3 1\n",
"4 2\n",
"Name: Channel, dtype: int64"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the y labels contain values as 1 and 2. I will need to convert it into 0 and 1 for further analysis. I will do it as follows-"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"# convert labels into binary values\n",
"\n",
"y[y == 2] = 0\n",
"\n",
"y[y == 1] = 1"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 0\n",
"1 0\n",
"2 0\n",
"3 1\n",
"4 0\n",
"Name: Channel, dtype: int64"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# again preview the y label\n",
"\n",
"y.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, I will convert the dataset into an optimized data structure called **Dmatrix** that XGBoost supports and gives it acclaimed performance and efficiency gains. I will do it as follows."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"# import XGBoost\n",
"import xgboost as xgb\n",
"\n",
"\n",
"# define data_dmatrix\n",
"data_dmatrix = xgb.DMatrix(data=X,label=y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 9. Split data into separate training and test set"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"# split X and y into training and testing sets\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 10. Train the XGBoost classifier\n",
"\n",
"\n",
"- Now, I will train the XGBoost classifier. We need to know different parameters that XGBoost provides. There are three types of parameters that we must set before running XGBoost. These parameters are as follows:-\n",
"\n",
"\n",
"### General parameters\n",
"\n",
"These parameters relate to which booster we are doing boosting. The common ones are tree or linear model.\n",
"\n",
"\n",
"### Booster parameters\n",
"\n",
"It depends on which booster we have chosen for boosting.\n",
"\n",
"\n",
"### Learning task parameters\n",
"\n",
"These parameters decide on the learning scenario. For example, regression tasks may use different parameters than ranking tasks. \n",
"\n",
"\n",
"### Command line parameters\n",
"\n",
"In addition there are command line parameters which relate to behaviour of CLI version of XGBoost.\n",
"\n",
"\n",
"The most important parameters that we should know about are as follows:-\n",
"\n",
"\n",
"**learning_rate** - It gives us the step size shrinkage which is used to prevent overfitting. Its range is [0,1].\n",
"\n",
"**max_depth** - It determines how deeply each tree is allowed to grow during any boosting round.\n",
"\n",
"**subsample** - It determines the percentage of samples used per tree. Low value of subsample can lead to underfitting.\n",
"\n",
"**colsample_bytree** - It determines the percentage of features used per tree. High value of it can lead to overfitting.\n",
"\n",
"**n_estimators** - It is the number of trees we want to build.\n",
"\n",
"**objective** - It determines the loss function to be used in the process. For example, `reg:linear` for regression problems, `reg:logistic` for classification problems with only decision, `binary:logistic` for classification problems with probability.\n",
"\n",
"\n",
"XGBoost also supports regularization parameters to penalize models as they become more complex and reduce them to simple models. These regularization parameters are as follows:-\n",
"\n",
"\n",
"**gamma** - It controls whether a given node will split based on the expected reduction in loss after the split. A higher value leads to fewer splits. It is supported only for tree-based learners.\n",
"\n",
"**alpha** - It gives us the `L1` regularization on leaf weights. A large value of it leads to more regularization.\n",
"\n",
"**lambda** - It gives us the `L2` regularization on leaf weights and is smoother than `L1` regularization.\n",
"\n",
"Though we are using trees as our base learners, we can also use XGBoost’s relatively less popular linear base learners and one other tree learner known as `dart`. We have to set the `booster` parameter to either `gbtree` (default), `gblinear` or `dart`.\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"XGBClassifier(alpha=10, base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
" colsample_bynode=1, colsample_bytree=1, gamma=0, learning_rate=1.0,\n",
" max_delta_step=0, max_depth=4, min_child_weight=1, missing=None,\n",
" n_estimators=100, n_jobs=1, nthread=None,\n",
" objective='binary:logistic', random_state=0, reg_alpha=0,\n",
" reg_lambda=1, scale_pos_weight=1, seed=None, silent=None,\n",
" subsample=1, verbosity=1)"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# import XGBClassifier\n",
"from xgboost import XGBClassifier\n",
"\n",
"\n",
"# declare parameters\n",
"params = {\n",
" 'objective':'binary:logistic',\n",
" 'max_depth': 4,\n",
" 'alpha': 10,\n",
" 'learning_rate': 1.0,\n",
" 'n_estimators':100\n",
" }\n",
" \n",
" \n",
" \n",
"# instantiate the classifier \n",
"xgb_clf = XGBClassifier(**params)\n",
"\n",
"\n",
"\n",
"# fit the classifier to the training data\n",
"xgb_clf.fit(X_train, y_train)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"XGBClassifier(alpha=10, base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
" colsample_bynode=1, colsample_bytree=1, gamma=0, learning_rate=1.0,\n",
" max_delta_step=0, max_depth=4, min_child_weight=1, missing=None,\n",
" n_estimators=100, n_jobs=1, nthread=None,\n",
" objective='binary:logistic', random_state=0, reg_alpha=0,\n",
" reg_lambda=1, scale_pos_weight=1, seed=None, silent=None,\n",
" subsample=1, verbosity=1)\n"
]
}
],
"source": [
"# alternatively view the parameters of the xgb trained model\n",
"print(xgb_clf)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 11. Make predictions with XGBoost Classifier"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"# make predictions on test data\n",
"y_pred = xgb_clf.predict(X_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 12. Check accuracy score"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"XGBoost model accuracy score: 0.9167\n"
]
}
],
"source": [
"# check accuracy score\n",
"from sklearn.metrics import accuracy_score\n",
"\n",
"print('XGBoost model accuracy score: {0:0.4f}'. format(accuracy_score(y_test, y_pred)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that XGBoost obtain very high accuracy score of 91.67%."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 13. k-fold Cross Validation using XGBoost\n",
"\n",
"\n",
"To build more robust models with XGBoost, we must do k-fold cross validation. In this way, we ensure that the original training dataset is used for both training and validation. Also, each entry is used for validation just once. XGBoost supports k-fold cross validation using the `cv()` method. In this method, we will specify several parameters which are as follows:- \n",
"\n",
"\n",
"**nfolds** - This parameter specifies the number of cross-validation sets we want to build. \n",
"\n",
"**num_boost_round** - It denotes the number of trees we build.\n",
"\n",
"**metrics** - It is the performance evaluation metrics to be considered during CV.\n",
"\n",
"**as_pandas** - It is used to return the results in a pandas DataFrame.\n",
"\n",
"**early_stopping_rounds** - This parameter stops training of the model early if the hold-out metric does not improve for a given number of rounds.\n",
"\n",
"**seed** - This parameter is used for reproducibility of results.\n",
"\n",
"We can use these parameters to build a k-fold cross-validation model by calling `XGBoost's CV()` method.\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"from xgboost import cv\n",
"\n",
"params = {\"objective\":\"binary:logistic\",'colsample_bytree': 0.3,'learning_rate': 0.1,\n",
" 'max_depth': 5, 'alpha': 10}\n",
"\n",
"xgb_cv = cv(dtrain=data_dmatrix, params=params, nfold=3,\n",
" num_boost_round=50, early_stopping_rounds=10, metrics=\"auc\", as_pandas=True, seed=123)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`xgb_cv` contains train and test `auc` metrics for each boosting round. Let's preview `xgb_cv`."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>train-auc-mean</th>\n",
" <th>train-auc-std</th>\n",
" <th>test-auc-mean</th>\n",
" <th>test-auc-std</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0.914998</td>\n",
" <td>0.009704</td>\n",
" <td>0.880965</td>\n",
" <td>0.021050</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0.934374</td>\n",
" <td>0.013263</td>\n",
" <td>0.923561</td>\n",
" <td>0.022810</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0.936252</td>\n",
" <td>0.013723</td>\n",
" <td>0.924433</td>\n",
" <td>0.025777</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0.943878</td>\n",
" <td>0.009032</td>\n",
" <td>0.927152</td>\n",
" <td>0.022228</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>0.957880</td>\n",
" <td>0.008845</td>\n",
" <td>0.935191</td>\n",
" <td>0.016437</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" train-auc-mean train-auc-std test-auc-mean test-auc-std\n",
"0 0.914998 0.009704 0.880965 0.021050\n",
"1 0.934374 0.013263 0.923561 0.022810\n",
"2 0.936252 0.013723 0.924433 0.025777\n",
"3 0.943878 0.009032 0.927152 0.022228\n",
"4 0.957880 0.008845 0.935191 0.016437"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"xgb_cv.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 14. Feature importance with XGBoost\n",
"\n",
"\n",
"XGBoost provides a way to examine the importance of each feature in the original dataset within the model. It involves counting the number of times each feature is split on across all boosting trees in the model. Then we visualize the result as a bar graph, with the features ordered according to how many times they appear. \n",
"\n",
"XGBoost has a **plot_importance()** function that helps us to achieve this task. Then we can visualize the features that has been given the highest important score among all the features. Thus XGBoost provides us a way to do feature selection.\n",
"\n",
"I will proceed as follows:-\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAc8AAAEWCAYAAAAASRzMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XucVXW9//HXm4uIjkKGmFpKHExMLmMa5Mls1PB4wbxk2hFP4CXUk3r0eOPYAaWjP00xtexYSAmBmpfKGx3xglvNIwnIzcpbOh5JElIRhwac2Xx+f+w1tB1nmL2G2bNnb97Px2M/Zq3v+q61Pp+Zgc98v2vtvRQRmJmZWeG6lToAMzOzcuPiaWZmlpKLp5mZWUounmZmZim5eJqZmaXk4mlmZpaSi6eZdShJP5Y0sdRxmBWT/D5Ps65BUi2wE5DNa/5MRLy5GcesAWZFxCc3L7ryJGk6sDwi/rPUsVhl8cjTrGs5KiKq8l7tLpwdQVKPUp5/c0jqXuoYrHK5eJqVAUlfkPS/klZLWpKMKJu2nSLpj5Lel/SqpDOS9m2B/wF2kVSXvHaRNF3SFXn710hanrdeK+kSSUuBtZJ6JPv9UtIqSa9JOncTsW48ftOxJV0saaWkFZKOkXSEpJckvSPp0rx9L5d0j6Q7k3yekzQ8b/tekjLJ9+H3kr7a7Lw3S/qNpLXAacAY4OIk9weSfhMk/Sk5/h8kHZt3jHGSfitpiqR3k1wPz9u+g6RbJb2ZbL83b9toSYuT2P5X0rCCf8BWdlw8zbo4SbsCs4ErgB2AC4FfStox6bISGA1sD5wCXC/pcxGxFjgceLMdI9l/Bo4E+gIbgAeAJcCuwCHAeZL+qcBjfQLYOtl3EnALcDKwL/AlYJKkgXn9jwbuTnK9HbhXUk9JPZM4Hgb6A+cAt0naM2/fk4Arge2AnwO3AdckuR+V9PlTct4+wGRglqSd844xEngR6AdcA/xUkpJtM4FtgL2TGK4HkPQ54GfAGcDHgZ8A90vqVeD3yMqMi6dZ13JvMnJZnTeqORn4TUT8JiI2RMQjwALgCICImB0Rf4qcJ8gVly9tZhw/iIg3IqIe+DywY0R8NyI+iIhXyRXAbxR4rAbgyohoAH5BrijdGBHvR8Tvgd8D+aO0hRFxT9L/++QK7xeSVxVwdRLHXOBBcoW+yX0R8XTyfVrXUjARcXdEvJn0uRN4GRiR1+X1iLglIrLADGBnYKekwB4OnBkR70ZEQ/L9BvgW8JOI+F1EZCNiBrA+idkqUNlezzCrUMdExKPN2nYHvi7pqLy2nsDjAMm04mXAZ8j9QbwNsGwz43ij2fl3kbQ6r6078FSBx3o7KUQA9cnXt/K215Mrih85d0RsSKaUd2naFhEb8vq+Tm5E21LcLZL0TeDfgQFJUxW5gt7kL3nn/1sy6KwiNxJ+JyLebeGwuwNjJZ2T17ZVXtxWYVw8zbq+N4CZEfGt5huSacFfAt8kN+pqSEasTdOMLd1Ov5ZcgW3yiRb65O/3BvBaROzRnuDb4VNNC5K6AZ8EmqabPyWpW14B3Q14KW/f5vl+aF3S7uRGzYcAz0REVtJi/v792pQ3gB0k9Y2I1S1suzIirizgOFYBPG1r1vXNAo6S9E+SukvaOrkR55PkRje9gFVAYzIKPTRv37eAj0vqk9e2GDgiufnlE8B5bZz/WWBNchNR7ySGIZI+32EZfti+ko5L7vQ9j9z05zzgd+QK/8XJNdAa4ChyU8GteQvIv566LbmCugpyN1sBQwoJKiJWkLsB678lfSyJ4cBk8y3AmZJGKmdbSUdK2q7AnK3MuHiadXER8Qa5m2guJfef/hvARUC3iHgfOBe4C3iX3A0z9+ft+wJwB/Bqch11F3I3vSwBasldH72zjfNnyRWpauA14K/ANHI33BTDfcCJ5PL5F+C45PriB8BXyV13/Cvw38A3kxxb81Pgs03XkCPiD8B1wDPkCutQ4OkUsf0LuWu4L5C7Ues8gIhYQO66501J3K8A41Ic18qMPyTBzLoMSZcDgyLi5FLHYrYpHnmamZml5OJpZmaWkqdtzczMUvLI08zMLCW/z7NC9e3bNwYNGlTqMDrc2rVr2XbbbUsdRlFUam7Oq/xUam6F5LVw4cK/RsSOm+yEi2fF2mmnnViwYEGpw+hwmUyGmpqaUodRFJWam/MqP5WaWyF5SXq9kGN52tbMzCwlF08zM7OUXDzNzMxScvE0MzNLycXTzMwsJRdPMzOzlFw8zczMUnLxNDMzS8nF08zMLCUXTzMzs5RcPM3MzFJy8TQzM0vJxdPMzCwlF08zM7OUXDzNzMxScvE0MzNLycXTzMwsJRdPMzOzlFw8zczMUnLxNDMzS8nF08zMLCUXTzMzKxsDBgxg6NChVFdXs99++21s/+EPf8iee+7J3nvvzcUXX1z0OHoU/QxdhKQssAzoCTQCM4AbImLDJvYZADwYEUMk7Qd8MyLO7YRwzcysFY8//jj9+vX70Pp9993H0qVL6dWrFytXrix6DFtM8QTqI6IaQFJ/4HagD3BZITtHxAJgQfHC61j1DVkGTJhd6jA63AVDGxlXgXlB5ebmvMpPqXOrvfrIVP1vvvlmJkyYQK9evQDo379/McL6kC1y2jYiVgLjgbOV013StZLmS1oq6Yzm+0iqkfRgslwl6VZJy5L+X0vab5a0QNLvJU3O2/dqSX9I+k5J2r4u6XlJSyQ9mbS1GEdy7oykeyS9IOk2SSr+d8rMrGuRxKGHHsq+++7L1KlTAXjppZd46qmnGDlyJF/+8peZP39+0ePYkkaeHxIRr0rqBvQHjgbei4jPS+oFPC3pYSBa2X1i0n8ogKSPJe3fiYh3JHUHHpM0DFgOHAsMjoiQ1DfpOwn4p4j4c17baa3EAbAPsDfwJvA08EXgtx3yzTAzKxNPP/00u+yyCytXrmTUqFEMHjyYxsZG3n33XebNm8f8+fM54YQTePXVVynmGGOLLZ6Jpu/socAwSccn632APYCXWtnvK8A3mlYi4t1k8QRJ48l9X3cGPgv8AVgHTJM0G3gw6fs0MF3SXcCv2ojjA+DZiFgOIGkxMIBmxTM593iAfv12ZNLQxsK+C2Vkp965KaVKVKm5Oa/yU+rcMpnMJre/9FLuv+Z99tmHO+64g2222YaBAwfyxBNPAPDBBx9w33330bdv3w/tV1dX1+axC7XFFk9JA4EssJJcET0nIuY06zOgtd1pNiqV9GngQuDzEfGupOnA1hHRKGkEcAi5gns2cHBEnClpJHAksFhS9SbiqAHW5zVlaeFnFxFTgakAuw0cFNctq7wf7wVDG6nEvKByc3Ne5afUudWOqWmxfe3atWzYsIHtttuOtWvXcumllzJp0iSGDx/Om2++SU1NDS+99BLdunXj6KOP/sjIM5PJUFPT8rHTqsyffBsk7Qj8GLgpmUqdA5wlaW5ENEj6DPDnTRziYXJF8LzkeB8DtgfWAu9J2gk4HMhIqgK2iYjfSJoHvJLs8w8R8Tvgd5KOAj4FpI2jVb17dufFlBfdy0Emk2n1H1a5q9TcnFf56aq5vfXWWxx77LEANDY2ctJJJ3HYYYfxwQcfcOqppzJkyBC22morZsyYUdQpW9iyimfvZLqz6a0qM4HvJ9umkZsGfS65EWcVcMwmjnUF8CNJz5MbBU6OiF9JWgT8HniV3LQswHbAfZK2JjeyPD9pv1bSHknbY8ASYGnKOMzMthgDBw5kyZIlH2nfaqutmDVrVqfGssUUz4jovoltG4BLk1e+94AhSZ8MkEmW64CxLRxnXCunGNFC3+NaCqWVODaeO9n37FbOY2ZmnWCLfKuKmZnZ5nDxNDMzS8nF08zMLCUXTzMzs5RcPM3MzFJy8TQzM0vJxdPMzCwlF08zM7OUXDzNzMxScvE0MzNLycXTzMwsJRdPMzOzlFw8zczMUnLxNDMzS8nF08zMLCUXTzMzs5RcPM3MzFJy8TQzM0vJxdPMrAvKZrPss88+jB49GoDTTjuN4cOHM2zYMI4//njq6upKHOGWrUexDiwpCywDegKNwAzghojYsIl9BgD/GBG3FyuutmxODHk59wD+CIyNiL91aIAFqm/IMmDC7FKcuqguGNrIuArMCyo3N+fVutqrj2x124033shee+3FmjVrALj++uvZfvvtAfj3f/93brrpJiZMmLBZ57f2K+bIsz4iqiNib2AUcARwWRv7DABOSnMSSd3bF17HxZCnKechwAfAmR0WVTOSivaHj5mV1vLly5k9ezann376xramwhkR1NfXI6lU4RmdNG0bESuB8cDZyuku6VpJ8yUtlXRG0vVq4EuSFks6v7V+kmokPS7pdnIjPSRNlPSCpEck3SHpwqT9HyQ9JGmhpKckDU7ap0v6gaT/lfSqpONbiWFvSc8m60sl7VFg2k8Bg5Jz3Zuc//eSxjd1kFQn6TpJz0l6TNKOBcT8fUmPA99r78/DzLq28847j2uuuYZu3T78X/Qpp5zCJz7xCV544QXOOeecEkVnAIqI4hxYqouIqmZt7wKDgaOB/hFxhaRewNPA14HdgQsjYnTSf/wm+s0GhkTEa5L2A6YB+5ObMn0O+ElETJH0GHBmRLwsaSRwVUQcLGk6sC1wYhLT/RExSFJNsxh+CMyLiNskbQV0j4j6TeWcjAp/CTwUETdL2iEi3pHUG5gPfDki3pYUwMnJsScluZ7dRsz9gKMjItvC+ceT+yOFfv123HfSDbcU/PMqFzv1hrda/O6Xv0rNzXm1buiufT7S9swzzzBv3jzOP/98Fi9ezJ133slVV121cXs2m+UHP/gBgwcP5vDDD9+8AFpRV1dHVVVV2x3LTCF5HXTQQQsjYr+2jtXZU39N8wyHAsPyRnt9gD3ITXXm21S/ZyPitaT9AOC+pqIm6YHkaxXwj8DdeVMcvfKOf29yDfYPknZqJeZngO9I+iTwq4h4eRP59Za0OFl+CvhpsnyupGOT5U8lObwNbADuTNpnAb8qIOa7WyqcABExFZgKsNvAQXHdssqb2b1gaCOVmBdUbm7Oq3W1Y2o+0jZnzhwWLlzIuHHjWLduHWvWrGHatGnMmjVrY58ePXpw7bXX8r3vFWcCKpPJUFPz0djKXUfm1Wm/0ZIGAllgJbkiek5EzGnWp6b5bpvot7ZZv5Z0A1ZHRHUr29e3dYyIuF3S74AjgTmSTo+Iua0cr775uZJYvwLsHxF/k5QBtm5l/ygg5rWttH9I757deXETNyOUq0wm0+J/OJWgUnNzXulcddVVG0eamUyGKVOmMHPmTF555RUGDRpERPDAAw8wePDgDj+3Fa5Trnkm1/J+DNwUuXniOcBZknom2z8jaVvgfWC7vF1b69fcb4GjJG2djNyOBIiINcBrkr6e7C9Jw9sI90MxJEX/1Yj4AXA/MCxl+n2Ad5PCORj4Qt62bkDTqPok4LftjNnMKlhEMHbsWIYOHcrQoUNZsWIFkyZNKnVYW7RijjybpjCb3qoyE/h+sm0aubtan1NubnIVcAywFGiUtASYDtzYSr8PiYj5ku4HlgCvAwuA95LNY4CbJf1nEssvkn6taR7D1sDJkhqAvwDfTfl9eAg4U9JS4EVgXt62tcDekhYm8Z7YzpjNrALV1NRsnGZ8+umnSxuMfUjRimdEtPoWkuQ646XJq7lDmq231C+TvPJNiYjLJW0DPAlcl5zrNeCwFmIY12y9Kvna0EIMV1GA5jdIJW3rgVav6kfERGBis7aCYjYzs9KopKv4UyV9ltxIcUZEPFfqgMzMrDJVTPGMiPZ+sEEqkj4OPNbCpkMi4u00x2pppGpmZl1fxRTPzpIUyNbuhDUzsy2APxjezMwsJRdPMzOzlFw8zczMUnLxNDMzS8nF08zMLCUXTzMzs5RcPM3MzFJy8TQzM0vJxdPMzCwlF08zM7OUXDzNzMxScvE0MzNLycXTzMwsJRdPMzOzlFw8zcxKLJvNss8++zB69GgAxowZw5577smQIUM49dRTaWhoKHGE1pyf59lOkrLAsrymYyKitkThfER9Q5YBE2aXOowOd8HQRsZVYF5Qubk5L6i9+shNbr/xxhvZa6+9WLNmDZArnrNmzQLgpJNOYtq0aZx11lmbF7B1KI88268+IqrzXrX5GyX5DxMza9Py5cuZPXs2p59++sa2I444AklIYsSIESxfvryEEVpLXDw7kKRxku6W9ADwsHKulfS8pGWSTkz6fVfS4uT1Z0m3Ju0nS3o2af+JpO5Je52kKyUtkTRP0k4lTNPMOtB5553HNddcQ7duH/3vuKGhgZkzZ3LYYYeVIDLbFI+O2q+3pMXJ8msRcWyyvD8wLCLekfQ1oBoYDvQD5kt6MiImAZMk9QGeAm6StBdwIvDFiGiQ9N/AGODnwLbAvIj4jqRrgG8BVzQPSNJ4YDxAv347MmloY5FSL52deuemyypRpebmvCCTybTY/swzz9DQ0MD777/P4sWLefvttz/Ud8qUKQwcOJBsNtvqMYqhrq6uU8/XWToyLxfP9quPiOoW2h+JiHeS5QOAOyIiC7wl6Qng88D9kgTcBlwfEQslnQ3sS67AAvQGVibH+QB4MFleCIxqKaCImApMBdht4KC4blnl/XgvGNpIJeYFlZub84LaMTUtts+ZM4eFCxcybtw41q1bx5o1a5g2bRqzZs1i8uTJ9OjRg7vuuqvFUWkxZTIZampajrmcdWRenrbteGvzlrWJfpcDyyPi1ry+M/Kuoe4ZEZcn2xoiIpLlLP6jx6wiXHXVVSxfvpza2lp+8YtfcPDBBzNr1iymTZvGnDlzuOOOOzq9cFph/J9wcT0JnCFpBrADcCBwkaTR5EaPNXl9HwPuk3R9RKyUtAOwXUS83p4T9+7ZnRfbuMOvHGUymVb/ii93lZqb80rvzDPPZPfdd2f//fcH4LjjjmPSpElFOZe1j4tncf2a3DXQJUAAF0fEXyRdAOwCPJtM0d4fEZMk/Se5G426AQ3At4F2FU8zKy81NTUbpxQbGyvvGnGlcfFsp4ioaqFtOjA9bz2Ai5JXfr+DWjnmncCdmzpXRNwD3NPOsM3MrAN4Mt3MzCwlF08zM7OUUhdPSR+TNKwYwZiZmZWDgoqnpIyk7ZM7QJcAt0r6fnFDMzMz65oKHXn2iYg1wHHArRGxL/CV4oVlZmbWdRVaPHtI2hk4gb9/0o2ZmdkWqdDi+V1gDvCniJgvaSDwcvHCMjMz67oKep9nRNwN3J23/irwtWIFZWZm1pUVesPQZyQ9Jun5ZH1Y8mk4ZmZmW5xCp21vAf6D3EfGERFLgW8UKygzM7OurNDiuU1EPNuszR++aGZmW6RCi+dfJf0DuQ83R9LxwIqiRWVmZtaFFfrB8N8m95DlwZL+DLwGjClaVGZmZl1Ym8UzeTzWfhHxFUnbAt0i4v3ih2ZmZtY1tTltGxEbgLOT5bUunGZmtqUr9JrnI5IulPQpSTs0vYoamZmZWRdV6DXPU5Ov385rC2Bgx4ZjZmbW9RX6CUOfLnYgZmbFsm7dOg488EDWr19PY2Mjxx9/PJMnT2bcuHE88cQT9OnTB4Dp06dTXV1d4mitHBRUPCV9s6X2iPh5x4bTNUnaCbge+ALwLvABcE1E/LqkgW1CfUOWARNmlzqMDnfB0EbGVWBeULm5dWZetVcf2WJ7r169mDt3LlVVVTQ0NHDAAQdw+OGHA3Dttddy/PHHd0p8VjkKnbb9fN7y1sAhwHNAxRdPSQLuBWZExElJ2+7AV5v16xERHf7BEcU6rtmWRBJVVVUANDQ00NDQQO6ftln7FHTDUESck/f6FrAPsFVxQ+syDgY+iIgfNzVExOsR8UNJ4yTdLekB4GHlXCvpeUnLJJ3YtI+ki5O2JZKuTtr+QdJDkhZKekrS4KR9uqTvS3ocuFbSy5J2TLZ1k/SKpH6d+l0wK3PZbJbq6mr69+/PqFGjGDlyJADf+c53GDZsGOeffz7r168vcZRWLhQR6XeSegJLI2Kvjg+pa5F0LvDpiDi/hW3jgCuAYRHxjqSvAWcChwH9gPnASKAamAh8JSL+JmmHpP9jwJkR8bKkkcBVEXGwpOnJ/kdHRFbSZcB7EXGDpEOBMyLiI0+1kTQeGA/Qr9+O+0664ZaO/naU3E694a36UkdRHJWaW2fmNXTXPm32qaurY+LEiZx77rlsv/327LDDDjQ0NHDdddexyy67MHbs2ILOVVdXt3E0W2kqNbdC8jrooIMWRsR+bR2r0GueD5B8NB+50epnyXtE2ZZE0o+AA8hd9/wR8EhEvJNsPgC4IyKywFuSniA35f1l4NaI+BtAUjirgH8E7s6bPuqVd6q7k+MA/Ay4D7iB3J3Pt7YUW0RMJfdJUOw2cFBct6zQWfnyccHQRioxL6jc3Dozr9oxNQX1W7hwIW+//TannHLKxratttqKKVOmUFNT2DEymUzBfctNpebWkXkV+hs9JW+5EXg9IpZ3SARd3+/Je3ZpRHw7mTJdkDStzevb2kUU8fc/Ppp0A1ZHRGu39m08bkS8IektSQeTG8n6oxHNUli1ahU9e/akb9++1NfX8+ijj3LJJZewYsUKdt55ZyKCe++9lyFDhpQ6VCsThRbPIyLikvwGSd9r3lah5gL/T9JZEXFz0rZNK32fBM6QNAPYATgQuIjcKHWSpNubTdu+JunrEXF3cmPSsIhY0sqxpwGzgJl5I9JW9e7ZnRdbufOwnGUymYJHF+WmUnPrCnmtWLGCsWPHks1m2bBhAyeccAKjR4/m4IMPZtWqVUQE1dXV/PjHP277YGYUXjxHAc0L5eEttFWciAhJxwDXS7oYWEVuVHgJ0LtZ918D+wNLyI00L46IvwAPSaoGFkj6APgNcCm5EeTNyYPFewK/SPZtyf3kpmtbnLI1s9YNGzaMRYsWfaR97ty5JYjGKsEmi6eks4B/BQZKWpq3aTvg6WIG1pVExApaf/j39Lx+QW6keVELx7gauLpZ22vkbi5q3ndcC+cZDiyJiBcKjdvMzIqjrZHn7cD/AFcBE/La38+7ScaKTNIE4Cx8rdPMrEvY5Ps8I+K9iKiNiH+OiNeBenLTkVWSduuUCI2IuDoido+I35Y6FjMzK/BDEiQdJellcg/BfgKoJTciNTMz2+IU+kiyK8h9rutLyYfEH8IWdM3TzMwsX6HFsyEi3ga6SeoWEY+T+9QcMzOzLU6hb1VZnXwizlPAbZJWkvuwBDMzsy1OoSPPo4G/AecBDwF/Ao4qVlBmZmZdWaEPw16bPIZrj4iYIWkboHtxQzMzM+uaCr3b9lvAPcBPkqZdyT3j0szMbItT6LTtt4EvAmsAIuJloH+xgjIzM+vKCi2e6yPig6YVST346FNCzMzMtgiFFs8nJF0K9JY0ityzPB8oXlhmZmZdV6HFcwK5p4ksA84g91SQ/yxWUGZmZl1ZW09V2S0i/i8iNgC3JC8zM7MtWlsjz4131Er6ZZFjMTMzKwttFU/lLQ8sZiBmZmbloq3iGa0sm5mZbbHaKp7DJa2R9D4wLFleI+l9SWs6I0AzszTWrVvHiBEjGD58OHvvvTeXXXbZh7afc845VFVVlSg6qxRtPQy7e0RsHxHbRUSPZLlpffvOCrKSSApJM/PWe0haJenBZP2rkiYky5dLujBZzkjarzRRm5WPXr16MXfuXJYsWcLixYt56KGHmDdvHgALFixg9erVJY7QKkGhT1WxjrMWGCKpd0TUA6OAPzdtjIj7gfs39yT1DVkGTJi9uYfpci4Y2si4CswLKje3YuVVe/WRLbZL2jiybGhooKGhAUlks1kuuugibr/9dn796193eDy2ZSn0fZ7Wsf4HaPqX/8/AHU0bJI2TdFNrO0rqJmmGpCuKHKNZ2cpms1RXV9O/f39GjRrFyJEjuemmm/jqV7/KzjvvXOrwrAJ45FkavwAmJVO1w4CfAV8qYL8ewG3A8xFxZfONksYD4wH69duRSUMr75GrO/XOjWQqUaXmVqy8MpnMJrffcMMN1NXVMXHiRHbZZRemTZvGDTfcQCaTIZvNtrl/W+rq6jb7GF1VpebWkXm5eJZARCyVNIDcqPM3KXb9CXBXS4UzOe5UYCrAbgMHxXXLKu/He8HQRioxL6jc3IqVV+2YmoL6LVy4kNWrV7Nq1SpOO+00ANavX8/pp5/OK6+80u7zZzIZamoKi6HcVGpuHZmXp21L535gCnlTtgX4X+AgSVsXJySz8rdq1aqNNwXV19fz6KOPsu+++/KXv/yF2tpaamtr2WabbTarcJpV3p+55eNnwHsRsUxSTYH7/BQ4ELhb0rER0epcWO+e3XmxlRsqylkmkyl4xFFuKjW3zs5rxYoVjB07lmw2y4YNGzjhhBMYPXp0p53ftgwuniUSEcuBG9ux3/cl9QFmShqTfO6wmSWGDRvGokWLNtmnrq6uk6KxSuXi2cki4iPvzo6IDJBJlqcD05Ply/P61OQtf/hd32Zm1ql8zdPMzCwlF08zM7OUXDzNzMxScvE0MzNLycXTzMwsJRdPMzOzlFw8zczMUnLxNDMzS8nF08zMLCUXTzMzs5RcPM3MzFJy8TQzM0vJxdPMzCwlF08zM7OUXDzNzMxScvE0MzNLycXTzMwsJRdPM+vy1q1bx4gRIxg+fDh77703l112GQBjxoxhzz33ZMiQIZx66qk0NDSUOFLbUrh4djBJWUmL814DOuCYtZL6bX50ZuWpV69ezJ07lyVLlrB48WIeeugh5s2bx5gxY3jhhRdYtmwZ9fX1TJs2rdSh2haiR6kDqED1EVHd2kZJPSKisehBNGQZMGF2sU/T6S4Y2si4CswLKje3NHnVXn1ki+2SqKqqAqChoYGGhgYkccQRR2zsM2LECJYvX775AZsVwCPPTiBpnKS7JT0APJy0XSRpvqSlkiYnbdtKmi1piaTnJZ2Yd5hzJD0naZmkwaXIw6yUstks1dXV9O/fn1GjRjFy5MiN2xoaGpg5cyaHHXZYCSO0LYkiotQxVBRJWWBZsvpaRBwraRxwBTAsIt6RdChwPHAGIOB+4BpgR+CwiPhWcqw+EfGepFrguoj4oaR/BT4XEae3cO7xwHiAfv123HfSDbcUM9WS2Kk3vFVf6iiKo1JzS5NbWN+bAAALnUlEQVTX0F37tNmnrq6OiRMncu655/LpT38agClTprD11ltz9tlnb06oqdTV1W0cDVeaSs2tkLwOOuighRGxX1vH8rRtx2tt2vaRiHgnWT40eS1K1quAPYCngCmSvgc8GBFP5e3/q+TrQuC4lk4cEVOBqQC7DRwU1y2rvB/vBUMbqcS8oHJzS5NX7ZiagvotXLiQt99+m1NOOYXJkyfTo0cP7rrrLrp167zJtEwmQ01NTaedrzNVam4dmZenbTvP2rxlAVdFRHXyGhQRP42Il4B9yY1cr5I0KW+f9cnXLP6jx7Ywq1atYvXq1QDU19fz6KOPMnjwYKZNm8acOXO44447OrVwmvk/4dKYA/yXpNsiok7SrkADuZ/HOxExS1IdMK69J+jdszsvtnLzRTnLZDIFj07KTaXm1hF5rVixgrFjx5LNZtmwYQMnnHACo0ePpkePHuy+++7sv//+ABx33HFMmjSpjaOZbT4XzxKIiIcl7QU8IwmgDjgZGARcK2kDuWJ6VumiNOs6hg0bxqJFiz7S3thY9BvXzVrk4tnBIuIjV6MjYjowvVnbjcCNzbr+idyotPn+A/KWFwA1mx2omZm1my8SmJmZpeTiaWZmlpKLp5mZWUounmZmZim5eJqZmaXk4mlmZpaSi6eZmVlKLp5mZmYpuXiamZml5OJpZmaWkounmZlZSi6eZmZmKbl4mpmZpeTiaWZmlpKLp5mZWUounmZmZim5eJqZmaXk4mmprFu3jhEjRjB8+HD23ntvLrvsslKHZGbW6Vw820FSVtJiSc9LekBS38041nclfaUj4yumXr16MXfuXJYsWcLixYt56KGHmDdvXqnDMjPrVD1KHUCZqo+IagBJM4BvA1e250ARMakjA2tS35BlwITZ7d6/9uojW2yXRFVVFQANDQ00NDQgqd3nMTMrRx55br5ngF2bViRdJGm+pKWSJue1T5T0gqRHJN0h6cKkfbqk45PlQyQtkrRM0s8k9UraayVNlvRcsm1wJ+f4Idlslurqavr378+oUaMYOXJkKcMxM+t0Lp6bQVJ34BDg/mT9UGAPYARQDewr6UBJ+wFfA/YBjgP2a+FYWwPTgRMjYii5WYGz8rr8NSI+B9wMXFisnArRvXt3Fi9ezPLly3n22Wd5/vnnSxmOmVmn87Rt+/SWtBgYACwEHknaD01ei5L1KnLFdDvgvoioB5D0QAvH3BN4LSJeStabpoNvSNZ/lXxdSK4Af4Sk8cB4gH79dmTS0Mb25AZAJpMpqN+AAQP40Y9+xIknntjuc6VRV1dXcGzlplJzc17lp1Jz68i8XDzbpz4iqiX1AR4kV+R+AAi4KiJ+kt9Z0vkFHLOtC4frk69ZWvm5RcRUYCrAbgMHxXXL2v/jrR1T02L7qlWr6NmzJ3379qW+vp6JEydyySWXUFPTcv+OlslkOu1cna1Sc3Ne5adSc+vIvFw8N0NEvCfpXOA+STcDc4D/knRbRNRJ2hVoAH4L/ETSVeS+50cCtzQ73AvAAEmDIuIV4F+AJ9obW++e3XmxlZt+NseKFSsYO3Ys2WyWDRs2cMIJJzB69OgOP4+ZWVfm4rmZImKRpCXANyJipqS9gGeSO1DrgJMjYr6k+4ElwOvAAuC9ZsdZJ+kU4G5JPYD5wI87M5dCDBs2jEWLFrXd0cysgrl4tkNEVDVbPypv+UbgxhZ2mxIRl0vaBngSuC7pPy5v38fI3VTU/HwD8pYXADWblYCZmW0WF8/OM1XSZ4GtgRkR8VypAzIzs/Zx8ewkEXFSqWMwM7OO4fd5mpmZpeTiaWZmlpKLp5mZWUounmZmZim5eJqZmaXk4mlmZpaSi6eZmVlKLp5mZmYpuXiamZml5OJpZmaWkounmZlZSi6eZmZmKbl4mpmZpeTiaWZmlpKLp5mZWUounmZmZim5eJqZmaXk4mlmZpaSi6eZmVlKLp5mZmYpKSJKHYMVgaT3gRdLHUcR9AP+WuogiqRSc3Ne5adScyskr90jYse2DtSjY+KxLujFiNiv1EF0NEkLKjEvqNzcnFf5qdTcOjIvT9uamZml5OJpZmaWkotn5Zpa6gCKpFLzgsrNzXmVn0rNrcPy8g1DZmZmKXnkaWZmlpKLp5mZWUounhVG0mGSXpT0iqQJpY5nc0j6maSVkp7Pa9tB0iOSXk6+fqyUMbaHpE9JelzSHyX9XtK/Je1lnZukrSU9K2lJktfkpP3Tkn6X5HWnpK1KHWt7SOouaZGkB5P1SsmrVtIySYslLUjayvp3EUBSX0n3SHoh+be2f0fm5eJZQSR1B34EHA58FvhnSZ8tbVSbZTpwWLO2CcBjEbEH8FiyXm4agQsiYi/gC8C3k59Tuee2Hjg4IoYD1cBhkr4AfA+4PsnrXeC0Esa4Of4N+GPeeqXkBXBQRFTnvQey3H8XAW4EHoqIwcBwcj+7DsvLxbOyjABeiYhXI+ID4BfA0SWOqd0i4kngnWbNRwMzkuUZwDGdGlQHiIgVEfFcsvw+uX/Uu1LmuUVOXbLaM3kFcDBwT9JednkBSPokcCQwLVkXFZDXJpT176Kk7YEDgZ8CRMQHEbGaDszLxbOy7Aq8kbe+PGmrJDtFxArIFSGgf4nj2SySBgD7AL+jAnJLpjYXAyuBR4A/AasjojHpUq6/kzcAFwMbkvWPUxl5Qe4PnIclLZQ0Pmkr99/FgcAq4NZkqn2apG3pwLxcPCuLWmjze5G6KElVwC+B8yJiTanj6QgRkY2IauCT5GZC9mqpW+dGtXkkjQZWRsTC/OYWupZVXnm+GBGfI3e559uSDix1QB2gB/A54OaI2AdYSwdPPbt4VpblwKfy1j8JvFmiWIrlLUk7AyRfV5Y4nnaR1JNc4bwtIn6VNFdEbgDJFFmG3DXdvpKaPke7HH8nvwh8VVItuUshB5MbiZZ7XgBExJvJ15XAr8n90VPuv4vLgeUR8btk/R5yxbTD8nLxrCzzgT2SuwC3Ar4B3F/imDra/cDYZHkscF8JY2mX5HrZT4E/RsT38zaVdW6SdpTUN1nuDXyF3PXcx4Hjk25ll1dE/EdEfDIiBpD7NzU3IsZQ5nkBSNpW0nZNy8ChwPOU+e9iRPwFeEPSnknTIcAf6MC8/AlDFUbSEeT+Ku4O/CwirixxSO0m6Q6ghtxjhN4CLgPuBe4CdgP+D/h6RDS/qahLk3QA8BSwjL9fQ7uU3HXPss1N0jByN2F0J/eH+V0R8V1JA8mN2HYAFgEnR8T60kXafpJqgAsjYnQl5JXk8OtktQdwe0RcKenjlPHvIoCkanI3eG0FvAqcQvJ7SQfk5eJpZmaWkqdtzczMUnLxNDMzS8nF08zMLCUXTzMzs5RcPM3MzFLq0XYXM7McSVlyb7FpckxE1JYoHLOS8VtVzKxgkuoioqoTz9cj7/NjzboMT9uaWYeRtLOkJ5NnQz4v6UtJ+2GSnkue9flY0raDpHslLZU0L/mQBSRdLmmqpIeBnycfNn+tpPlJ3zNKmKIZ4GlbM0und/LUFIDXIuLYZttPAuYkn1LTHdhG0o7ALcCBEfGapB2SvpOBRRFxjKSDgZ+Tew4owL7AARFRnzzp472I+LykXsDTkh6OiNeKmajZprh4mlka9clTU1ozH/hZ8sH390bE4uQj7Z5sKnZ5H4d2APC1pG2upI9L6pNsuz8i6pPlQ4Fhkpo+R7YPsAfg4mkl4+JpZh0mIp5MHml1JDBT0rXAalp+XNemHuu1tlm/cyJiTocGa7YZfM3TzDqMpN3JPfvyFnJPjvkc8AzwZUmfTvo0Tds+CYxJ2mqAv7byXNM5wFnJaBZJn0meAGJWMh55mllHqgEuktQA1AHfjIhVyXXLX0nqRu4ZiqOAy4FbJS0F/sbfHxXV3DRgAPBc8ji3VcAxxUzCrC1+q4qZmVlKnrY1MzNLycXTzMwsJRdPMzOzlFw8zczMUnLxNDMzS8nF08zMLCUXTzMzs5T+P7zA8e67xhusAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"xgb.plot_importance(xgb_clf)\n",
"plt.rcParams['figure.figsize'] = [6, 4]\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the feature `Grocery` has been given the highest importance score among all the features. Thus XGBoost also gives us a way to do Feature Selection."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 15. Results and conclusion\n",
"\n",
"\n",
"1.\tIn this project, I implement XGBoost with Python and Scikit-Learn to classify the customers from two different channels as Horeca (Hotel/Retail/Café) customers or Retail channel (nominal) customers.\n",
"\n",
"2.\tThe y labels contain values as 1 and 2. I have converted them into 0 and 1 for further analysis.\n",
"3.\tI have trained the XGBoost classifier and found the accuracy score to be 91.67%.\n",
"\n",
"4.\tI have done the hyperparameter tuning in XGBoost by doing k-fold cross-validation.\n",
"\n",
"5.\tI find the most important feature in XGBoost to be `Grocey`. I did it using the **plot_importance()** function in XGBoost that helps us to achieve this task. \n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
@kmlknta21
Copy link

-Nice. It is helpful to run in Jupyter Notebook. Thank you

@malambomutila
Copy link

From the Feature Importance graph, Delicassen has the highest F score. Doesn't this mean that Delicassen was the most important feature as opposed to Grocery which was fourth best?

@ajitbalakrishnan
Copy link

No answer to malambomutila comment?

@Jason2Brownlee
Copy link

Excellent case study!

I was only able to get an accuracy of about 93% with xgboost.

@p-dot-max
Copy link

Thanks a lot

@peet-droid
Copy link

I like how you described each parameter meaning I had no idea you could use Drop out using D.A.R.T

@icebeartellsnolies
Copy link

From the Feature Importance graph, Delicassen has the highest F score. Doesn't this mean that Delicassen was the most important feature as opposed to Grocery which was fourth best?

yes u are right @malambomutila

@snailcoder
Copy link

nice work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment