Skip to content

Instantly share code, notes, and snippets.

@lasershow

lasershow/Porto Seguro Categorical Encoding & Simple E2E pipeline JP.ipynb

Last active March 19, 2018 04:50
Show Gist options
  • Save lasershow/18f2245bab5c52eb10f7c15bfdb016e2 to your computer and use it in GitHub Desktop.
Save lasershow/18f2245bab5c52eb10f7c15bfdb016e2 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Porto Seguro Categorical Encoding & Simple E2E pipeline JP.ipynb
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"from sklearn.preprocessing import StandardScaler\n",
"from sklearn.metrics import roc_auc_score\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.neighbors import KNeighborsClassifier\n",
"import time\n",
"\n",
"pd.set_option('display.max_columns', 500)\n",
"pd.set_option('display.max_colwidth', 500)\n",
"pd.set_option('display.max_rows', 1000)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 他のkaggle競技者へ\n",
"\n",
"このノートブックはKaggleを通してスキルを身に付けたい生徒さんに向けて書かれたものの一つです。ここでは主にKaggleで使われる様々なテクニックをデモンストレーションしていきます。多くの人がLB（リーダーボード）を過学習させることに集中してしまいますが、まずは便利な機械学習やCV（分割交差検証）のテクニックを学んで少しずつ積み重ねていきましょう。\n",
"\n",
"チームになって競う生徒さんもいらっしゃるのでカーネルを使って情報を共有することにしました。どうぞ以下でも積極的にコメント等してください。\n",
"\n",
"では楽しんでください！"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1. Porto Seguo データ概要\n",
"\n",
"このチャレンジでは運転者が保険金請求を行うかどうかを予測する予測モデルを構築します。与えられたデータはすでに処理され、カテゴリカルデータが名前とともに綺麗にラベル付けされています。順不同のカテゴリカルフィーチャもあれば論理的に名付けられた順序値も含まれています。 \n",
"\n",
"データ詳細でも書かれていた通り、\n",
"* *bin* はバイナリフィーチャ\n",
"* *cat* はカテゴリカルフィーチャ\n",
"* これ以外のものは連続的数値変数か順序的定性変数となります。\n",
"\n",
"また、-1という値は欠損値を表しています。\n",
"\n",
"まずは簡単なデータ処理をし、学習およびテストデータのフィーチャがどのようなものなのかまとめてみましょう。"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# テストと学習データを取り込み、２つを結合させます。\n",
"train=pd.read_csv('../input/train.csv')\n",
"test=pd.read_csv('../input/test.csv')\n",
"sample_submission=pd.read_csv('../input/sample_submission.csv')"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"traintest=train.drop(['id','target'], axis=1).append(test.drop(['id'], axis=1))\n",
"cols=traintest.columns\n",
"\n",
"# フィーチャ（特徴量）統計のためカラム（縦列）の名前を定義します。\n",
"# 名の通りフィーチャから以下の統計を求めます。\n",
"# nunique: 一意的な値の個数\n",
"# freq1: 出現頻度が最も高い値\n",
"# freq1_val: 出現頻度の最も高い値の出現回数\n",
"# freq2: 二番目に出現頻度の高い値 \n",
"# freq2_val: 二番目に出現頻度の高い値の出現回数\n",
"# freq3: 三番目に出現頻度の高い値（あれば）\n",
"# freq3_val: 三番目に出現頻度の高い値の出現回数（あれば）\n",
"# .describe機能を使った統計も行います。\n",
"\n",
"stat_cols= ['nunique','freq1','freq1_val', 'freq2', 'req2_val',\n",
" 'freq3', 'freq3_val'] + traintest[cols[0]].describe().index.tolist()[1:]\n",
"\n",
"stat_cols=['feature']+stat_cols\n",
"\n",
"feature_stat=pd.DataFrame(columns=stat_cols)\n",
"i=0\n",
"\n",
"for col in cols:\n",
" stat_vals=[]\n",
" \n",
" # 統計値を求めます。\n",
" stat_vals.append(col)\n",
" stat_vals.append(traintest[col].nunique())\n",
" stat_vals.append(traintest[col].value_counts().index[0])\n",
" stat_vals.append(traintest[col].value_counts().iloc[0])\n",
" stat_vals.append(traintest[col].value_counts().index[1])\n",
" stat_vals.append(traintest[col].value_counts().iloc[1])\n",
" \n",
" if len(traintest[col].value_counts())>2:\n",
" stat_vals.append(traintest[col].value_counts().index[2])\n",
" stat_vals.append(traintest[col].value_counts().iloc[2])\n",
" else:\n",
" stat_vals.append(np.nan)\n",
" stat_vals.append(np.nan)\n",
" \n",
" stat_vals+=traintest[col].describe().tolist()[1:]\n",
"\n",
" feature_stat.loc[i]=stat_vals\n",
" i+=1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"このリストを一意的な値の個数で整理して、カテゴリカルフィーチャを詳しく見てみましょう。これは後で「特別扱い」が必要になってくるものです。"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style>\n",
" .dataframe thead tr:only-child th {\n",
" text-align: right;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>feature</th>\n",
" <th>nunique</th>\n",
" <th>freq1</th>\n",
" <th>freq1_val</th>\n",
" <th>freq2</th>\n",
" <th>req2_val</th>\n",
" <th>freq3</th>\n",
" <th>freq3_val</th>\n",
" <th>mean</th>\n",
" <th>std</th>\n",
" <th>min</th>\n",
" <th>25%</th>\n",
" <th>50%</th>\n",
" <th>75%</th>\n",
" <th>max</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>ps_car_08_cat</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>1238365</td>\n",
" <td>0</td>\n",
" <td>249663</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>0.832219</td>\n",
" <td>0.373672</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>ps_ind_04_cat</td>\n",
" <td>3</td>\n",
" <td>0</td>\n",
" <td>866864</td>\n",
" <td>1</td>\n",
" <td>620936</td>\n",
" <td>-1</td>\n",
" <td>228</td>\n",
" <td>0.417135</td>\n",
" <td>0.493396</td>\n",
" <td>-1.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>ps_car_02_cat</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>1234979</td>\n",
" <td>0</td>\n",
" <td>253039</td>\n",
" <td>-1</td>\n",
" <td>10</td>\n",
" <td>0.829937</td>\n",
" <td>0.375706</td>\n",
" <td>-1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>ps_car_03_cat</td>\n",
" <td>3</td>\n",
" <td>-1</td>\n",
" <td>1028142</td>\n",
" <td>1</td>\n",
" <td>276842</td>\n",
" <td>0</td>\n",
" <td>183044</td>\n",
" <td>-0.504896</td>\n",
" <td>0.788713</td>\n",
" <td>-1.0</td>\n",
" <td>-1.0</td>\n",
" <td>-1.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>ps_car_05_cat</td>\n",
" <td>3</td>\n",
" <td>-1</td>\n",
" <td>666910</td>\n",
" <td>1</td>\n",
" <td>431560</td>\n",
" <td>0</td>\n",
" <td>389558</td>\n",
" <td>-0.158162</td>\n",
" <td>0.844506</td>\n",
" <td>-1.0</td>\n",
" <td>-1.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>ps_car_07_cat</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>1383070</td>\n",
" <td>0</td>\n",
" <td>76138</td>\n",
" <td>-1</td>\n",
" <td>28820</td>\n",
" <td>0.910097</td>\n",
" <td>0.347212</td>\n",
" <td>-1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>30</th>\n",
" <td>ps_car_10_cat</td>\n",
" <td>3</td>\n",
" <td>1</td>\n",
" <td>1475460</td>\n",
" <td>0</td>\n",
" <td>12136</td>\n",
" <td>2</td>\n",
" <td>432</td>\n",
" <td>0.992135</td>\n",
" <td>0.091565</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>2.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>ps_ind_02_cat</td>\n",
" <td>5</td>\n",
" <td>1</td>\n",
" <td>1079327</td>\n",
" <td>2</td>\n",
" <td>309747</td>\n",
" <td>3</td>\n",
" <td>70172</td>\n",
" <td>1.358745</td>\n",
" <td>0.663639</td>\n",
" <td>-1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>2.0</td>\n",
" <td>4.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>ps_car_09_cat</td>\n",
" <td>6</td>\n",
" <td>2</td>\n",
" <td>883326</td>\n",
" <td>0</td>\n",
" <td>486510</td>\n",
" <td>1</td>\n",
" <td>72947</td>\n",
" <td>1.328302</td>\n",
" <td>0.978743</td>\n",
" <td>-1.0</td>\n",
" <td>0.0</td>\n",
" <td>2.0</td>\n",
" <td>2.0</td>\n",
" <td>4.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>ps_ind_05_cat</td>\n",
" <td>8</td>\n",
" <td>0</td>\n",
" <td>1319412</td>\n",
" <td>6</td>\n",
" <td>51877</td>\n",
" <td>4</td>\n",
" <td>45706</td>\n",
" <td>0.406955</td>\n",
" <td>1.353300</td>\n",
" <td>-1.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>6.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>ps_car_04_cat</td>\n",
" <td>10</td>\n",
" <td>0</td>\n",
" <td>1241334</td>\n",
" <td>1</td>\n",
" <td>80561</td>\n",
" <td>2</td>\n",
" <td>59088</td>\n",
" <td>0.725556</td>\n",
" <td>2.154316</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>9.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>ps_car_01_cat</td>\n",
" <td>13</td>\n",
" <td>11</td>\n",
" <td>518725</td>\n",
" <td>7</td>\n",
" <td>449617</td>\n",
" <td>6</td>\n",
" <td>155779</td>\n",
" <td>8.293596</td>\n",
" <td>2.508307</td>\n",
" <td>-1.0</td>\n",
" <td>7.0</td>\n",
" <td>7.0</td>\n",
" <td>11.0</td>\n",
" <td>11.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>ps_car_06_cat</td>\n",
" <td>18</td>\n",
" <td>11</td>\n",
" <td>329890</td>\n",
" <td>1</td>\n",
" <td>295574</td>\n",
" <td>0</td>\n",
" <td>275497</td>\n",
" <td>6.560714</td>\n",
" <td>5.500869</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>7.0</td>\n",
" <td>11.0</td>\n",
" <td>17.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>31</th>\n",
" <td>ps_car_11_cat</td>\n",
" <td>104</td>\n",
" <td>104</td>\n",
" <td>212989</td>\n",
" <td>103</td>\n",
" <td>61062</td>\n",
" <td>64</td>\n",
" <td>55391</td>\n",
" <td>62.256845</td>\n",
" <td>33.007217</td>\n",
" <td>1.0</td>\n",
" <td>32.0</td>\n",
" <td>65.0</td>\n",
" <td>94.0</td>\n",
" <td>104.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" feature nunique freq1 freq1_val freq2 req2_val freq3 freq3_val \\\n",
"28 ps_car_08_cat 2 1 1238365 0 249663 NaN NaN \n",
"3 ps_ind_04_cat 3 0 866864 1 620936 -1 228 \n",
"22 ps_car_02_cat 3 1 1234979 0 253039 -1 10 \n",
"23 ps_car_03_cat 3 -1 1028142 1 276842 0 183044 \n",
"25 ps_car_05_cat 3 -1 666910 1 431560 0 389558 \n",
"27 ps_car_07_cat 3 1 1383070 0 76138 -1 28820 \n",
"30 ps_car_10_cat 3 1 1475460 0 12136 2 432 \n",
"1 ps_ind_02_cat 5 1 1079327 2 309747 3 70172 \n",
"29 ps_car_09_cat 6 2 883326 0 486510 1 72947 \n",
"4 ps_ind_05_cat 8 0 1319412 6 51877 4 45706 \n",
"24 ps_car_04_cat 10 0 1241334 1 80561 2 59088 \n",
"21 ps_car_01_cat 13 11 518725 7 449617 6 155779 \n",
"26 ps_car_06_cat 18 11 329890 1 295574 0 275497 \n",
"31 ps_car_11_cat 104 104 212989 103 61062 64 55391 \n",
"\n",
" mean std min 25% 50% 75% max \n",
"28 0.832219 0.373672 0.0 1.0 1.0 1.0 1.0 \n",
"3 0.417135 0.493396 -1.0 0.0 0.0 1.0 1.0 \n",
"22 0.829937 0.375706 -1.0 1.0 1.0 1.0 1.0 \n",
"23 -0.504896 0.788713 -1.0 -1.0 -1.0 0.0 1.0 \n",
"25 -0.158162 0.844506 -1.0 -1.0 0.0 1.0 1.0 \n",
"27 0.910097 0.347212 -1.0 1.0 1.0 1.0 1.0 \n",
"30 0.992135 0.091565 0.0 1.0 1.0 1.0 2.0 \n",
"1 1.358745 0.663639 -1.0 1.0 1.0 2.0 4.0 \n",
"29 1.328302 0.978743 -1.0 0.0 2.0 2.0 4.0 \n",
"4 0.406955 1.353300 -1.0 0.0 0.0 0.0 6.0 \n",
"24 0.725556 2.154316 0.0 0.0 0.0 0.0 9.0 \n",
"21 8.293596 2.508307 -1.0 7.0 7.0 11.0 11.0 \n",
"26 6.560714 5.500869 0.0 1.0 7.0 11.0 17.0 \n",
"31 62.256845 33.007217 1.0 32.0 65.0 94.0 104.0 "
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"feature_stat[feature_stat['feature'].str.contains(\"cat\")].sort_values(by=['nunique'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"いくつかあることがわかります。特に ps_car_11_cat のレベル数が高いようです。"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 2. カテゴリカル・フィーチャー・エンコーディング\n",
"\n",
"このデータセットのカテゴリカルフィーチャは既に数値型変換され、運転タイプに関わる様々なクラス（例えばFWDやRWD、4WD）がそれぞれ0,1,2というように数字で表されています。機械学習アルゴリズムの多くは文字列よりも数値データを好むため、数値形式でないデータは予めこのように変換しておくと良いです。\n",
"\n",
"ただし、0,1,2というように、FFから4DWまで運転タイプをランク付ける正しい順序は必ずしも存在しないため、これが返ってまた問題になってしまうこともあります。\n",
"\n",
"順序付けずにエンコーディングする方法として、フリークエンシーエンコーディング（frequency encoding）やバイナリエンコーディング（binary encoding）があります。早速試してみましょう。"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.1 フリークエンシー・エンコーディング\n",
"\n",
"フリークエンシー・エンコーディングはカテゴリカルフィーチャのレベルをデータで現れた分で表す方法です。頻出なレベルは大抵同じような性質を持つためこのような操作が行われることが多いですが、最終的にはCV（分割交差検証）を行ってこの操作が本当にプラスをもたらしてくれたのか確認しなければなりません。\n",
"\n",
"まずは一つ便利な機能関数を書いてみましょう。"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"# 学習用データとテストデータから'cols'を取り出しフリークエンシー（出現頻度）エンコーディングを行います。\n",
"def freq_encoding(cols, train_df, test_df):\n",
" # 新しいデータセットを以下のデータセットに保存します。\n",
" result_train_df=pd.DataFrame()\n",
" result_test_df=pd.DataFrame()\n",
"\n",
" for col in cols:\n",
" \n",
" # 学習用セットの中でフィーチャーの出現頻度をデータフレームとして読み込んでいきます。\n",
" col_freq=col+'_freq'\n",
" freq=train_df[col].value_counts()\n",
" freq=pd.DataFrame(freq)\n",
" freq.reset_index(inplace=True)\n",
" freq.columns=[[col,col_freq]]\n",
"\n",
" # 'freq'データフレームを学習用データと融合します。\n",
" temp_train_df=pd.merge(train_df[[col]], freq, how='left', on=col)\n",
" temp_train_df.drop([col], axis=1, inplace=True)\n",
"\n",
" # 'freq'データフレームをテストデータと融合します。\n",
" temp_test_df=pd.merge(test_df[[col]], freq, how='left', on=col)\n",
" temp_test_df.drop([col], axis=1, inplace=True)\n",
"\n",
" # 学習用セットで現れなかったレベルがテストセットに現れなかった場合、頻度を０と設定します。\n",
" temp_test_df.fillna(0, inplace=True)\n",
" temp_test_df[col_freq]=temp_test_df[col_freq].astype(np.int32)\n",
"\n",
" if result_train_df.shape[0]==0:\n",
" result_train_df=temp_train_df\n",
" result_test_df=temp_test_df\n",
" else:\n",
" result_train_df=pd.concat([result_train_df, temp_train_df],axis=1)\n",
" result_test_df=pd.concat([result_test_df, temp_test_df],axis=1)\n",
" \n",
" return result_train_df, result_test_df\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"この関数を私たちのカテゴリカルフィーチャに対して実行してみましょう。"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"cat_cols=['ps_ind_02_cat','ps_car_04_cat', 'ps_car_09_cat',\n",
" 'ps_ind_05_cat', 'ps_car_01_cat', 'ps_car_11_cat']\n",
"\n",
"# 学習とテストデータセットのフリークエンシーフィーチャのためのデータフレームの作成\n",
"train_freq, test_freq=freq_encoding(cat_cols, train, test)\n",
"\n",
"# 元の学習とテストデータセットに結合させます。\n",
"train=pd.concat([train, train_freq], axis=1)\n",
"test=pd.concat([test,test_freq], axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"フリークエンシー・フィーチャと元のフィーチャを比べてみましょう。"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style>\n",
" .dataframe thead tr:only-child th {\n",
" text-align: right;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ps_ind_02_cat</th>\n",
" <th>ps_ind_02_cat_freq</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2</td>\n",
" <td>123573</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1</td>\n",
" <td>431859</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4</td>\n",
" <td>11378</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" <td>431859</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2</td>\n",
" <td>123573</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ps_ind_02_cat ps_ind_02_cat_freq\n",
"0 2 123573\n",
"1 1 431859\n",
"2 4 11378\n",
"3 1 431859\n",
"4 2 123573"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train[train.columns[train.columns.str.contains('ps_ind_02_cat')]].head(5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ps_ind_02_catの出現数が別のレベルに割り当てられました。\n",
"\n",
"変換後は元のフィーチャを取り除きますが、まずはその前にバイナリエンコーディングを行っておきましょう。"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.2 バイナリ・エンコーディング（変換）\n",
"\n",
"カテゴリカルエンコーディングの中でもone-hot-codingという方法が頻繁に使われます。各レベルに0ベクトルを割り当て、レベルnのn列目には1を割り当てます。例えば先ほど取り上げた運転タイプFWD,RWD,4WDの場合、FWDは001、RWDは010、4WDは001と表されます。\n",
"\n",
"データを0,1,2と順序付けたくないときに便利な方法ですが、新たに次元の問題が出てきます。もともとレベルがn個あった列をn個のダミー変数列で表しました。特徴量が３つのように少ない場合は問題ありませんが、ps_car_11_catのようにレベル数が多い場合もともと50列しかなかったデータセットに100以上の列を足してしまいます。これを３倍以上大きなデータセットで行なってしまうと[「次元の呪い」](https://ja.wikipedia.org/wiki/%E6%AC%A1%E5%85%83%E3%81%AE%E5%91%AA%E3%81%84)と呼ばれる問題に陥ってしまいます。\n",
"\n",
"KDnuggetsの記事[*\"Beyond One-Hot\"*](https://www.kdnuggets.com/2015/12/beyond-one-hot-exploration-categorical-variables.html)で上手にまとめられていますが、バイナリエンコーディングはダミー変数列の数を減らし、数値型変換では避けられない順序付けを無くすことができます。カテゴリカルフィーチャから既に数値型に変換されたレベルをバイナリ表記で表すということがポイントです。従って、レベルが４つ（0,1,2,3）あるフィーチャは、0は01、1は01、2は10、3は11、と変換されます。\n",
"\n",
"実際にカテゴリカル変数をバイナリに変換する関数を作成してみましょう。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# カテゴリカル変数をバイナリに変換する関数を作成します。\n",
"# 学習用セットとテストセット、エンコードされるフィーチャーをとり、\n",
"# 入力フィーチャをバイナリー表示に変換されたデータセットが２つ返ってきます。\n",
"# この関数は符号化されるフィーチャが、nをフィーチャのレベル数としたとき\n",
"# 既に0からn-1の範囲で数値型に変換されていることを仮定しています。\n",
"\n",
"def binary_encoding(train_df, test_df, feat):\n",
" # 数値型変換で使用された最大値を計算。\n",
" train_feat_max = train_df[feat].max()\n",
" test_feat_max = test_df[feat].max()\n",
" if train_feat_max > test_feat_max:\n",
" feat_max = train_feat_max\n",
" else:\n",
" feat_max = test_feat_max\n",
" \n",
" # 欠損値にはfeat_max+1を使います。\n",
" train_df.loc[train_df[feat] == -1, feat] = feat_max + 1\n",
" test_df.loc[test_df[feat] == -1, feat] = feat_max + 1\n",
" \n",
" # 有り得るすべてのフィーチャの集合体を作成します。\n",
" union_val = np.union1d(train_df[feat].unique(), test_df[feat].unique())\n",
"\n",
" # フィーチャから小数点表示で最大値を抜き出します。\n",
" max_dec = union_val.max()\n",
" \n",
" # max_devをバイナリ表示するのに必要な桁数を計算します。\n",
" max_bin_len = len(\"{0:b}\".format(max_dec))\n",
" index = np.arange(len(union_val))\n",
" columns = list([feat])\n",
" \n",
" # フィーチャ全てのレベルを取得するのにバイナリ変換フィーチャ用のデータフレームを作成。\n",
" bin_df = pd.DataFrame(index=index, columns=columns)\n",
" bin_df[feat] = union_val\n",
" \n",
" # フィーチャの各レベルのバイナリ表示を取得。 \n",
" feat_bin = bin_df[feat].apply(lambda x: \"{0:b}\".format(x).zfill(max_bin_len))\n",
" \n",
" # バイナリ表示を異なる桁数に分割する。\n",
" splitted = feat_bin.apply(lambda x: pd.Series(list(x)).astype(np.uint8))\n",
" splitted.columns = [feat + '_bin_' + str(x) for x in splitted.columns]\n",
" bin_df = bin_df.join(splitted)\n",
" \n",
" # バイナリ変換フィーチャ用のデータフレームを学習用セットとテストセットで結合させ、完成です！ \n",
" train_df = pd.merge(train_df, bin_df, how='left', on=[feat])\n",
" test_df = pd.merge(test_df, bin_df, how='left', on=[feat])\n",
" return train_df, test_df\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"バイナリ変換関数を実行してみましょう。"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"cat_cols=['ps_ind_02_cat','ps_car_04_cat', 'ps_car_09_cat',\n",
" 'ps_ind_05_cat', 'ps_car_01_cat']\n",
"\n",
"train, test=binary_encoding(train, test, 'ps_ind_02_cat')\n",
"train, test=binary_encoding(train, test, 'ps_car_04_cat')\n",
"train, test=binary_encoding(train, test, 'ps_car_09_cat')\n",
"train, test=binary_encoding(train, test, 'ps_ind_05_cat')\n",
"train, test=binary_encoding(train, test, 'ps_car_01_cat')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ps_ind_02_catが含まれている列を全て見てみるとバイナリフィーチャが与えられていることがわかります。"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style>\n",
" .dataframe thead tr:only-child th {\n",
" text-align: right;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ps_ind_02_cat</th>\n",
" <th>ps_ind_02_cat_freq</th>\n",
" <th>ps_ind_02_cat_bin_0_x</th>\n",
" <th>ps_ind_02_cat_bin_1_x</th>\n",
" <th>ps_ind_02_cat_bin_2_x</th>\n",
" <th>ps_ind_02_cat_bin_0_y</th>\n",
" <th>ps_ind_02_cat_bin_1_y</th>\n",
" <th>ps_ind_02_cat_bin_2_y</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2</td>\n",
" <td>123573</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1</td>\n",
" <td>431859</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4</td>\n",
" <td>11378</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" <td>431859</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2</td>\n",
" <td>123573</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ps_ind_02_cat ps_ind_02_cat_freq ps_ind_02_cat_bin_0_x \\\n",
"0 2 123573 0 \n",
"1 1 431859 0 \n",
"2 4 11378 1 \n",
"3 1 431859 0 \n",
"4 2 123573 0 \n",
"\n",
" ps_ind_02_cat_bin_1_x ps_ind_02_cat_bin_2_x ps_ind_02_cat_bin_0_y \\\n",
"0 1 0 0 \n",
"1 0 1 0 \n",
"2 0 0 1 \n",
"3 0 1 0 \n",
"4 1 0 0 \n",
"\n",
" ps_ind_02_cat_bin_1_y ps_ind_02_cat_bin_2_y \n",
"0 1 0 \n",
"1 0 1 \n",
"2 0 0 \n",
"3 0 1 \n",
"4 1 0 "
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train[train.columns[train.columns.str.contains('ps_ind_02_cat')]].head(5)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style>\n",
" .dataframe thead tr:only-child th {\n",
" text-align: right;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ps_car_11_cat</th>\n",
" <th>ps_car_11_cat_freq</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>12</td>\n",
" <td>7246</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>19</td>\n",
" <td>5097</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>60</td>\n",
" <td>7992</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>104</td>\n",
" <td>85083</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>82</td>\n",
" <td>10470</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ps_car_11_cat ps_car_11_cat_freq\n",
"0 12 7246\n",
"1 19 5097\n",
"2 60 7992\n",
"3 104 85083\n",
"4 82 10470"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train[train.columns[train.columns.str.contains('ps_car_11_cat')]].head(5)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"注意深く見てみるとps_car_11_catにはバイナリ変換が行われなかったことがわかります。このカテゴリカルフィーチャには100以上のレベルがあり、実際バイナリ変換を行うと新たに40列以上作成されます。これはone-hotエンコーディングと比べれば小さな削減ではありますが、このデータセットを50列以上から90列以上へと80%拡大させてしまいます。より列数の多い大きなデータセットを扱うときに行っても良いですが、この場合はノイズを増やしてしまうだけなのであまり有利ではありません。\n",
"\n",
"ではこの場合どうしたら良いのでしょうか。[*Triskelion*](https://www.kaggle.com/triskelion)さんが書かれた[*feature engineering*](https://www.slideshare.net/HJvanVeen/feature-engineering-72376750)で取りあげられているトリックに挑戦してみて下さい。k-foldを使ったターゲット・エンコーディングも案ですが、今回ここでは省略させていただき、早速モデリングの方へ進みましょう。"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. モデル作成のパイプライン\n",
"\n",
"これまで準備してきた学習およびテストデータを使ってモデルを作成していきましょう。その前にデータセットをもう少し小さくしてみましょう。 [*Heads or Tails*](https://www.kaggle.com/headsortails) が自身のカーネル [*interactive EDA*](https://www.kaggle.com/headsortails/steering-wheel-of-fortune-porto-seguro-eda)でも取り上げてくれたように、calが含まれるフィーチャはあまり便利ではないようなので全て取り除いておきます。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"col_to_drop = train.columns[train.columns.str.startswith('ps_calc_')]\n",
"train.drop(col_to_drop, axis=1, inplace=True) \n",
"test.drop(col_to_drop, axis=1, inplace=True) "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"既にフリークエンシー及びバイナリエンコーディングを行なったため、もとのフィーチャもここで取り除いておきます。ps_car_11_catだけ扱いが違っていたため残しておきます。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cat_cols=['ps_ind_02_cat','ps_car_04_cat', 'ps_car_09_cat', 'ps_ind_05_cat', 'ps_car_01_cat']\n",
"train.drop(cat_cols, axis=1, inplace=True) \n",
"test.drop(cat_cols, axis=1, inplace=True) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ローカルの学習・テストデータを用意します。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"localtrain, localval=train_test_split(train, test_size=0.25, random_state=2017)\n",
"\n",
"drop_cols=['id','target']\n",
"y_localtrain=localtrain['target']\n",
"x_localtrain=localtrain.drop(drop_cols, axis=1)\n",
"\n",
"y_localval=localval['target']\n",
"x_localval=localval.drop(drop_cols, axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"続いて簡単な機械学習モデルを作成し私たちのデータに対して予測を行なっていきます。ランダムフォレストとロジスティック回帰を使ってみましょう。\n",
"\n",
"## 3.1 ランダムフォレストのモデル作成と予測 \n",
"SklearnのランダムフォレストAPIを使って学習を行い、ローカルテストデータに対して予測を行なっていきます。"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Start training...\n",
"it takes 25.005 seconds to train and predict\n"
]
}
],
"source": [
"print('Start training...')\n",
"start_time=time.time()\n",
"\n",
"rf=RandomForestClassifier(n_estimators=250, n_jobs=6, min_samples_split=5, max_depth=7,\n",
" criterion='gini', random_state=0)\n",
"\n",
"rf.fit(x_localtrain, y_localtrain)\n",
"rf_valprediction=rf.predict_proba(x_localval)[:,1]\n",
"\n",
"end_time = time.time()\n",
"print(\"it takes %.3f seconds to train and predict\" % (end_time - start_time))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[*競技フォーラム*](https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/40368)でも多数取り上げられていますが、AUCを簡単にジニ係数に変換する方法があります。ウィキペディアの[*ジニ係数*](https://ja.wikipedia.org/wiki/%E3%82%B8%E3%83%8B%E4%BF%82%E6%95%B0)でも詳しくまとめられています。"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Random Forest Validation AUC is 0.628736\n",
"Random Forest Validation Normalised Gini Coefficient is 0.257471\n"
]
}
],
"source": [
"rf_val_auc=roc_auc_score(y_localval, rf_valprediction)\n",
"rf_val_gininorm=2*rf_val_auc-1\n",
"\n",
"print('Random Forest Validation AUC is {:.6f}'.format(rf_val_auc))\n",
"print('Random Forest Validation Normalised Gini Coefficient is {:.6f}'.format(rf_val_gininorm))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"テストデータを使って予測を行いましょう。"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"x_test=test.drop(['id'], axis=1)\n",
"y_testprediction=rf.predict_proba(x_test)[:,1]\n",
"\n",
"rf_submission=sample_submission.copy()\n",
"rf_submission['target']=y_testprediction\n",
"rf_submission.to_csv('rf_submission.csv', compression='gzip', index=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3.2 ランダムフォレストのモデル作成と予測\n",
"\n",
"Sklearnを使えばロジスティック回帰の学習と予測はランダムフォレストと似ています。ただしこの場合、まずはフィーチャスケーリング(feature scaling)を行ない、スケーリングされた特徴量を使って学習と予測を行わなければなりません。"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Start training...\n",
"it takes 7.414 seconds to train and predict\n"
]
}
],
"source": [
"scaler = StandardScaler().fit(x_localtrain.values)\n",
"x_localtrain_scaled = scaler.transform(x_localtrain)\n",
"x_localval_scaled = scaler.transform(x_localval)\n",
"x_test_scaled = scaler.transform(x_test)\n",
"\n",
"print('Start training...')\n",
"start_time=time.time()\n",
"\n",
"logit=LogisticRegression()\n",
"logit.fit(x_localtrain_scaled, y_localtrain)\n",
"logit_valprediction=logit.predict_proba(x_localval_scaled)[:,1]\n",
"\n",
"end_time = time.time()\n",
"print(\"it takes %.3f seconds to train and predict\" % (end_time - start_time))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ランダムフォレストと同様、予測を行います。"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Logistic Regression Validation AUC is 0.627760\n",
"Logistic Regression Validation Normalised Gini Coefficient is 0.255521\n"
]
}
],
"source": [
"logit_val_auc=roc_auc_score(y_localval, logit_valprediction)\n",
"logit_val_gininorm=2*logit_val_auc-1\n",
"\n",
"print('Logistic Regression Validation AUC is {:.6f}'.format(logit_val_auc))\n",
"print('Logistic Regression Validation Normalised Gini Coefficient is {:.6f}'.format(logit_val_gininorm))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ロジック回帰を使って予測を行なってみましょう。"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"y_testprediction=logit.predict_proba(x_test_scaled)[:,1]\n",
"\n",
"logit_submission=sample_submission.copy()\n",
"logit_submission['target']=y_testprediction\n",
"logit_submission.to_csv('logit_submission.csv', compression='gzip', index=False)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment