Skip to content

Instantly share code, notes, and snippets.

@Inquisitive-Geek

Inquisitive-Geek/Outbrain_click_analysis.ipynb

Created January 24, 2017 02:57
Show Gist options
  • Save Inquisitive-Geek/9afb2679d52cb0ec36b61f5e17d662b8 to your computer and use it in GitHub Desktop.
Save Inquisitive-Geek/9afb2679d52cb0ec36b61f5e17d662b8 to your computer and use it in GitHub Desktop.
Download ZIP
A Python Notebook with explains how Outbrain data was loaded into Spark
Display the source blob
Display the rendered blob
Raw
Outbrain_click_analysis.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As part of Kaggle competition 'Outbrain Click Prediction', this analysis was attempted. It turned out to be unsuccessful as I was not able to get the predictive model up and running in a PySpark cluster. Also, if I run this code any more, all my free credits in the Google Cloud Cluster will be used up. That said, I learnt how to start a Google cluster and run an IPython notebook in it. I also learnt how to install other packages on a cluster straight from an IPython notebook. I obviously don't take credit for any of it but I guess it will be useful for me and for others when they want to use it. I know they are trivial things but the geek in me is happy."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing Packages"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, as shown below, a pip install pandas won't work. Rather use the statement os.system(\"sudo apt-get install python-pandas -y\") and you'll not be facing a rough time like me figuring out how to install packages like numpy, pandas and so on. Also, I noticed that for some reason I had to append python- in front of the package name. It could be a way to tell the cluster that a Python package is being installed."
]
},
{
"cell_type": "code",
"execution_count": 204,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from pyspark.sql.types import *\n",
"import pyspark.sql.functions as F\n",
"import sys\n",
"\n",
"sys.path.append('/usr/lib/python2.7/dist-packages')\n",
"\n",
"os.system(\"sudo apt-get install python-pandas -y\")\n",
"os.system(\"sudo apt-get install python-numpy -y\")\n",
"os.system(\"sudo apt-get install python-scipy -y\")\n",
"os.system(\"sudo apt-get install python-sklearn -y\")\n",
"\n",
"\n",
"import pandas\n",
"import numpy\n",
"import scipy\n",
"import sklearn\n",
"import pyspark.mllib as ml"
]
},
{
"cell_type": "code",
"execution_count": 210,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn.ensemble import GradientBoostingClassifier"
]
},
{
"cell_type": "code",
"execution_count": 214,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from pyspark.ml.classification import RandomForestClassifier"
]
},
{
"cell_type": "code",
"execution_count": 222,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from pyspark.ml import Pipeline\n",
"from pyspark.ml.classification import RandomForestClassifier as RF\n",
"from pyspark.ml.feature import StringIndexer, VectorIndexer, VectorAssembler, SQLTransformer\n",
"from pyspark.ml.evaluation import MulticlassClassificationEvaluator, BinaryClassificationEvaluator\n",
"from pyspark.ml.tuning import CrossValidator, ParamGridBuilder"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data load"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following code first creates the necessary structures for the Spark data frames and then code is written to load the files stored in a Google File Store folder. Here, dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us represents the name of the File Store which contains all the input files. The code for loading data seems pretty self-explanatory."
]
},
{
"cell_type": "code",
"execution_count": 114,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"page_views_schema = StructType(\n",
" [StructField(\"uuid\", StringType(), True),\n",
" StructField(\"document_id\", IntegerType(), True),\n",
" StructField(\"timestamp\", IntegerType(), True),\n",
" StructField(\"platform\", IntegerType(), True),\n",
" StructField(\"geo_location\", StringType(), True),\n",
" StructField(\"traffic_source\", IntegerType(), True)]\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 115,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"page_views_df = spark.read.schema(page_views_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"page_views_df = page_views_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/page_views.csv\")"
]
},
{
"cell_type": "code",
"execution_count": 116,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Row(uuid=u'1fd5f051fba643', document_id=120, timestamp=31905835, platform=1, geo_location=u'RS', traffic_source=2)"
]
},
"execution_count": 116,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"page_views_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 117,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"document_categories_schema = StructType(\n",
" [StructField(\"document_id\", IntegerType(), True),\n",
" StructField(\"category_id\", IntegerType(), True),\n",
" StructField(\"confidence_level_cat\", DoubleType(), True)]\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 118,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"document_categories_df = spark.read.schema(document_categories_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"document_categories_df = document_categories_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/documents_categories.csv\") "
]
},
{
"cell_type": "code",
"execution_count": 119,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Row(document_id=1595802, category_id=1611, confidence_level_cat=0.92)"
]
},
"execution_count": 119,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_categories_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 120,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"document_meta_schema = StructType(\n",
" [StructField(\"document_id\", IntegerType(), True),\n",
" StructField(\"source_id\", IntegerType(), True),\n",
" StructField(\"publisher_id\", IntegerType(), True),\n",
" StructField(\"publish_time\", TimestampType(), True)\n",
" ])"
]
},
{
"cell_type": "code",
"execution_count": 121,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"document_meta_df = spark.read.schema(document_meta_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"document_meta_df = document_meta_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/documents_meta.csv\") "
]
},
{
"cell_type": "code",
"execution_count": 122,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Row(document_id=1595802, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 6, 5, 0, 0))"
]
},
"execution_count": 122,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_meta_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 123,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"clicks_train_schema = StructType(\n",
" [StructField(\"display_id\", IntegerType(), True),\n",
" StructField(\"ad_id\", IntegerType(), True),\n",
" StructField(\"clicked\", IntegerType(), True)]\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 124,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"clicks_train_df = spark.read.schema(clicks_train_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"clicks_train_df = clicks_train_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/clicks_train.csv\") "
]
},
{
"cell_type": "code",
"execution_count": 248,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(display_id=1, ad_id=42337, clicked=0),\n",
" Row(display_id=1, ad_id=139684, clicked=0),\n",
" Row(display_id=1, ad_id=144739, clicked=1),\n",
" Row(display_id=1, ad_id=156824, clicked=0),\n",
" Row(display_id=1, ad_id=279295, clicked=0),\n",
" Row(display_id=1, ad_id=296965, clicked=0),\n",
" Row(display_id=2, ad_id=125211, clicked=0),\n",
" Row(display_id=2, ad_id=156535, clicked=0),\n",
" Row(display_id=2, ad_id=169564, clicked=0),\n",
" Row(display_id=2, ad_id=308455, clicked=1),\n",
" Row(display_id=3, ad_id=71547, clicked=0),\n",
" Row(display_id=3, ad_id=95814, clicked=0),\n",
" Row(display_id=3, ad_id=152141, clicked=0),\n",
" Row(display_id=3, ad_id=183846, clicked=0),\n",
" Row(display_id=3, ad_id=228657, clicked=1)]"
]
},
"execution_count": 248,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clicks_train_df.head(15)"
]
},
{
"cell_type": "code",
"execution_count": 126,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"clicks_test_schema = StructType(\n",
" [StructField(\"display_id\", IntegerType(), True),\n",
" StructField(\"ad_id\", IntegerType(), True)]\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 127,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"clicks_test_df = spark.read.schema(clicks_test_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"clicks_test_df = clicks_test_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/clicks_test.csv\") "
]
},
{
"cell_type": "code",
"execution_count": 128,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(display_id=16874594, ad_id=66758),\n",
" Row(display_id=16874594, ad_id=150083),\n",
" Row(display_id=16874594, ad_id=162754),\n",
" Row(display_id=16874594, ad_id=170392),\n",
" Row(display_id=16874594, ad_id=172888),\n",
" Row(display_id=16874594, ad_id=180797),\n",
" Row(display_id=16874595, ad_id=8846),\n",
" Row(display_id=16874595, ad_id=30609),\n",
" Row(display_id=16874595, ad_id=143982),\n",
" Row(display_id=16874596, ad_id=11430),\n",
" Row(display_id=16874596, ad_id=57197),\n",
" Row(display_id=16874596, ad_id=132820),\n",
" Row(display_id=16874596, ad_id=153260),\n",
" Row(display_id=16874596, ad_id=173005),\n",
" Row(display_id=16874596, ad_id=288385)]"
]
},
"execution_count": 128,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clicks_test_df.head(15)"
]
},
{
"cell_type": "code",
"execution_count": 195,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"events_schema = StructType(\n",
" [StructField(\"display_id\", IntegerType(), True),\n",
" StructField(\"uuid\", StringType(), True),\n",
" StructField(\"document_id_e\", IntegerType(), True),\n",
" StructField(\"timestamp\", IntegerType(), True),\n",
" StructField(\"platform\", IntegerType(), True),\n",
" StructField(\"geo_location\", StringType(), True)]\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 196,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"events_df = spark.read.schema(events_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"events_df = events_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/events.csv\") "
]
},
{
"cell_type": "code",
"execution_count": 197,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(display_id=1, uuid=u'cb8c55702adb93', document_id_e=379743, timestamp=61, platform=3, geo_location=u'US>SC>519'),\n",
" Row(display_id=2, uuid=u'79a85fa78311b9', document_id_e=1794259, timestamp=81, platform=2, geo_location=u'US>CA>807'),\n",
" Row(display_id=3, uuid=u'822932ce3d8757', document_id_e=1179111, timestamp=182, platform=2, geo_location=u'US>MI>505'),\n",
" Row(display_id=4, uuid=u'85281d0a49f7ac', document_id_e=1777797, timestamp=234, platform=2, geo_location=u'US>WV>564'),\n",
" Row(display_id=5, uuid=u'8d0daef4bf5b56', document_id_e=252458, timestamp=338, platform=2, geo_location=u'SG>00'),\n",
" Row(display_id=6, uuid=u'7765b4faae4ad4', document_id_e=1773517, timestamp=395, platform=3, geo_location=u'US>OH>510'),\n",
" Row(display_id=7, uuid=u'2cc3f6457d16da', document_id_e=1149661, timestamp=602, platform=2, geo_location=u'US>MT>762'),\n",
" Row(display_id=8, uuid=u'166fc654d73c98', document_id_e=1330329, timestamp=638, platform=2, geo_location=u'US>PA>566'),\n",
" Row(display_id=9, uuid=u'9dddccf70f6067', document_id_e=1772126, timestamp=667, platform=1, geo_location=u'US>FL>528'),\n",
" Row(display_id=10, uuid=u'b09a0e92aa4d17', document_id_e=157455, timestamp=693, platform=1, geo_location=u'US')]"
]
},
"execution_count": 197,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"events_df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": 132,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"document_entities_schema = StructType(\n",
" [StructField(\"document_id\", IntegerType(), True),\n",
" StructField(\"entity_id\", StringType(), True),\n",
" StructField(\"confidence_level_ent\", DoubleType(), True)]\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 133,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"document_entities_df = spark.read.schema(document_entities_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"document_entities_df = document_entities_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/documents_entities.csv\")"
]
},
{
"cell_type": "code",
"execution_count": 134,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Row(document_id=1524246, entity_id=u'f9eec25663db4cd83183f5c805186f16', confidence_level_ent=0.672865314504701)"
]
},
"execution_count": 134,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_entities_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 135,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"document_topics_schema = StructType(\n",
" [StructField(\"document_id\", IntegerType(), True),\n",
" StructField(\"topic_id\", IntegerType(), True),\n",
" StructField(\"confidence_level_top\", DoubleType(), True)]\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 136,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"document_topics_df = spark.read.schema(document_topics_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"document_topics_df = document_topics_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/documents_topics.csv\")"
]
},
{
"cell_type": "code",
"execution_count": 137,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(document_id=1595802, topic_id=140, confidence_level_top=0.0731131601068925),\n",
" Row(document_id=1595802, topic_id=16, confidence_level_top=0.0594164867373976),\n",
" Row(document_id=1595802, topic_id=143, confidence_level_top=0.0454207537554526),\n",
" Row(document_id=1595802, topic_id=170, confidence_level_top=0.0388674285182961),\n",
" Row(document_id=1524246, topic_id=113, confidence_level_top=0.196450402209685),\n",
" Row(document_id=1524246, topic_id=260, confidence_level_top=0.14287816210395),\n",
" Row(document_id=1524246, topic_id=92, confidence_level_top=0.0331591277639303),\n",
" Row(document_id=1524246, topic_id=168, confidence_level_top=0.0140903418233461),\n",
" Row(document_id=1524246, topic_id=54, confidence_level_top=0.00878222028049753),\n",
" Row(document_id=1524246, topic_id=207, confidence_level_top=0.00828237207366929)]"
]
},
"execution_count": 137,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_topics_df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": 138,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(document_id=1524246, entity_id=u'f9eec25663db4cd83183f5c805186f16', confidence_level_ent=0.672865314504701),\n",
" Row(document_id=1524246, entity_id=u'55ebcfbdaff1d6f60b3907151f38527a', confidence_level_ent=0.399113728441297),\n",
" Row(document_id=1524246, entity_id=u'839907a972930b17b125eb0247898412', confidence_level_ent=0.392095749652966),\n",
" Row(document_id=1524246, entity_id=u'04d8f9a1ad48f126d5806a9236872604', confidence_level_ent=0.213996376305138),\n",
" Row(document_id=1617787, entity_id=u'612a1d17685a498aff4f036c1ee02c16', confidence_level_ent=0.386192829940441),\n",
" Row(document_id=1617787, entity_id=u'fb8c6cb0879e0de876298177eb1d3fcc', confidence_level_ent=0.364116465983032),\n",
" Row(document_id=1617787, entity_id=u'793c6a6cf386edb820600d49df045475', confidence_level_ent=0.349168245516694),\n",
" Row(document_id=1617787, entity_id=u'b525b84d5ed52a34565b8fb870555abe', confidence_level_ent=0.287004519610819),\n",
" Row(document_id=1617787, entity_id=u'758cb9cb3014607cb4a691cbd485cc94', confidence_level_ent=0.237956988738312),\n",
" Row(document_id=1617787, entity_id=u'd523aaba6d3916f8b7039fcce0f29639', confidence_level_ent=0.235798527030791)]"
]
},
"execution_count": 138,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_entities_df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": 139,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(document_id=1595802, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 6, 5, 0, 0)),\n",
" Row(document_id=1524246, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 5, 26, 11, 0)),\n",
" Row(document_id=1617787, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 5, 27, 0, 0)),\n",
" Row(document_id=1615583, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 6, 7, 0, 0)),\n",
" Row(document_id=1615460, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 6, 20, 0, 0)),\n",
" Row(document_id=1615354, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 6, 10, 0, 0)),\n",
" Row(document_id=1614611, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 6, 5, 13, 0)),\n",
" Row(document_id=1614235, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 6, 9, 0, 0)),\n",
" Row(document_id=1614225, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 6, 9, 0, 0)),\n",
" Row(document_id=1488264, source_id=1, publisher_id=603, publish_time=datetime.datetime(2016, 5, 23, 13, 0))]"
]
},
"execution_count": 139,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_meta_df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": 140,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(document_id=1595802, category_id=1611, confidence_level_cat=0.92),\n",
" Row(document_id=1595802, category_id=1610, confidence_level_cat=0.07),\n",
" Row(document_id=1524246, category_id=1807, confidence_level_cat=0.92),\n",
" Row(document_id=1524246, category_id=1608, confidence_level_cat=0.07),\n",
" Row(document_id=1617787, category_id=1807, confidence_level_cat=0.92),\n",
" Row(document_id=1617787, category_id=1608, confidence_level_cat=0.07),\n",
" Row(document_id=1615583, category_id=1305, confidence_level_cat=0.92),\n",
" Row(document_id=1615583, category_id=1806, confidence_level_cat=0.07),\n",
" Row(document_id=1615460, category_id=1613, confidence_level_cat=0.540646372),\n",
" Row(document_id=1615460, category_id=1603, confidence_level_cat=0.041136137)]"
]
},
"execution_count": 140,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_categories_df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": 141,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"page_views_sample_df = spark.read.schema(page_views_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"page_views_sample_df = page_views_sample_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/page_views_sample.csv\") "
]
},
{
"cell_type": "code",
"execution_count": 142,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Row(uuid=u'1fd5f051fba643', document_id=120, timestamp=31905835, platform=1, geo_location=u'RS', traffic_source=2)"
]
},
"execution_count": 142,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"page_views_sample_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 143,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"sample_submission_schema = StructType(\n",
" [StructField(\"display_id\", IntegerType(), True),\n",
" StructField(\"ad_id\", StringType(), True)]\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 144,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"sample_submission_df = spark.read.schema(sample_submission_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"sample_submission_df = sample_submission_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/sample_submission.csv\") "
]
},
{
"cell_type": "code",
"execution_count": 145,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(display_id=16874594, ad_id=u'66758 150083 162754 170392 172888 180797'),\n",
" Row(display_id=16874595, ad_id=u'8846 30609 143982'),\n",
" Row(display_id=16874596, ad_id=u'11430 57197 132820 153260 173005 288385 289122 289915'),\n",
" Row(display_id=16874597, ad_id=u'137858 143981 155945 180965 182039 285834 305790 308836'),\n",
" Row(display_id=16874598, ad_id=u'67292 145937 250082 335632'),\n",
" Row(display_id=16874599, ad_id=u'91681 163776 173130 210516 213116 296295'),\n",
" Row(display_id=16874600, ad_id=u'2150 30682 57591 70529 114836 133050'),\n",
" Row(display_id=16874601, ad_id=u'14082 92003 118470 129490 140942 190713'),\n",
" Row(display_id=16874602, ad_id=u'131316 154918 268548 269017 281563'),\n",
" Row(display_id=16874603, ad_id=u'44460 140423 156050 163139')]"
]
},
"execution_count": 145,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sample_submission_df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": 146,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"promoted_content_schema = StructType(\n",
" [StructField(\"ad_id\", IntegerType(), True),\n",
" StructField(\"document_id\", IntegerType(), True),\n",
" StructField(\"campaign_id\", IntegerType(), True),\n",
" StructField(\"advertiser_id\", IntegerType(), True)]\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 147,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"promoted_content_df = spark.read.schema(promoted_content_schema).options(header='true', inferschema='false', nullValue='\\\\N')\n",
"promoted_content_df = promoted_content_df.csv(\"gs://dataproc-b10d4d38-fd55-4d88-8a9f-ba1154a55132-us/data/promoted_content.csv\")"
]
},
{
"cell_type": "code",
"execution_count": 148,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(ad_id=1, document_id=6614, campaign_id=1, advertiser_id=7),\n",
" Row(ad_id=2, document_id=471467, campaign_id=2, advertiser_id=7),\n",
" Row(ad_id=3, document_id=7692, campaign_id=3, advertiser_id=7),\n",
" Row(ad_id=4, document_id=471471, campaign_id=2, advertiser_id=7),\n",
" Row(ad_id=5, document_id=471472, campaign_id=2, advertiser_id=7),\n",
" Row(ad_id=6, document_id=12736, campaign_id=1, advertiser_id=7),\n",
" Row(ad_id=7, document_id=12808, campaign_id=1, advertiser_id=7),\n",
" Row(ad_id=8, document_id=471477, campaign_id=2, advertiser_id=7),\n",
" Row(ad_id=9, document_id=13379, campaign_id=1, advertiser_id=7),\n",
" Row(ad_id=10, document_id=13885, campaign_id=1, advertiser_id=7)]"
]
},
"execution_count": 148,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"promoted_content_df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": 149,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 149,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"promoted_content_df.select('ad_id').distinct().count() == promoted_content_df.count()\n",
"#In case you want the list of values for that column, do promoted_content_df.select('ad_id').distinct().rdd.map(lambda r: r[0]).collect()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data Merging"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, the clicks and the events data set are merged using a left join. After that, I successively merged the various files related to documents to create one big joined table. In the end, the tables related to clicks, events and documents are merged together for the final table which is to be used to build a model on."
]
},
{
"cell_type": "code",
"execution_count": 150,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"clicks_train_events_df = clicks_train_df.join(events_df, on=['display_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 151,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 151,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clicks_train_events_df.count() == clicks_train_df.count()"
]
},
{
"cell_type": "code",
"execution_count": 152,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"clicks_test_events_df = clicks_test_df.join(events_df, on=['display_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 153,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(display_id=148, ad_id=89351, clicked=1, uuid=u'9adce6a5363308', document_id=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, ad_id=152140, clicked=0, uuid=u'9adce6a5363308', document_id=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, ad_id=152656, clicked=0, uuid=u'9adce6a5363308', document_id=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, ad_id=326768, clicked=0, uuid=u'9adce6a5363308', document_id=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=463, ad_id=28270, clicked=0, uuid=u'85d999a1dd5483', document_id=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, ad_id=38813, clicked=1, uuid=u'85d999a1dd5483', document_id=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, ad_id=104911, clicked=0, uuid=u'85d999a1dd5483', document_id=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, ad_id=170148, clicked=0, uuid=u'85d999a1dd5483', document_id=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, ad_id=187292, clicked=0, uuid=u'85d999a1dd5483', document_id=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, ad_id=221740, clicked=0, uuid=u'85d999a1dd5483', document_id=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764')]"
]
},
"execution_count": 153,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clicks_train_events_df.head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The below code is just to validate if the join worked fine. "
]
},
{
"cell_type": "code",
"execution_count": 154,
"metadata": {
"collapsed": false,
"scrolled": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(display_id=1, ad_id=42337, clicked=0),\n",
" Row(display_id=1, ad_id=139684, clicked=0),\n",
" Row(display_id=1, ad_id=144739, clicked=1),\n",
" Row(display_id=1, ad_id=156824, clicked=0),\n",
" Row(display_id=1, ad_id=279295, clicked=0),\n",
" Row(display_id=1, ad_id=296965, clicked=0)]"
]
},
"execution_count": 154,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clicks_train_df[clicks_train_df['display_id'] == 1].head(10)"
]
},
{
"cell_type": "code",
"execution_count": 155,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(display_id=148, uuid=u'9adce6a5363308', document_id=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612')]"
]
},
"execution_count": 155,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"events_df[events_df['display_id'] == 148].head(10)"
]
},
{
"cell_type": "code",
"execution_count": 156,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(display_id=1, ad_id=42337, clicked=0, uuid=u'cb8c55702adb93', document_id=379743, timestamp=61, platform=3, geo_location=u'US>SC>519'),\n",
" Row(display_id=1, ad_id=139684, clicked=0, uuid=u'cb8c55702adb93', document_id=379743, timestamp=61, platform=3, geo_location=u'US>SC>519'),\n",
" Row(display_id=1, ad_id=144739, clicked=1, uuid=u'cb8c55702adb93', document_id=379743, timestamp=61, platform=3, geo_location=u'US>SC>519'),\n",
" Row(display_id=1, ad_id=156824, clicked=0, uuid=u'cb8c55702adb93', document_id=379743, timestamp=61, platform=3, geo_location=u'US>SC>519'),\n",
" Row(display_id=1, ad_id=279295, clicked=0, uuid=u'cb8c55702adb93', document_id=379743, timestamp=61, platform=3, geo_location=u'US>SC>519'),\n",
" Row(display_id=1, ad_id=296965, clicked=0, uuid=u'cb8c55702adb93', document_id=379743, timestamp=61, platform=3, geo_location=u'US>SC>519')]"
]
},
"execution_count": 156,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clicks_train_events_df[clicks_train_events_df['display_id'] == 1].head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now the above data frame is merged with the promoted_content data frame on the 'ad_id' field."
]
},
{
"cell_type": "code",
"execution_count": 157,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"clicks_train_e_promo_cont_df = clicks_train_events_df.join(promoted_content_df, on=['ad_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 158,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"clicks_test_e_promo_cont_df = clicks_test_events_df.join(promoted_content_df, on=['ad_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 159,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 159,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_meta_df.select('document_id').distinct().count() == document_meta_df.count()"
]
},
{
"cell_type": "code",
"execution_count": 160,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [
{
"ename": "AnalysisException",
"evalue": "u\"Reference 'document_id' is ambiguous, could be: document_id#826, document_id#908.;\"",
"output_type": "error",
"traceback": [
"\u001b[0;31m\u001b[0m",
"\u001b[0;31mAnalysisException\u001b[0mTraceback (most recent call last)",
"\u001b[0;32m<ipython-input-160-5ef69e334e94>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mclicks_train_e_p_d_meta_df\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclicks_train_e_promo_cont_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdocument_meta_df\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mon\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'document_id'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhow\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'left'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/usr/lib/spark/python/pyspark/sql/dataframe.pyc\u001b[0m in \u001b[0;36mjoin\u001b[0;34m(self, other, on, how)\u001b[0m\n\u001b[1;32m 636\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 637\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhow\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbasestring\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"how should be basestring\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 638\u001b[0;31m \u001b[0mjdf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mother\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jseq\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mon\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhow\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 639\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 640\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mon\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mColumn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"on should be Column or list of Column\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1131\u001b[0m \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1132\u001b[0m return_value = get_return_value(\n\u001b[0;32m-> 1133\u001b[0;31m answer, self.gateway_client, self.target_id, self.name)\n\u001b[0m\u001b[1;32m 1134\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1135\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtemp_arg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtemp_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/sql/utils.pyc\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m 67\u001b[0m e.java_exception.getStackTrace()))\n\u001b[1;32m 68\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'org.apache.spark.sql.AnalysisException: '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 69\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mAnalysisException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m': '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstackTrace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 70\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'org.apache.spark.sql.catalyst.analysis'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 71\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mAnalysisException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m': '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstackTrace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mAnalysisException\u001b[0m: u\"Reference 'document_id' is ambiguous, could be: document_id#826, document_id#908.;\""
]
}
],
"source": [
"clicks_train_e_p_d_meta_df = clicks_train_e_promo_cont_df.join(document_meta_df, on=['document_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"clicks_train_e_promo_cont_df.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"events_df.select('display_id').distinct().count() == events_df.count()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"clicks_train_df.select('display_id').distinct().count() == clicks_train_df.count()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since merging events and document info is giving an error, I'll first try to merge clicks and document information for finding out the information about what document and ad combinations lead to clicks."
]
},
{
"cell_type": "code",
"execution_count": 173,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"c_train_pr_cont_df = clicks_train_df.join(promoted_content_df, on=['ad_id'], how='left')\n",
"c_test_pr_cont_df = clicks_test_df.join(promoted_content_df, on=['ad_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 174,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 174,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clicks_train_df.select('ad_id').distinct().count() == clicks_train_df.count()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"This shows that an ad(ad_id) could be in multiple sets of recommendations."
]
},
{
"cell_type": "code",
"execution_count": 172,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(ad_id=148, display_id=9944564, clicked=0, document_id=471520, campaign_id=34, advertiser_id=19),\n",
" Row(ad_id=148, display_id=3195987, clicked=0, document_id=471520, campaign_id=34, advertiser_id=19),\n",
" Row(ad_id=148, display_id=3262691, clicked=0, document_id=471520, campaign_id=34, advertiser_id=19),\n",
" Row(ad_id=148, display_id=3267986, clicked=0, document_id=471520, campaign_id=34, advertiser_id=19),\n",
" Row(ad_id=148, display_id=3678937, clicked=0, document_id=471520, campaign_id=34, advertiser_id=19),\n",
" Row(ad_id=148, display_id=3783605, clicked=0, document_id=471520, campaign_id=34, advertiser_id=19),\n",
" Row(ad_id=148, display_id=4349019, clicked=0, document_id=471520, campaign_id=34, advertiser_id=19),\n",
" Row(ad_id=148, display_id=8869255, clicked=0, document_id=471520, campaign_id=34, advertiser_id=19),\n",
" Row(ad_id=148, display_id=8944408, clicked=0, document_id=471520, campaign_id=34, advertiser_id=19),\n",
" Row(ad_id=471, display_id=9643855, clicked=0, document_id=76819, campaign_id=59, advertiser_id=76)]"
]
},
"execution_count": 172,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c_train_pr_cont_df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": 162,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"document_m_t_df = document_meta_df.join(document_topics_df, on=['document_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 167,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=153, confidence_level_top=0.0752369712571181),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=140, confidence_level_top=0.0719832117961455),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=8, confidence_level_top=0.0610842713790318),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=172, confidence_level_top=0.0420603297928532),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=244, confidence_level_top=0.0397126495989423),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=179, confidence_level_top=0.0368485664530311),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=36, confidence_level_top=0.0307454540640941),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=2, confidence_level_top=0.0260491404387252),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=64, confidence_level_top=0.0160546425977903),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=10, confidence_level_top=0.0109180591249006),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=216, confidence_level_top=0.00837061271519606)]"
]
},
"execution_count": 167,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_m_t_df[document_m_t_df['document_id'] == 148].head(30)"
]
},
{
"cell_type": "code",
"execution_count": 164,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"document_m_t_e_df = document_m_t_df.join(document_entities_df, on=['document_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 168,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=153, confidence_level_top=0.0752369712571181, entity_id=u'e1c74838563ef5d205063b0d95afa414', confidence_level_ent=0.632025758366959),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=153, confidence_level_top=0.0752369712571181, entity_id=u'6fd68f102042c6554cb2592fae942264', confidence_level_ent=0.404909109697693),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=153, confidence_level_top=0.0752369712571181, entity_id=u'ae3de5466bfa10459eebcbe02ac9b3ee', confidence_level_ent=0.276364701967659),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=153, confidence_level_top=0.0752369712571181, entity_id=u'9da9595caa381755c9353ae7179f2117', confidence_level_ent=0.272666585584267),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=153, confidence_level_top=0.0752369712571181, entity_id=u'b973c2e55831fb4025003e0259aa820f', confidence_level_ent=0.266430219967596),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=153, confidence_level_top=0.0752369712571181, entity_id=u'6eb92e281e46d463ce80317efd785d68', confidence_level_ent=0.259955185083912),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=153, confidence_level_top=0.0752369712571181, entity_id=u'c323569535ca4c3d2ce474f4d825cc80', confidence_level_ent=0.221496101629345),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=153, confidence_level_top=0.0752369712571181, entity_id=u'daf2f4c9cd8dbf10482f06200e613939', confidence_level_ent=0.212876549025939),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=140, confidence_level_top=0.0719832117961455, entity_id=u'e1c74838563ef5d205063b0d95afa414', confidence_level_ent=0.632025758366959),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=140, confidence_level_top=0.0719832117961455, entity_id=u'6fd68f102042c6554cb2592fae942264', confidence_level_ent=0.404909109697693),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=140, confidence_level_top=0.0719832117961455, entity_id=u'ae3de5466bfa10459eebcbe02ac9b3ee', confidence_level_ent=0.276364701967659),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=140, confidence_level_top=0.0719832117961455, entity_id=u'9da9595caa381755c9353ae7179f2117', confidence_level_ent=0.272666585584267),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=140, confidence_level_top=0.0719832117961455, entity_id=u'b973c2e55831fb4025003e0259aa820f', confidence_level_ent=0.266430219967596),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=140, confidence_level_top=0.0719832117961455, entity_id=u'6eb92e281e46d463ce80317efd785d68', confidence_level_ent=0.259955185083912),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=140, confidence_level_top=0.0719832117961455, entity_id=u'c323569535ca4c3d2ce474f4d825cc80', confidence_level_ent=0.221496101629345),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=140, confidence_level_top=0.0719832117961455, entity_id=u'daf2f4c9cd8dbf10482f06200e613939', confidence_level_ent=0.212876549025939),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=8, confidence_level_top=0.0610842713790318, entity_id=u'e1c74838563ef5d205063b0d95afa414', confidence_level_ent=0.632025758366959),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=8, confidence_level_top=0.0610842713790318, entity_id=u'6fd68f102042c6554cb2592fae942264', confidence_level_ent=0.404909109697693),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=8, confidence_level_top=0.0610842713790318, entity_id=u'ae3de5466bfa10459eebcbe02ac9b3ee', confidence_level_ent=0.276364701967659),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=8, confidence_level_top=0.0610842713790318, entity_id=u'9da9595caa381755c9353ae7179f2117', confidence_level_ent=0.272666585584267),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=8, confidence_level_top=0.0610842713790318, entity_id=u'b973c2e55831fb4025003e0259aa820f', confidence_level_ent=0.266430219967596),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=8, confidence_level_top=0.0610842713790318, entity_id=u'6eb92e281e46d463ce80317efd785d68', confidence_level_ent=0.259955185083912),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=8, confidence_level_top=0.0610842713790318, entity_id=u'c323569535ca4c3d2ce474f4d825cc80', confidence_level_ent=0.221496101629345),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=8, confidence_level_top=0.0610842713790318, entity_id=u'daf2f4c9cd8dbf10482f06200e613939', confidence_level_ent=0.212876549025939),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=172, confidence_level_top=0.0420603297928532, entity_id=u'e1c74838563ef5d205063b0d95afa414', confidence_level_ent=0.632025758366959),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=172, confidence_level_top=0.0420603297928532, entity_id=u'6fd68f102042c6554cb2592fae942264', confidence_level_ent=0.404909109697693),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=172, confidence_level_top=0.0420603297928532, entity_id=u'ae3de5466bfa10459eebcbe02ac9b3ee', confidence_level_ent=0.276364701967659),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=172, confidence_level_top=0.0420603297928532, entity_id=u'9da9595caa381755c9353ae7179f2117', confidence_level_ent=0.272666585584267),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=172, confidence_level_top=0.0420603297928532, entity_id=u'b973c2e55831fb4025003e0259aa820f', confidence_level_ent=0.266430219967596),\n",
" Row(document_id=148, source_id=1787, publisher_id=118, publish_time=datetime.datetime(2008, 7, 1, 0, 0), topic_id=172, confidence_level_top=0.0420603297928532, entity_id=u'6eb92e281e46d463ce80317efd785d68', confidence_level_ent=0.259955185083912)]"
]
},
"execution_count": 168,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_m_t_e_df[document_m_t_e_df['document_id'] == 148].head(30)"
]
},
{
"cell_type": "code",
"execution_count": 169,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"36728674"
]
},
"execution_count": 169,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_m_t_e_df.count()"
]
},
{
"cell_type": "code",
"execution_count": 170,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"document_m_t_e_c_df = document_m_t_e_df.join(document_categories_df, on=['document_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 171,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"72558106"
]
},
"execution_count": 171,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"document_m_t_e_c_df.count()"
]
},
{
"cell_type": "code",
"execution_count": 176,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"c_train_pr_doc_det_df = c_train_pr_cont_df.join(document_m_t_e_c_df, on=['document_id'], how='left')\n",
"c_test_pr_doc_det_df = c_test_pr_cont_df.join(document_m_t_e_c_df, on=['document_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 177,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[]"
]
},
"execution_count": 177,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c_train_pr_doc_det_df[c_train_pr_doc_det_df['document_id'] == 148].head(30)"
]
},
{
"cell_type": "code",
"execution_count": 180,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1482391843"
]
},
"execution_count": 180,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c_train_pr_doc_det_df.count()"
]
},
{
"cell_type": "code",
"execution_count": 182,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(document_id=29834, ad_id=45, display_id=15862091, clicked=0, campaign_id=26, advertiser_id=39, source_id=2586, publisher_id=278, publish_time=datetime.datetime(2016, 6, 28, 0, 0), topic_id=256, confidence_level_top=0.212099170435346, entity_id=None, confidence_level_ent=None, category_id=1513, confidence_level_cat=0.7712613),\n",
" Row(document_id=29834, ad_id=45, display_id=15862091, clicked=0, campaign_id=26, advertiser_id=39, source_id=2586, publisher_id=278, publish_time=datetime.datetime(2016, 6, 28, 0, 0), topic_id=256, confidence_level_top=0.212099170435346, entity_id=None, confidence_level_ent=None, category_id=1514, confidence_level_cat=0.058682925),\n",
" Row(document_id=29834, ad_id=45, display_id=15864222, clicked=0, campaign_id=26, advertiser_id=39, source_id=2586, publisher_id=278, publish_time=datetime.datetime(2016, 6, 28, 0, 0), topic_id=256, confidence_level_top=0.212099170435346, entity_id=None, confidence_level_ent=None, category_id=1513, confidence_level_cat=0.7712613),\n",
" Row(document_id=29834, ad_id=45, display_id=15864222, clicked=0, campaign_id=26, advertiser_id=39, source_id=2586, publisher_id=278, publish_time=datetime.datetime(2016, 6, 28, 0, 0), topic_id=256, confidence_level_top=0.212099170435346, entity_id=None, confidence_level_ent=None, category_id=1514, confidence_level_cat=0.058682925),\n",
" Row(document_id=29834, ad_id=45, display_id=15897944, clicked=0, campaign_id=26, advertiser_id=39, source_id=2586, publisher_id=278, publish_time=datetime.datetime(2016, 6, 28, 0, 0), topic_id=256, confidence_level_top=0.212099170435346, entity_id=None, confidence_level_ent=None, category_id=1513, confidence_level_cat=0.7712613),\n",
" Row(document_id=29834, ad_id=45, display_id=15897944, clicked=0, campaign_id=26, advertiser_id=39, source_id=2586, publisher_id=278, publish_time=datetime.datetime(2016, 6, 28, 0, 0), topic_id=256, confidence_level_top=0.212099170435346, entity_id=None, confidence_level_ent=None, category_id=1514, confidence_level_cat=0.058682925),\n",
" Row(document_id=29834, ad_id=45, display_id=15935932, clicked=0, campaign_id=26, advertiser_id=39, source_id=2586, publisher_id=278, publish_time=datetime.datetime(2016, 6, 28, 0, 0), topic_id=256, confidence_level_top=0.212099170435346, entity_id=None, confidence_level_ent=None, category_id=1513, confidence_level_cat=0.7712613),\n",
" Row(document_id=29834, ad_id=45, display_id=15935932, clicked=0, campaign_id=26, advertiser_id=39, source_id=2586, publisher_id=278, publish_time=datetime.datetime(2016, 6, 28, 0, 0), topic_id=256, confidence_level_top=0.212099170435346, entity_id=None, confidence_level_ent=None, category_id=1514, confidence_level_cat=0.058682925),\n",
" Row(document_id=29834, ad_id=45, display_id=16027643, clicked=0, campaign_id=26, advertiser_id=39, source_id=2586, publisher_id=278, publish_time=datetime.datetime(2016, 6, 28, 0, 0), topic_id=256, confidence_level_top=0.212099170435346, entity_id=None, confidence_level_ent=None, category_id=1513, confidence_level_cat=0.7712613),\n",
" Row(document_id=29834, ad_id=45, display_id=16027643, clicked=0, campaign_id=26, advertiser_id=39, source_id=2586, publisher_id=278, publish_time=datetime.datetime(2016, 6, 28, 0, 0), topic_id=256, confidence_level_top=0.212099170435346, entity_id=None, confidence_level_ent=None, category_id=1514, confidence_level_cat=0.058682925)]"
]
},
"execution_count": 182,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c_train_pr_doc_det_df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": 246,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"87141731"
]
},
"execution_count": 246,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clicks_train_df.count()"
]
},
{
"cell_type": "code",
"execution_count": 198,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# c_train_pr_doc_det_ev_df = c_train_pr_doc_det_df.join(events_df, on=['document_id','display_id'], how='left')\n",
"# c_test_pr_doc_det_ev_df = c_test_pr_doc_det_df.join(events_df, on=['document_id','display_id'], how='left')\n",
"# ( c_train_pr_doc_det_df.document_id == events_df.document_id) & (c_train_pr_doc_det_df.display_id == events_df.display_id)\n",
"# c_train_pr_doc_det_ev_df = c_train_pr_doc_det_df.join(events_df, ( c_train_pr_doc_det_df.document_id == events_df.document_id) & (c_train_pr_doc_det_df.display_id == events_df.display_id), how='left')\n",
"c_train_pr_doc_det_ev_df = c_train_pr_doc_det_df.join(events_df, on=['display_id'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 185,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1482391843"
]
},
"execution_count": 185,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c_train_pr_doc_det_ev_df.count()"
]
},
{
"cell_type": "code",
"execution_count": 199,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(display_id=148, document_id=990613, ad_id=89351, clicked=1, campaign_id=7617, advertiser_id=2181, source_id=9457, publisher_id=None, publish_time=datetime.datetime(2015, 12, 9, 0, 0), topic_id=148, confidence_level_top=0.071109160240813, entity_id=u'956ae28fab890a14b653f114817152fb', confidence_level_ent=0.200192220085654, category_id=1408, confidence_level_cat=0.435300024, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=990613, ad_id=89351, clicked=1, campaign_id=7617, advertiser_id=2181, source_id=9457, publisher_id=None, publish_time=datetime.datetime(2015, 12, 9, 0, 0), topic_id=148, confidence_level_top=0.071109160240813, entity_id=u'956ae28fab890a14b653f114817152fb', confidence_level_ent=0.200192220085654, category_id=1802, confidence_level_cat=0.033120654, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=234, confidence_level_top=0.431677893732324, entity_id=u'5cdc05917c5e17756de7a71380b0e881', confidence_level_ent=0.233631438484806, category_id=1403, confidence_level_cat=0.92, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=234, confidence_level_top=0.431677893732324, entity_id=u'5cdc05917c5e17756de7a71380b0e881', confidence_level_ent=0.233631438484806, category_id=1408, confidence_level_cat=0.07, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=234, confidence_level_top=0.431677893732324, entity_id=u'83b2d3695d411592f77cc6af68622b95', confidence_level_ent=0.227887682258509, category_id=1403, confidence_level_cat=0.92, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=234, confidence_level_top=0.431677893732324, entity_id=u'83b2d3695d411592f77cc6af68622b95', confidence_level_ent=0.227887682258509, category_id=1408, confidence_level_cat=0.07, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=234, confidence_level_top=0.431677893732324, entity_id=u'6ac578848abcd92da25e7d7e6bca6e4a', confidence_level_ent=0.217674532363693, category_id=1403, confidence_level_cat=0.92, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=234, confidence_level_top=0.431677893732324, entity_id=u'6ac578848abcd92da25e7d7e6bca6e4a', confidence_level_ent=0.217674532363693, category_id=1408, confidence_level_cat=0.07, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=198, confidence_level_top=0.0155304057686555, entity_id=u'5cdc05917c5e17756de7a71380b0e881', confidence_level_ent=0.233631438484806, category_id=1403, confidence_level_cat=0.92, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=198, confidence_level_top=0.0155304057686555, entity_id=u'5cdc05917c5e17756de7a71380b0e881', confidence_level_ent=0.233631438484806, category_id=1408, confidence_level_cat=0.07, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=198, confidence_level_top=0.0155304057686555, entity_id=u'83b2d3695d411592f77cc6af68622b95', confidence_level_ent=0.227887682258509, category_id=1403, confidence_level_cat=0.92, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=198, confidence_level_top=0.0155304057686555, entity_id=u'83b2d3695d411592f77cc6af68622b95', confidence_level_ent=0.227887682258509, category_id=1408, confidence_level_cat=0.07, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=198, confidence_level_top=0.0155304057686555, entity_id=u'6ac578848abcd92da25e7d7e6bca6e4a', confidence_level_ent=0.217674532363693, category_id=1403, confidence_level_cat=0.92, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1086755, ad_id=152656, clicked=0, campaign_id=10511, advertiser_id=2151, source_id=7654, publisher_id=None, publish_time=datetime.datetime(2015, 11, 17, 0, 0), topic_id=198, confidence_level_top=0.0155304057686555, entity_id=u'6ac578848abcd92da25e7d7e6bca6e4a', confidence_level_ent=0.217674532363693, category_id=1408, confidence_level_cat=0.07, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1060089, ad_id=152140, clicked=0, campaign_id=19032, advertiser_id=1593, source_id=7744, publisher_id=1204, publish_time=datetime.datetime(2016, 1, 23, 1, 0), topic_id=227, confidence_level_top=0.334231753378017, entity_id=u'20469b5d11097141d0a3e67a61d5fbad', confidence_level_ent=0.203818477105146, category_id=1403, confidence_level_cat=0.92, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1060089, ad_id=152140, clicked=0, campaign_id=19032, advertiser_id=1593, source_id=7744, publisher_id=1204, publish_time=datetime.datetime(2016, 1, 23, 1, 0), topic_id=227, confidence_level_top=0.334231753378017, entity_id=u'20469b5d11097141d0a3e67a61d5fbad', confidence_level_ent=0.203818477105146, category_id=1510, confidence_level_cat=0.07, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1060089, ad_id=152140, clicked=0, campaign_id=19032, advertiser_id=1593, source_id=7744, publisher_id=1204, publish_time=datetime.datetime(2016, 1, 23, 1, 0), topic_id=24, confidence_level_top=0.155360672049449, entity_id=u'20469b5d11097141d0a3e67a61d5fbad', confidence_level_ent=0.203818477105146, category_id=1403, confidence_level_cat=0.92, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1060089, ad_id=152140, clicked=0, campaign_id=19032, advertiser_id=1593, source_id=7744, publisher_id=1204, publish_time=datetime.datetime(2016, 1, 23, 1, 0), topic_id=24, confidence_level_top=0.155360672049449, entity_id=u'20469b5d11097141d0a3e67a61d5fbad', confidence_level_ent=0.203818477105146, category_id=1510, confidence_level_cat=0.07, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1379561, ad_id=326768, clicked=0, campaign_id=23374, advertiser_id=2348, source_id=9419, publisher_id=1144, publish_time=datetime.datetime(2016, 5, 1, 6, 0), topic_id=193, confidence_level_top=0.3374300487159, entity_id=u'3d6c006d132611e4ee097cb5814a42b4', confidence_level_ent=0.214850161504542, category_id=1403, confidence_level_cat=0.386098148, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=148, document_id=1379561, ad_id=326768, clicked=0, campaign_id=23374, advertiser_id=2348, source_id=9419, publisher_id=1144, publish_time=datetime.datetime(2016, 5, 1, 6, 0), topic_id=193, confidence_level_top=0.3374300487159, entity_id=u'3d6c006d132611e4ee097cb5814a42b4', confidence_level_ent=0.214850161504542, category_id=1408, confidence_level_cat=0.029377033, uuid=u'9adce6a5363308', document_id_e=1205772, timestamp=11202, platform=2, geo_location=u'US>LA>612'),\n",
" Row(display_id=463, document_id=1116986, ad_id=187292, clicked=0, campaign_id=15430, advertiser_id=769, source_id=8254, publisher_id=None, publish_time=datetime.datetime(2015, 9, 28, 0, 0), topic_id=136, confidence_level_top=0.391139428660311, entity_id=u'69bfbe84e30ee1f81f44d612550050a3', confidence_level_ent=0.280951353134529, category_id=1702, confidence_level_cat=0.92, uuid=u'85d999a1dd5483', document_id_e=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, document_id=1116986, ad_id=187292, clicked=0, campaign_id=15430, advertiser_id=769, source_id=8254, publisher_id=None, publish_time=datetime.datetime(2015, 9, 28, 0, 0), topic_id=136, confidence_level_top=0.391139428660311, entity_id=u'69bfbe84e30ee1f81f44d612550050a3', confidence_level_ent=0.280951353134529, category_id=2005, confidence_level_cat=0.07, uuid=u'85d999a1dd5483', document_id_e=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, document_id=747296, ad_id=240837, clicked=0, campaign_id=25854, advertiser_id=1809, source_id=6523, publisher_id=699, publish_time=datetime.datetime(2015, 6, 19, 9, 0), topic_id=66, confidence_level_top=0.409251863877797, entity_id=u'87b429af71749ed33cd414d73ea027a7', confidence_level_ent=0.362639961752306, category_id=1611, confidence_level_cat=0.92, uuid=u'85d999a1dd5483', document_id_e=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, document_id=747296, ad_id=240837, clicked=0, campaign_id=25854, advertiser_id=1809, source_id=6523, publisher_id=699, publish_time=datetime.datetime(2015, 6, 19, 9, 0), topic_id=66, confidence_level_top=0.409251863877797, entity_id=u'87b429af71749ed33cd414d73ea027a7', confidence_level_ent=0.362639961752306, category_id=1708, confidence_level_cat=0.07, uuid=u'85d999a1dd5483', document_id_e=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, document_id=747296, ad_id=240837, clicked=0, campaign_id=25854, advertiser_id=1809, source_id=6523, publisher_id=699, publish_time=datetime.datetime(2015, 6, 19, 9, 0), topic_id=66, confidence_level_top=0.409251863877797, entity_id=u'7ad29499e89189387523774217c5ee77', confidence_level_ent=0.346109152265502, category_id=1611, confidence_level_cat=0.92, uuid=u'85d999a1dd5483', document_id_e=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, document_id=747296, ad_id=240837, clicked=0, campaign_id=25854, advertiser_id=1809, source_id=6523, publisher_id=699, publish_time=datetime.datetime(2015, 6, 19, 9, 0), topic_id=66, confidence_level_top=0.409251863877797, entity_id=u'7ad29499e89189387523774217c5ee77', confidence_level_ent=0.346109152265502, category_id=1708, confidence_level_cat=0.07, uuid=u'85d999a1dd5483', document_id_e=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, document_id=747296, ad_id=240837, clicked=0, campaign_id=25854, advertiser_id=1809, source_id=6523, publisher_id=699, publish_time=datetime.datetime(2015, 6, 19, 9, 0), topic_id=66, confidence_level_top=0.409251863877797, entity_id=u'be21ef1d6397ee571149592fd307b997', confidence_level_ent=0.267092747237342, category_id=1611, confidence_level_cat=0.92, uuid=u'85d999a1dd5483', document_id_e=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, document_id=747296, ad_id=240837, clicked=0, campaign_id=25854, advertiser_id=1809, source_id=6523, publisher_id=699, publish_time=datetime.datetime(2015, 6, 19, 9, 0), topic_id=66, confidence_level_top=0.409251863877797, entity_id=u'be21ef1d6397ee571149592fd307b997', confidence_level_ent=0.267092747237342, category_id=1708, confidence_level_cat=0.07, uuid=u'85d999a1dd5483', document_id_e=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, document_id=910765, ad_id=38813, clicked=1, campaign_id=5499, advertiser_id=1419, source_id=7373, publisher_id=None, publish_time=datetime.datetime(2016, 6, 10, 0, 0), topic_id=26, confidence_level_top=0.348044979621646, entity_id=u'8e233b1d2cd2bfe82a9ccdf8b6c81038', confidence_level_ent=0.388940317908544, category_id=1205, confidence_level_cat=0.92, uuid=u'85d999a1dd5483', document_id_e=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764'),\n",
" Row(display_id=463, document_id=910765, ad_id=38813, clicked=1, campaign_id=5499, advertiser_id=1419, source_id=7373, publisher_id=None, publish_time=datetime.datetime(2016, 6, 10, 0, 0), topic_id=26, confidence_level_top=0.348044979621646, entity_id=u'8e233b1d2cd2bfe82a9ccdf8b6c81038', confidence_level_ent=0.388940317908544, category_id=1202, confidence_level_cat=0.07, uuid=u'85d999a1dd5483', document_id_e=1787133, timestamp=34492, platform=1, geo_location=u'US>SD>764')]"
]
},
"execution_count": 199,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c_train_pr_doc_det_ev_df.head(30)"
]
},
{
"cell_type": "code",
"execution_count": 200,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1482391843"
]
},
"execution_count": 200,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c_train_pr_doc_det_ev_df[c_train_pr_doc_det_ev_df['uuid'] != 'None'].count()"
]
},
{
"cell_type": "code",
"execution_count": 245,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[Row(display_id=16004523, uuid=u'48136136ea989c', document_id_e=1714149, timestamp=1070958883, platform=1, geo_location=u'US>NY>501')]"
]
},
"execution_count": 245,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"events_df[events_df['display_id'] == '16004523'].head(10)"
]
},
{
"cell_type": "code",
"execution_count": 211,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"est = GradientBoostingClassifier(n_estimators=200, max_depth=3)"
]
},
{
"cell_type": "code",
"execution_count": 216,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"c_train_pr_doc_det_ev_pandas_df_train_X = c_train_pr_doc_det_ev_df.select('display_id','document_id','ad_id','campaign_id','advertiser_id','source_id',\n",
" 'publisher_id','publish_time','topic_id','confidence_level_top','entity_id',\n",
" 'confidence_level_ent','category_id','confidence_level_cat','uuid',\n",
" 'document_id_e','timestamp','platform','geo_location')"
]
},
{
"cell_type": "code",
"execution_count": 217,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"c_train_pr_doc_det_ev_pandas_df_train_y = c_train_pr_doc_det_ev_df.select('clicked')"
]
},
{
"cell_type": "code",
"execution_count": 219,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"rf = RandomForestClassifier(numTrees=100, maxDepth=10, labelCol=\"indexed\", seed=42)"
]
},
{
"cell_type": "code",
"execution_count": 231,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"c_train_pr_doc_det_ev_df_final = c_train_pr_doc_det_ev_df.select('display_id','document_id','ad_id','campaign_id','advertiser_id','source_id',\n",
" 'publisher_id','topic_id','confidence_level_top',\n",
" 'confidence_level_ent','category_id','confidence_level_cat',\n",
" 'document_id_e','platform','clicked')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"c_test_pr_doc_det_ev_df_final = c_test_pr_doc_det_ev_df.select('display_id','document_id','ad_id','campaign_id','advertiser_id','source_id',\n",
" 'publisher_id','topic_id','confidence_level_top',\n",
" 'confidence_level_ent','category_id','confidence_level_cat',\n",
" 'document_id_e','platform','clicked')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model building"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is the part where all my attempst to build a model have failed so far. I've tried so many things that I don't even know what part relates to what attempt :P. \n",
"To run a model in PySpark, a pipeline needs to be built. A simple one is attempted here."
]
},
{
"cell_type": "code",
"execution_count": 247,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"cols_now = ['display_id','document_id','ad_id','campaign_id','advertiser_id','source_id',\n",
"'publisher_id','topic_id','confidence_level_top',\n",
"'confidence_level_ent','category_id','confidence_level_cat',\n",
"'document_id_e','platform']\n",
"assembler_features = VectorAssembler(inputCols=cols_now, outputCol='features')\n",
"labelIndexer = StringIndexer(inputCol='clicked', outputCol=\"label\")\n",
"tmp = [assembler_features, labelIndexer]\n",
"pipeline = Pipeline(stages=tmp)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 242,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"PythonRDD[1611] at RDD at PythonRDD.scala:48"
]
},
"execution_count": 242,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from pyspark.ml.feature import VectorAssembler\n",
"\n",
"assembler = VectorAssembler(\n",
" inputCols=cols_now,\n",
" outputCol=\"features\")\n",
"\n",
"transformed = assembler.transform(c_train_pr_doc_det_ev_df_final)\n",
"\n",
"from pyspark.mllib.regression import LabeledPoint\n",
"from pyspark.sql.functions import col\n",
"\n",
"(transformed.select(col('clicked').alias(\"label\"), col(\"features\"))\n",
" .rdd\n",
" .map(lambda row: LabeledPoint(row.label, row.features)))"
]
},
{
"cell_type": "code",
"execution_count": 243,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Row(display_id=148, document_id=1060089, ad_id=152140, campaign_id=19032, advertiser_id=1593, source_id=7744, publisher_id=1204, topic_id=227, confidence_level_top=0.334231753378017, confidence_level_ent=0.203818477105146, category_id=1403, confidence_level_cat=0.92, document_id_e=1205772, platform=2, clicked=0, features=DenseVector([148.0, 1060089.0, 152140.0, 19032.0, 1593.0, 7744.0, 1204.0, 227.0, 0.3342, 0.2038, 1403.0, 0.92, 1205772.0, 2.0]))"
]
},
"execution_count": 243,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"transformed.head()"
]
},
{
"cell_type": "code",
"execution_count": 233,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"allData = pipeline.fit(c_train_pr_doc_det_ev_df_final).transform(c_train_pr_doc_det_ev_df_final)"
]
},
{
"cell_type": "code",
"execution_count": 234,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"DataFrame[display_id: int, document_id: int, ad_id: int, campaign_id: int, advertiser_id: int, source_id: int, publisher_id: int, topic_id: int, confidence_level_top: double, confidence_level_ent: double, category_id: int, confidence_level_cat: double, document_id_e: int, platform: int, clicked: int, features: vector, label: double]"
]
},
"execution_count": 234,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"allData.cache()"
]
},
{
"cell_type": "code",
"execution_count": 255,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "TypeError",
"evalue": "Invalid param value given for param \"inputCol\". Could not convert <class 'pyspark.sql.dataframe.DataFrame'> to string type",
"output_type": "error",
"traceback": [
"\u001b[0;31m\u001b[0m",
"\u001b[0;31mTypeError\u001b[0mTraceback (most recent call last)",
"\u001b[0;32m<ipython-input-255-67616eb5a4f5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;31m# Set maxCategories so features with > 4 distinct values are treated as continuous.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;31m#featureIndexer = VectorIndexer(inputCol=cols_now, outputCol='features'), outputCol=\"indexedFeatures\", maxCategories=4).fit(c_train_pr_doc_det_ev_df_final)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mfeatureIndexer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mVectorIndexer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputCol\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtransformed\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutputCol\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"indexedFeatures\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmaxCategories\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m4\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc_train_pr_doc_det_ev_df_final\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/usr/lib/spark/python/pyspark/__init__.pyc\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 96\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Method %s forces keyword arguments.\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 97\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_input_kwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 98\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 99\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/ml/feature.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, maxCategories, inputCol, outputCol)\u001b[0m\n\u001b[1;32m 1967\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_setDefault\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmaxCategories\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m20\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1968\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__init__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_input_kwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1969\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetParams\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1970\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1971\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0mkeyword_only\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/__init__.pyc\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 96\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Method %s forces keyword arguments.\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 97\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_input_kwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 98\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 99\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/ml/feature.py\u001b[0m in \u001b[0;36msetParams\u001b[0;34m(self, maxCategories, inputCol, outputCol)\u001b[0m\n\u001b[1;32m 1977\u001b[0m \"\"\"\n\u001b[1;32m 1978\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetParams\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_input_kwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1979\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_set\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1980\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1981\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0msince\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"1.4.0\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/ml/param/__init__.py\u001b[0m in \u001b[0;36m_set\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m 431\u001b[0m \u001b[0mvalue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtypeConverter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 432\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 433\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Invalid param value given for param \"%s\". %s'\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 434\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_paramMap\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 435\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mTypeError\u001b[0m: Invalid param value given for param \"inputCol\". Could not convert <class 'pyspark.sql.dataframe.DataFrame'> to string type"
]
}
],
"source": [
"# Attempting a new pipeline\n",
"labelIndexer = StringIndexer(inputCol=\"clicked\", outputCol=\"indexedLabel\").fit(c_train_pr_doc_det_ev_df_final)\n",
"# Automatically identify categorical features, and index them.\n",
"# Set maxCategories so features with > 4 distinct values are treated as continuous.\n",
"#featureIndexer = VectorIndexer(inputCol=cols_now, outputCol='features'), outputCol=\"indexedFeatures\", maxCategories=4).fit(c_train_pr_doc_det_ev_df_final)\n",
"featureIndexer = VectorIndexer(inputCol=transformed, outputCol=\"indexedFeatures\", maxCategories=4).fit(c_train_pr_doc_det_ev_df_final)\n"
]
},
{
"cell_type": "code",
"execution_count": 240,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "NameError",
"evalue": "name 'featureIndexer' is not defined",
"output_type": "error",
"traceback": [
"\u001b[0;31m\u001b[0m",
"\u001b[0;31mNameError\u001b[0mTraceback (most recent call last)",
"\u001b[0;32m<ipython-input-240-1cb9bb624e61>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;31m# Chain indexers and forest in a Pipeline\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mpipeline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mPipeline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstages\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mlabelIndexer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeatureIndexer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrf\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0;31m# Train model. This also runs the indexers.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mNameError\u001b[0m: name 'featureIndexer' is not defined"
]
}
],
"source": [
"\n",
"\n",
"# Train a RandomForest model.\n",
"rf = RandomForestClassifier(labelCol=\"indexedLabel\", featuresCol=\"indexedFeatures\")\n",
"\n",
"# Chain indexers and forest in a Pipeline\n",
"pipeline = Pipeline(stages=[labelIndexer, featureIndexer, rf])\n",
"\n",
"# Train model. This also runs the indexers.\n",
"model = pipeline.fit(c_train_pr_doc_det_ev_df)\n",
"\n",
"# Make predictions.\n",
"# predictions = model.transform(testData)"
]
},
{
"cell_type": "code",
"execution_count": 239,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "IllegalArgumentException",
"evalue": "u'Field \"features\" does not exist.'",
"output_type": "error",
"traceback": [
"\u001b[0;31m\u001b[0m",
"\u001b[0;31mIllegalArgumentException\u001b[0mTraceback (most recent call last)",
"\u001b[0;32m<ipython-input-239-fec559531d39>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mpyspark\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mml\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclassification\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mLogisticRegression\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0mlr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mLogisticRegression\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmaxIter\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mregParam\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0melasticNetParam\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.8\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mlrModel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc_train_pr_doc_det_ev_df_final\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/usr/lib/spark/python/pyspark/ml/base.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, dataset, params)\u001b[0m\n\u001b[1;32m 62\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 63\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 64\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 65\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 66\u001b[0m raise ValueError(\"Params must be either a param map or a list/tuple of param maps, \"\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/ml/wrapper.py\u001b[0m in \u001b[0;36m_fit\u001b[0;34m(self, dataset)\u001b[0m\n\u001b[1;32m 211\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 212\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 213\u001b[0;31m \u001b[0mjava_model\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit_java\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 214\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjava_model\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 215\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/ml/wrapper.py\u001b[0m in \u001b[0;36m_fit_java\u001b[0;34m(self, dataset)\u001b[0m\n\u001b[1;32m 208\u001b[0m \"\"\"\n\u001b[1;32m 209\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_transfer_params_to_java\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 210\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_java_obj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 211\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 212\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1131\u001b[0m \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1132\u001b[0m return_value = get_return_value(\n\u001b[0;32m-> 1133\u001b[0;31m answer, self.gateway_client, self.target_id, self.name)\n\u001b[0m\u001b[1;32m 1134\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1135\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtemp_arg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtemp_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/sql/utils.pyc\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m 77\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mQueryExecutionException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m': '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstackTrace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 78\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'java.lang.IllegalArgumentException: '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 79\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mIllegalArgumentException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m': '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstackTrace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 80\u001b[0m \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 81\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mdeco\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mIllegalArgumentException\u001b[0m: u'Field \"features\" does not exist.'"
]
}
],
"source": [
"from pyspark.ml.classification import LogisticRegression\n",
"lr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)\n",
"lrModel = lr.fit(c_train_pr_doc_det_ev_df_final)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Make predictions.\n",
"predictions = model.transform(c_test_pr_doc_det_ev_df_final)"
]
},
{
"cell_type": "code",
"execution_count": 235,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "Py4JJavaError",
"evalue": "An error occurred while calling o1142.fit.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 388.0 failed 4 times, most recent failure: Lost task 0.3 in stage 388.0 (TID 12339, cluster-1-w-3.c.ultra-dimension-155718.internal): org.apache.spark.SparkException: Failed to execute user defined function($anonfun$3: (struct<col1:double,col2:double,col3:double,col4:double,col5:double,col6:double,col7:double,col8:double,col9:double,col10:double,col11:double,col12:double,col13:double,col14:double>) => vector)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)\n\tat org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:139)\n\tat org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:213)\n\tat org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:935)\n\tat org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:926)\n\tat org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)\n\tat org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:926)\n\tat org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:670)\n\tat org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:281)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:86)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.spark.SparkException: Values to assemble cannot be null.\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:160)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:143)\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)\n\tat org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:143)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:99)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:98)\n\t... 39 more\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1454)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1442)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1441)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1441)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\n\tat scala.Option.foreach(Option.scala:257)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1667)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1622)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1611)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1873)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1886)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1899)\n\tat org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1324)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\n\tat org.apache.spark.rdd.RDD.withScope(RDD.scala:358)\n\tat org.apache.spark.rdd.RDD.take(RDD.scala:1298)\n\tat org.apache.spark.ml.tree.impl.DecisionTreeMetadata$.buildMetadata(DecisionTreeMetadata.scala:112)\n\tat org.apache.spark.ml.tree.impl.RandomForest$.run(RandomForest.scala:105)\n\tat org.apache.spark.ml.classification.RandomForestClassifier.train(RandomForestClassifier.scala:118)\n\tat org.apache.spark.ml.classification.RandomForestClassifier.train(RandomForestClassifier.scala:45)\n\tat org.apache.spark.ml.Predictor.fit(Predictor.scala:90)\n\tat org.apache.spark.ml.Predictor.fit(Predictor.scala:71)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:280)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:214)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.spark.SparkException: Failed to execute user defined function($anonfun$3: (struct<col1:double,col2:double,col3:double,col4:double,col5:double,col6:double,col7:double,col8:double,col9:double,col10:double,col11:double,col12:double,col13:double,col14:double>) => vector)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)\n\tat org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:139)\n\tat org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:213)\n\tat org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:935)\n\tat org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:926)\n\tat org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)\n\tat org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:926)\n\tat org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:670)\n\tat org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:281)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:86)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\t... 1 more\nCaused by: org.apache.spark.SparkException: Values to assemble cannot be null.\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:160)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:143)\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)\n\tat org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:143)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:99)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:98)\n\t... 39 more\n",
"output_type": "error",
"traceback": [
"\u001b[0;31m\u001b[0m",
"\u001b[0;31mPy4JJavaError\u001b[0mTraceback (most recent call last)",
"\u001b[0;32m<ipython-input-235-30cd410c4edf>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mrf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mRF\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlabelCol\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'label'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeaturesCol\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'features'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mnumTrees\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m200\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mfit\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mallData\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/usr/lib/spark/python/pyspark/ml/base.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, dataset, params)\u001b[0m\n\u001b[1;32m 62\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 63\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 64\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 65\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 66\u001b[0m raise ValueError(\"Params must be either a param map or a list/tuple of param maps, \"\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/ml/wrapper.py\u001b[0m in \u001b[0;36m_fit\u001b[0;34m(self, dataset)\u001b[0m\n\u001b[1;32m 211\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 212\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 213\u001b[0;31m \u001b[0mjava_model\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit_java\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 214\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjava_model\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 215\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/ml/wrapper.py\u001b[0m in \u001b[0;36m_fit_java\u001b[0;34m(self, dataset)\u001b[0m\n\u001b[1;32m 208\u001b[0m \"\"\"\n\u001b[1;32m 209\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_transfer_params_to_java\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 210\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_java_obj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 211\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 212\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1131\u001b[0m \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1132\u001b[0m return_value = get_return_value(\n\u001b[0;32m-> 1133\u001b[0;31m answer, self.gateway_client, self.target_id, self.name)\n\u001b[0m\u001b[1;32m 1134\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1135\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtemp_arg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtemp_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/pyspark/sql/utils.pyc\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m 61\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdeco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 62\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 63\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 64\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mpy4j\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPy4JJavaError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 65\u001b[0m \u001b[0ms\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjava_exception\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/spark/python/lib/py4j-0.10.3-src.zip/py4j/protocol.py\u001b[0m in \u001b[0;36mget_return_value\u001b[0;34m(answer, gateway_client, target_id, name)\u001b[0m\n\u001b[1;32m 317\u001b[0m raise Py4JJavaError(\n\u001b[1;32m 318\u001b[0m \u001b[0;34m\"An error occurred while calling {0}{1}{2}.\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 319\u001b[0;31m format(target_id, \".\", name), value)\n\u001b[0m\u001b[1;32m 320\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 321\u001b[0m raise Py4JError(\n",
"\u001b[0;31mPy4JJavaError\u001b[0m: An error occurred while calling o1142.fit.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 388.0 failed 4 times, most recent failure: Lost task 0.3 in stage 388.0 (TID 12339, cluster-1-w-3.c.ultra-dimension-155718.internal): org.apache.spark.SparkException: Failed to execute user defined function($anonfun$3: (struct<col1:double,col2:double,col3:double,col4:double,col5:double,col6:double,col7:double,col8:double,col9:double,col10:double,col11:double,col12:double,col13:double,col14:double>) => vector)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)\n\tat org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:139)\n\tat org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:213)\n\tat org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:935)\n\tat org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:926)\n\tat org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)\n\tat org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:926)\n\tat org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:670)\n\tat org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:281)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:86)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.spark.SparkException: Values to assemble cannot be null.\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:160)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:143)\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)\n\tat org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:143)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:99)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:98)\n\t... 39 more\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1454)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1442)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1441)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1441)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\n\tat scala.Option.foreach(Option.scala:257)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1667)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1622)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1611)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1873)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1886)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1899)\n\tat org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1324)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\n\tat org.apache.spark.rdd.RDD.withScope(RDD.scala:358)\n\tat org.apache.spark.rdd.RDD.take(RDD.scala:1298)\n\tat org.apache.spark.ml.tree.impl.DecisionTreeMetadata$.buildMetadata(DecisionTreeMetadata.scala:112)\n\tat org.apache.spark.ml.tree.impl.RandomForest$.run(RandomForest.scala:105)\n\tat org.apache.spark.ml.classification.RandomForestClassifier.train(RandomForestClassifier.scala:118)\n\tat org.apache.spark.ml.classification.RandomForestClassifier.train(RandomForestClassifier.scala:45)\n\tat org.apache.spark.ml.Predictor.fit(Predictor.scala:90)\n\tat org.apache.spark.ml.Predictor.fit(Predictor.scala:71)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:280)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:214)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.spark.SparkException: Failed to execute user defined function($anonfun$3: (struct<col1:double,col2:double,col3:double,col4:double,col5:double,col6:double,col7:double,col8:double,col9:double,col10:double,col11:double,col12:double,col13:double,col14:double>) => vector)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)\n\tat org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:139)\n\tat org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:213)\n\tat org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:935)\n\tat org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:926)\n\tat org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)\n\tat org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:926)\n\tat org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:670)\n\tat org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:281)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:86)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\t... 1 more\nCaused by: org.apache.spark.SparkException: Values to assemble cannot be null.\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:160)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:143)\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)\n\tat org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:143)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:99)\n\tat org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:98)\n\t... 39 more\n"
]
}
],
"source": [
"rf = RF(labelCol='label', featuresCol='features',numTrees=200)\n",
"fit = rf.fit(allData)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"c_test_pr_doc_det_ev_df_final = c_test_pr_doc_det_ev_df.select('display_id','document_id','ad_id','campaign_id','advertiser_id','source_id',\n",
" 'publisher_id','topic_id','confidence_level_top',\n",
" 'confidence_level_ent','category_id','confidence_level_cat',\n",
" 'document_id_e','platform')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"test_data = pipeline.fit(c_test_pr_doc_det_ev_df_final).transform(c_test_pr_doc_det_ev_df_final)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "PySpark",
"language": "python",
"name": "pyspark"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment