SVM with SGD clarification for question http://apache-spark-user-list.1001560.n3.nabble.com/Need-help-with-SVM-tt27955.html

0.question_svm_with_sgd.md

Hello, I am using linear SVM to train my model and generate a line through my data. However my model always predicts 1 for all the feature examples. Here is my code:

print data_rdd.take(5) [LabeledPoint(1.0, [1.9643,4.5957]), LabeledPoint(1.0, [2.2753,3.8589]), LabeledPoint(1.0, [2.9781,4.5651]), LabeledPoint(1.0, [2.932,3.5519]), LabeledPoint(1.0, [3.5772,2.856])]

---

from pyspark.mllib.classification import SVMWithSGD from pyspark.mllib.linalg import Vectors from sklearn.svm import SVC data_rdd=x_df.map(lambda x:LabeledPoint(x[1],x[0]))

model = SVMWithSGD.train(data_rdd, iterations=1000,regParam=1)

X=x_df.map(lambda x:x[0]).collect() Y=x_df.map(lambda x:x[1]).collect()

---

pred=[] for i in X: pred.append(model.predict(i)) print pred

[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

1.svm_with_sgd.py

# code tested with pyspark 
# pyspark --packages com.databricks:spark-csv_2.10:1.5.0

from pyspark.ml.feature import VectorAssembler
from pyspark.mllib.classification import LabeledPoint
from pyspark.mllib.classification import SVMWithSGD

# read data
df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('data.csv')

# prepare data
assembler = VectorAssembler(inputCols=["X", "Y"], outputCol="features")
data = assembler.transform(df)

# create RDD[LabeledPoint]
rdd = data.map(lambda row: LabeledPoint(row.label, row.features))

# Train SVM With SGD model
model = SVMWithSGD.train(rdd, iterations=1000,regParam=1.0,intercept=True,step=0.1)

# Create unlabled data
unlabeled_data = data.map(lambda x : x.features)

# Make prediction on unlabeled data and collect
model.predict(unlabeled_data).collect()
# [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0]

2.data.csv

Thank you for helping me. But I dont understand what did I do wrong earlier. I had done the same thing-split the data into features and labels and then train the data. Moreover right now the predictions are also not entirely correct(less 0's at the end). Do we need to specify threshold?

I didn't specify a threshold but you should be careful when splitting the data actually to perform stratified sampling so you don't up just with 1 labels in one split and 0s in the other.