Skip to content

Instantly share code, notes, and snippets.

@FavioVazquez

FavioVazquez/errors_transfer.py

Created May 2, 2018 15:38
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save FavioVazquez/dcd72fe4f0f4204736d46ba57112cb97 to your computer and use it in GitHub Desktop.
Save FavioVazquez/dcd72fe4f0f4204736d46ba57112cb97 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
errors_transfer.py
from pyspark.sql.types import DoubleType
from pyspark.sql.functions import expr
from pyspark.sql.functions import *
from pyspark.sql.types import *
def _p1(v):
return float(v.array[1])y
take_one = udf(_p1, DoubleType())
df = tested_df.withColumn("p", take_one(tested_df.probability))
wrong_df = df.orderBy(expr("abs(p - label)"), ascending=False)
wrong_df.select("image.origin", "p", "label").show(10)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment