Skip to content

Instantly share code, notes, and snippets.

@bveliqi

bveliqi/relationsship.scala

Last active October 29, 2017 21:48
Show Gist options
  • Save bveliqi/2aee1dfe9a848f3059b8e4aad74c3641 to your computer and use it in GitHub Desktop.
Save bveliqi/2aee1dfe9a848f3059b8e4aad74c3641 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
relationsship.scala
val relationships = df.select("id", "screenName", "friends")
.withColumn("friends", regexp_replace('friends, "\\[ | ]", ""))
.withColumn("friends", split('friends, "; "))
.withColumn("friend_id", explode('friends))
.withColumn("friend_id", regexp_replace('friend_id, "\"", ""))
.withColumn("id", 'id.cast(LongType))
.withColumn("friend_id", 'friend_id.cast(LongType))
.drop("friends")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment