Skip to content

Instantly share code, notes, and snippets.

@tromika
Created May 23, 2016 09:36
Show Gist options
  • Save tromika/889d6d62175ccdaedc8fc3085442c146 to your computer and use it in GitHub Desktop.
Save tromika/889d6d62175ccdaedc8fc3085442c146 to your computer and use it in GitHub Desktop.
Apache Spark SQL INET_NTOA function pyspark dummy version
%pyspark
import socket,struct
import re
def inetNtoa(ip,num):
ip = re.match("^([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})$",socket.inet_ntoa(struct.pack('!I', ip)))
str = ''
if(num==0 or num > 3):
return ip.group(0)
else:
for i in range(1,num+1):
str+=ip.group(i)+'.'
return str
# Usage INET_NTOA(col, number) - Number 0 or number higher than 3 you got the full ip - if number is 1,2,3 you got a subnet ip range.
sqlContext.registerFunction("INET_NTOA", lambda x,y: inetNtoa(x,y))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment