Get Homebrew installed on your mac if you don't already have it
Install highlight. "brew install highlight". (This brings down Lua and Boost as well)
Get Homebrew installed on your mac if you don't already have it
Install highlight. "brew install highlight". (This brings down Lua and Boost as well)
// Based on following links: | |
// http://andrew.hedges.name/experiments/haversine/ | |
// http://www.movable-type.co.uk/scripts/latlong.html | |
df | |
.withColumn("a", pow(sin(toRadians($"destination_latitude" - $"origin_latitude") / 2), 2) + cos(toRadians($"origin_latitude")) * cos(toRadians($"destination_latitude")) * pow(sin(toRadians($"destination_longitude" - $"origin_longitude") / 2), 2)) | |
.withColumn("distance", atan2(sqrt($"a"), sqrt(-$"a" + 1)) * 2 * 6371) | |
>>> | |
+--------------+-------------------+-------------+----------------+---------------+----------------+--------------------+---------------------+--------------------+------------------+ | |
|origin_airport|destination_airport| origin_city|destination_city|origin_latitude|origin_longitude|destination_latitude|destination_longitude| a| distance| |
http https://data.giss.nasa.gov/gistemp/tabledata_v4/GLB.Ts+dSST.csv | sed 1d | | |
spyql " | |
SELECT Year, cols[1:13] AS temps | |
FROM csv | |
TO json" | | |
spyql " | |
SELECT json->Year + ((row_number-1)%12)/12 AS year, json->temps AS temp | |
FROM json | |
EXPLODE json->temps | |
WHERE json->temps is not Null |