(Cleaning up for CQL3 interface using DataStax driver)
Let's say you have a large amount of Geo data. Let's say it's GPS data from thousands of autonomous drones mapping geo areas around the country. Or it could be something boring like a truck fleet positional information.
Either way, it's lots and lots of data in the form of Coordinates associated with some kind of value. And now that you have gazillion geo data, you want to visualize this data. A heatmap, perhaps.
If the data is relatively small as in up to a few hundred K's, you can simply put this into a heatmap and it'll probably work.
Well, if you have a fast Mac like me, that is.
The problem is, when you have huge amounts of data - data so large that you can't get clear picture without visualizing it - that's when this exact approach starts to fail. Heatmaps just can't take in millions of data points and allow you to scroll/zoom around at will.
Just as you would with any raw big data, we need to aggregate this geo data. In geo terms, this means instead of putting every lat/lon data on the map, we aggregate it into areas, then show the agg counts of the areas as heat on the map.
There are a couple of ways to do this. One is to use something like a GeoHash. This will then let you do greater than/less than query. This, however, is a sucky way to do this, because ge/le on two axis means a rectangle. Heat is concentrated in circular shapes, not rectangle. You don't want to show heatmap of big rectangles, that's just silly. You want circular geo aggregation instead.
This is where GeoHex comes in. A GeoHex divides the entire world into equally shaped hexagons. Instead of 500 million points on a map, a geohex of 5-10 mile radius allows you to aggregate data into a couple of hundred K's worth of aggregated data points, making it very manageable.
You can generate geohex from lat/lon fairly easily. You can generate multiple geohexes (of varying sizes) from same lat/lon fairly easily as well.
OTOH, processing millions of geohex an take a very long time especially if you use something that crawls. Like Ruby. Well, Ruby isn't the only one either. Do it in Go or whatever. You can write it in any language, it still will crawl as long as you do it in a serial fashion.
Now, code that translates lat/lon into various geohexes is dirt easy to prop it into raw MapReduce in Hadoop (or better yet, Spark) and see amazing speed up.
GeoHexApp.java, GeoHexMapper.java and GeoHexFileReducer.java below do this. 50 million records (47.5M to be exact) can be processed within a couple of minutes with this code, running under Hadoop 2.5.0 on a Retina MacBookPro 15 with Yarn vcore set to 8.
The mapper uses concatenated key to generate multiple geohexes on one run. This means GeoHex is prefixed with size code, so it can be separated into different size groups, or queried using size prefix.
(I initially tried this with a Pig script using UDF, but gave up after realizing it's much easier with raw MapReduce)
So now we have millions of lat/lon data aggregated into a few hundred thousand. We now want to store this data for easy retrieval.
For this, I now put data directly into Cassandra from Reducer as seen in GeoHexCassandraReducer.java. Aggregate count is put in to a composite key of (primary, geohex) which essentially makes it into a super wide column. In this case a row with almost 700K columns to be exact.
This means I have a day's worth of aggregated geo data, separated into ~700K different geohexes in one row. Tomorrow, I can add next day's data as a row. And another row the day after, etc.
Writing to Cassandra is a little slower than writing to a file, of course. It took a whopping 12 minutes to aggregate 47million rows into 700K aggregates, which were then written to Cassandra. Note that I did not do any tuning on Cassandra or Hadoop for this.
(wip)
Great article! Small typo fix (OTOTH -> OTOH):
https://gist.github.com/cyrbon/7088357e57c22de5e3bfbf3cecbf6c69/revisions