goelprateek

**********************
**Gist
**********************
 
This gist details how to inner join two large datasets on the map-side, leveraging the join capability 
in mapreduce.  Such a join makes sense if both input datasets are too large to qualify for distribution 
through distributedcache, and can be implemented if both input datasets can be joined by the join key 
and both input datasets are sorted in the same order, by the join key.
 
There are two critical pieces to engaging the join behavior: 

goelprateek

My blog has an introduction to reduce side join in Java map reduce-
http://hadooped.blogspot.com/2013/09/reduce-side-join-options-in-java-map.html

goelprateek

# clean everything up
echo "killing mongod and mongos"
killall mongod
killall mongos
echo "removing data files"
rm -rf data/config
rm -rf data/shard*

# For mac make sure rlimits are high enough to open all necessary connections
ulimit -n 2048

goelprateek

package com.voicestreams.spark;


import org.apache.commons.io.FileUtils;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;

goelprateek

git branch -m old_branch new_branch         # Rename branch locally    
git push origin :old_branch                 # Delete the old branch    
git push --set-upstream origin new_branch   # Push the new branch, set local branch to track the new remote