Skip to content

Instantly share code, notes, and snippets.

@sudhir-reddy
Created May 21, 2015 16:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sudhir-reddy/4eb430d4be9f9f404c38 to your computer and use it in GitHub Desktop.
Save sudhir-reddy/4eb430d4be9f9f404c38 to your computer and use it in GitHub Desktop.
public class RoundRobinTableInputFormat extends TableInputFormat {
@Override
public List<InputSplit> getSplits(JobContext context) throws IOException {
List<InputSplit> inputSplits = super.getSplits(context); // Get splits from HBase TableInputFormat gives
List<InputSplit> roundRobinInputSplits = new ArrayList<InputSplit>();
Map<String, List<InputSplit>> regionServersInputSplits = new HashMap<String, List<InputSplit>>();
// Prepare a hashmap with each region server as key and list of Input Splits as value
if (inputSplits != null && inputSplits.size() != 0) {
for (InputSplit inputSplit : inputSplits) {
if (inputSplit instanceof TableSplit) {
String regionServer = ((TableSplit) inputSplit).getRegionLocation();
if (!Strings.isNullOrEmpty(regionServer)) {
List<InputSplit> splitList = regionServersInputSplits.get(regionServer);
if (splitList == null) {
splitList = new LinkedList<InputSplit>();
regionServersInputSplits.put(regionServer, splitList);
}
splitList.add(inputSplit);
continue;
}
}
// If TableSplit or region server not found, add it.
roundRobinInputSplits.add(inputSplit);
}
}
while (true) { //For each regionServer keep adding one split, till all are added
if (regionServersInputSplits.isEmpty()) {
break;
}
Iterator<String> iterator = regionServersInputSplits.keySet().iterator();
while (iterator.hasNext()){
String regionServer = iterator.next();
List<InputSplit> inputSplitListForRegion = regionServersInputSplits.get(regionServer);
if(! inputSplitListForRegion.isEmpty()) {
roundRobinInputSplits.add(inputSplitListForRegion.remove(0));
}
if(inputSplitListForRegion.isEmpty()) {
iterator.remove();
}
}
}
return roundRobinInputSplits;
}
@Meghaditya
Copy link

while (true) { //For each regionServer keep adding one split, till all are added
if (regionServersInputSplits.isEmpty()) {
break;
}
...
}

could be replaced with

while (!regionServersInputSplits.isEmpty()) {
...
}

@VarunBatraIT
Copy link

"inputSplits.size() != 0" VS "inputSplits.isEmpty()"

http://pmd.sourceforge.net/pmd-4.3.0/rules/design.html

@saintstack
Copy link

Is it ok putting this class over here apache/hbase#2947 ? Thanks (and thanks for the nice accompanying blog).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment