Skip to content

Instantly share code, notes, and snippets.

@c21
Created April 22, 2021 04:44
Show Gist options
  • Save c21/5a9d7abb915147832f72c9c1a9047ff4 to your computer and use it in GitHub Desktop.
Save c21/5a9d7abb915147832f72c9c1a9047ff4 to your computer and use it in GitHub Desktop.
first level controlled by capacity, but not counter. second level controlled by counter
testFallbackStartsAt = Some(2, 3)
val bitMaxCapacity = testFallbackStartsAt match {
case Some(c1, _) => log2(c1)
case _ => sqlContext.conf.fastHashAggregateRowMaxCapacityBit
}
// 1st level hash map
agg_buffer = fastHashMap.findOrInsert(key);
if (agg_buffer == null) {
// generated. code for key in unsafe row format
...
if (counter < 3) {
// 2nd level hash map
agg_buffer = regularHashMap.getAggregationBufferFromUnsafeRow(key_in_unsafe_row, ...);
}
if (agg_buffer == null) {
// sort-based fallback
regularHashMap.destructAndCreateExternalSorter();
...
counter = 0;
}
}
counter += 1;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment