Skip to content

Instantly share code, notes, and snippets.

@abramsm
Last active December 15, 2015 17:29
Show Gist options
  • Save abramsm/5297210 to your computer and use it in GitHub Desktop.
Save abramsm/5297210 to your computer and use it in GitHub Desktop.
Testing merging n small sets in HyperLogLogPLus
public static void main(final String[] args) throws Throwable {
long startTime = System.currentTimeMillis();
int numSets = 10;
int setSize = 1 * 1000 * 1000;
int repeats = 5;
HyperLogLogPlus[] counters = new HyperLogLogPlus[numSets];
for (int i = 0; i < numSets; i++) {
counters[i] = new HyperLogLogPlus(15, 15);
}
for (int i = 0; i < numSets; i++) {
for (int j = 0; j < setSize; j++) {
String val = UUID.randomUUID().toString();
for (int z = 0; z < repeats; z++)
{
counters[i].offer(val);
}
}
}
ICardinality merged = counters[0];
long sum = merged.cardinality();
for (int i = 1; i < numSets; i++) {
sum += counters[i].cardinality();
merged = merged.merge(counters[i]);
}
long trueSize = numSets * setSize;
System.out.println("True Cardinality: " + trueSize);
System.out.println("Summed Cardinality: " + sum);
System.out.println("Merged Cardinality: " + merged.cardinality());
System.out.println("Merged Error: " + (merged.cardinality() - trueSize) / (float)trueSize);
System.out.println("Duration: " + ((System.currentTimeMillis() - startTime)/1000) + "s");
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment