Skip to content

Instantly share code, notes, and snippets.

@joel-bernstein
Last active September 17, 2018 01:58
Show Gist options
  • Save joel-bernstein/025789728494e360b71192a0fc97968c to your computer and use it in GitHub Desktop.
Save joel-bernstein/025789728494e360b71192a0fc97968c to your computer and use it in GitHub Desktop.
plist(parallel(workerCollection,
numWorkers="1",
sort="year asc, month asc, day asc",
plist(facet(week1, zkHost="zk1", q="*:*", buckets="year, month, day", bucketSorts="year asc, month asc, day asc", count(*)),
facet(week2, zkHost="zk1", q="*:*", buckets="year, month, day", bucketSorts="year asc, month asc, day asc", count(*)),
facet(week3, zkHost="zk1", q="*:*", buckets="year, month, day", bucketSorts="year asc, month asc, day asc", count(*)))),
parallel(workerCollection,
numWorkers="1",
sort="year asc, month asc, day asc",
plist(facet(week4, zkHost="zk2", q="*:*", buckets="year, month, day", bucketSorts="year asc, month asc, day asc", count(*)),
facet(week5, zkHost="zk2", q="*:*", buckets="year, month, day", bucketSorts="year asc, month asc, day asc", count(*)),
facet(week6, zkHost="zk2", q="*:*", buckets="year, month, day", bucketSorts="year asc, month asc, day asc", count(*)))),
parallel(workerCollection,
numWorkers="1",
sort="year asc, month asc, day asc",
plist(facet(week7, zkHost="zk3", q="*:*", buckets="year, month, day", bucketSorts="year asc, month asc, day asc", count(*)),
facet(week8, zkHost="zk3", q="*:*", buckets="year, month, day", bucketSorts="year asc, month asc, day asc", count(*)),
facet(week9, zkHost="zk3", q="*:*", buckets="year, month, day", bucketSorts="year asc, month asc, day asc", count(*)))))
@joel-bernstein
Copy link
Author

The data is organized in chronological Solr Cloud collections by week. New collections are added every week. Each collection has two shards and two replicas and 200+ million records. Aggregating across 10 years of data would involve:

104,000,000,000 documents
520 collections
2080 Solr Instances

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment