Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@dvas0004
Last active March 5, 2018 08:35
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dvas0004/b4fe2960f2aeae4f9775f46e5238e4b7 to your computer and use it in GitHub Desktop.
Save dvas0004/b4fe2960f2aeae4f9775f46e5238e4b7 to your computer and use it in GitHub Desktop.
snippet for elasticsearch Java scroll API
SearchResponse scrollResp = client.prepareSearch("filebeat-*")
// set the timeout value - important for long running scrolls
.setScroll(new TimeValue(60*60))
// example of an "OR" filter using "should" boolean expression
.setQuery(QueryBuilders.boolQuery().should(QueryBuilders.termQuery("Type", "TRAFFIC"))
.should(QueryBuilders.termQuery("pfsense_type", "filterlog:")).minimumShouldMatch("1"))
// example of filter to query documents between a time range
.setPostFilter(QueryBuilders.rangeQuery("@timestamp").from(QueryFrom).to(QueryTo))
.setSize(1000).execute().actionGet(); //10,000 hits per shard will be returned for each scroll
//Scroll until no hits are returned
while (true) {
for (SearchHit hit : scrollResp.getHits().getHits()) {
//process the search hit
}
//update the scroll ID
scrollResp = client.prepareSearchScroll(scrollResp.getScrollId()).setScroll(new TimeValue(timeValueSetting)).execute().actionGet();
//Break condition: No hits are returned so the scroll is finished
if (scrollResp.getHits().getHits().length == 0) {
break;
}
}
@ddmytrenko
Copy link

Hi, @dvas0004! Thanks for the code snippet! Did you try this approach with RxJava?

@CyberDick
Copy link

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment