Skip to content

Instantly share code, notes, and snippets.

@segalziv

segalziv/logstash.conf Secret

Created Feb 16, 2021
Embed
What would you like to do?
Logstash config to fetch Elasticsearch logs from Elastic Cloud
###################################################################################################################################################
#
# Logstash pipeline to retrieve Elastic Cloud logs and store them in a local file
#
# Command line:
# $> $LOGSTASH_HOME/bin/logstash -f /path/to/es-logs-to-file.conf -w 1
#
# Note: If you want the logs to be ordered by timestamp, it is important to run this Logstash pipeline with only a single worker thread (-w 1)
###################################################################################################################################################
input {
elasticsearch {
# Connect to Elastic Cloud on AWS, GCP or Azure
cloud_id => "cluster-name:jhsdshHD3673hHHD..."
cloud_auth => "<username>:<password>"
# query
docinfo => true
index => "elastic-cloud-logs*"
scroll => "2m"
size => 1000
query => '{ "sort": { "@timestamp": "asc" }, "query": {"terms": {"event.dataset": ["elasticsearch.server", "elasticsearch.gc", "elasticsearch.slowlog", "elasticsearch.deprecation"]}} }'
}
}
filter {
}
output {
file {
path => "/path/to/logs.json"
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment