Skip to content

Instantly share code, notes, and snippets.

@jhmartin
Created May 21, 2015 15:47
Show Gist options
  • Star 18 You must be signed in to star a gist
  • Fork 4 You must be signed in to fork a gist
  • Save jhmartin/c09e450853e273ee88f2 to your computer and use it in GitHub Desktop.
Save jhmartin/c09e450853e273ee88f2 to your computer and use it in GitHub Desktop.
Multiple Elasticsearch outputs in logstash
#http://pastebin.com/xGt6vv9R from yardenbar
output {
if [type] == "postfix" {
elasticsearch {
host => [ "ES_HOSTNAME" ]
protocol => "transport"
cluster => "elasticsearch"
index => "postfix-%{+YYYY.MM.dd}"
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates.d/postfix.json"
template_name => "postfix"
}
}
else if [type] in ["nginx", "nginx_error"] {
elasticsearch {
host => [ "ES_HOSTNAME" ]
protocol => "transport"
cluster => "elasticsearch"
index => "nginx-%{+YYYY.MM.dd}"
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates.d/nginx.json"
template_name => "nginx"
}
}
else if [type] == "rails" {
elasticsearch {
host => [ "ES_HOSTNAME" ]
protocol => "transport"
cluster => "elasticsearch"
index => "rails-%{+YYYY.MM.dd}"
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates.d/rails.json"
template_name => "rails"
}
}
else if [type] == "webpagetest" {
elasticsearch {
host => [ "ES_HOSTNAME" ]
protocol => "transport"
cluster => "elasticsearch"
index => "webpagetest-global"
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates.d/webpagetest.json"
template_name => "webpagetest"
}
}
else if [type] == "push-notification" {
elasticsearch {
host => [ "ES_HOSTNAME" ]
protocol => "transport"
cluster => "elasticsearch"
index => "push-notification-%{+YYYY.MM.dd}"
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates.d/push_notification.json"
template_name => "push_notification"
}
}
else if [type] == "fastly" {
elasticsearch {
host => [ "ES_HOSTNAME" ]
protocol => "transport"
cluster => "elasticsearch"
index => "fastly-%{+YYYY.MM.dd}"
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates.d/fastly.json"
template_name => "fastly"
}
}
else {
file {
path => "/var/log/logstash/unknown_messages.log"
}
}
}
Copy link

ghost commented Jan 27, 2016

Thanks JHMartin, can you please post your "template" files as well?

@rocky4bmw
Copy link

@jhmartin canyou share json templates please

@bhushanvinay
Copy link

bhushanvinay commented Nov 9, 2016

@jhmartin can you please share the json templates please. it will be a treat for us. Thank you very much for this.

@liangguo
Copy link

Great, I'm searching it for a longtime

@alexander-marquardt
Copy link

alexander-marquardt commented Sep 2, 2018

If you don't want to have the type fields in the documents that are output, you can also consider adding metadata which will be used for driving data to different outputs, and removing the 'type' fields. I have written a blog post that demonstrates this approach at https://alexmarquardt.com/2018/08/31/using-logstash-to-drive-filtered-data-from-a-single-source-into-multiple-output-destinations/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment