Skip to content

Instantly share code, notes, and snippets.

@elmer
Forked from lusis/A.md
Created June 1, 2012 08:01
Show Gist options
  • Save elmer/2850108 to your computer and use it in GitHub Desktop.
Save elmer/2850108 to your computer and use it in GitHub Desktop.
dirt simple basic config for talking to an external elasticsearch server

The thing that trips most people up is that the parameters to the web cli (--backend elasticsearch://blah:9300/blah) do NOT set the output destination.

The web app is its own process with its own args. It knows nothing about the agent config file.

Remember that you could simply use the same jar like so:

java -jar logstash.jar web --backend elasticsearch://elasticsearch:9300/clustername

and run the web interface without the agent process.

Port numbers and transport client

What also trips people up is port assignments. Logstash connects to elasticsearch as a transport client. This means it becomes a part of the elasticsearch cluster but does not store any data.

If you're running agent, web and an external ES process on the same machine, all three processes will try and use port 9300. This won't work. The problem is that it MIGHT work depending on startup order. If you start ES first, it will grab port 9300. When logstash comes up, the agent and web processes will increment to 9301 and 9302.

Now let's imagine you shut everything down and start logstash first. Likely agent will get 9300 and web will get 9301. I'm not positive here but ES will I THINK jump to 9302. The problem is that you've told the web to talk to 9300 which is now the agent.

The best bet here if you're running on the same machine is to lock ES to a different port and use that. That way there's never an issue. This could cause problems when you start to cluster though. So really if you're running ES externally from logstash (i.e. not embedded) run it on a different instance/machine.

Quick note on versions

Elasticsearch maintains compatibility only between minor versions. All clients and servers connecting via transport MUST be on the same major version.

This means that 0.18.x nodes can talk to a 0.18.y server regardless of what x and y are. A 0.19.x client or server cannot talk to a 0.18.x client or server.

Long story short

If you're trying to run ES on the same machine as your logstash web or agent process, just use the embedded ES. You aren't buying yourself anything.

java -jar logstash.jar agent -f logstash.conf -- web --backend elasticsearch://elasticsearchip:9300/clustername
input { stdin { type => "stdin-test" } }
output { elasticsearch { host => "host-or-ip-of-es-server" } }
# Check elasticsearch.yml. If there's a custom cluster name there that's NOT logstash
# use this instead
#output { elasticsearch { host => "host-or-ip-of-es-server" cluster => "name-from-elastic-search-config-file"} }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment