RickardCardell

{
    "jobCounters": {
        "counterGroup": [
            {
                "counter": [
                    {
                        "mapCounterValue": 0,
                        "name": "FILE_BYTES_READ",
                        "reduceCounterValue": 292032,
                        "totalCounterValue": 292032

RickardCardell

Debugging bulk requests from Logstash to Elasticsearch

We use the ELK-stack to centralize the Hadoop logs. The logs are collected by Logstash and pushed to an Elasticsearch cluster.

• Logstash 2.1, 2.4
• Elasticsearch 1.7

Recently we have had issues with Logstash agents hanging upon sending bulk reqests to Elasticsearch. All its pipelines could stall stall for 15 minutes or even get stuck requiring a restart. Updating the Logstash agents to v2.4 did improve the situation a bit but we still had a lot of long lasting timeouts. Let's check the logs!

{:timestamp=>"2016-10-19T04:39:08.017000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '["http://elk1:9200", "http://elk2:9200", "http://elk3:9200", "http://elk4:9200", "http://elk5:9200"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection timed out", :class=>"Manticore::SocketException", :level=>:error}