Skip to content

Instantly share code, notes, and snippets.

@tulios

tulios/KafkaJS_ECONNRESET_retry.md

Last active February 27, 2019 13:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tulios/2bbb08863978153fae81143a86e6e8d3 to your computer and use it in GitHub Desktop.
Save tulios/2bbb08863978153fae81143a86e6e8d3 to your computer and use it in GitHub Desktop.
Download ZIP
KafkaJS_ECONNRESET_retry.md
Raw
KafkaJS_ECONNRESET_retry.md

Test 1)

Forcing the seed broker to fail with ECONNRESET. The consumer example on the repo was changed to:

const kafka = new Kafka({
  brokers: ['127.0.0.1:9094'],
  // other configs
})

Command to reset the socket connections:

sudo tcpkill -i lo0 -9 host 127.0.0.1 and port 9094

Result:

KAFKAJS_LOG_LEVEL=debug node examples/consumer.js
{"level":"DEBUG","timestamp":"2019-02-27T13:02:45.251Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9094","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:02:45.260Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9094","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:02:45.261Z","logger":"kafkajs","message":"[BrokerPool] KafkaJSConnectionError: Connection error: read ECONNRESET","retryCount":0,"retryTime":328}
{"level":"ERROR","timestamp":"2019-02-27T13:02:45.262Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection error: read ECONNRESET","retryCount":0,"retryTime":328}
{"level":"DEBUG","timestamp":"2019-02-27T13:02:45.593Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9094","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:02:45.595Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9094","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:02:45.595Z","logger":"kafkajs","message":"[BrokerPool] KafkaJSConnectionError: Connection error: read ECONNRESET","retryCount":1,"retryTime":726}
{"level":"ERROR","timestamp":"2019-02-27T13:02:45.595Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection error: read ECONNRESET","retryCount":1,"retryTime":726}
{"level":"DEBUG","timestamp":"2019-02-27T13:02:46.323Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9094","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:02:46.324Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9094","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:02:46.324Z","logger":"kafkajs","message":"[BrokerPool] KafkaJSConnectionError: Connection error: read ECONNRESET","retryCount":2,"retryTime":1718}
{"level":"ERROR","timestamp":"2019-02-27T13:02:46.325Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection error: read ECONNRESET","retryCount":2,"retryTime":1718}
{"level":"DEBUG","timestamp":"2019-02-27T13:02:48.044Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9094","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:02:48.045Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9094","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:02:48.046Z","logger":"kafkajs","message":"[BrokerPool] KafkaJSConnectionError: Connection error: read ECONNRESET","retryCount":3,"retryTime":3074}
{"level":"ERROR","timestamp":"2019-02-27T13:02:48.046Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection error: read ECONNRESET","retryCount":3,"retryTime":3074}
{"level":"DEBUG","timestamp":"2019-02-27T13:02:51.123Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9094","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:02:51.125Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9094","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:02:51.125Z","logger":"kafkajs","message":"[BrokerPool] KafkaJSConnectionError: Connection error: read ECONNRESET","retryCount":4,"retryTime":6914}
{"level":"ERROR","timestamp":"2019-02-27T13:02:51.125Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection error: read ECONNRESET","retryCount":4,"retryTime":6914}
{"level":"DEBUG","timestamp":"2019-02-27T13:02:58.039Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9094","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:02:58.041Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9094","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:02:58.041Z","logger":"kafkajs","message":"[BrokerPool] KafkaJSConnectionError: Connection error: read ECONNRESET","retryCount":5,"retryTime":12676}
{"level":"ERROR","timestamp":"2019-02-27T13:02:58.041Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection error: read ECONNRESET","retryCount":5,"retryTime":12676}
[example/consumer] Connection error: read ECONNRESET { KafkaJSNumberOfRetriesExceeded
  Caused by: KafkaJSConnectionError: Connection error: read ECONNRESET
    at TLSSocket.onError (/Users/tulio.ornelas/Projects/github/kafkajs/src/network/connection.js:132:23)
    at emitOne (events.js:116:13)
    at TLSSocket.emit (events.js:211:7)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
  name: 'KafkaJSNumberOfRetriesExceeded',
  retriable: false,
  originalError:
   { KafkaJSConnectionError: Connection error: read ECONNRESET
    at TLSSocket.onError (/Users/tulio.ornelas/Projects/github/kafkajs/src/network/connection.js:132:23)
    at emitOne (events.js:116:13)
    at TLSSocket.emit (events.js:211:7)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
     name: 'KafkaJSConnectionError',
     retriable: true,
     broker: '127.0.0.1:9094',
     code: 'ECONNRESET' },
  retryCount: 5,
  retryTime: 12676 }

The connection is retried but eventually gives up since this is the initial connection. The number of retries can be configured.

Test 2)

Forcing one of the brokers to fail with ECONNRESET after the initial connection. It uses the same client from the previous test.

Command to reset the socket connections:

sudo tcpkill -i lo0 -9 host 127.0.0.1 and port 9097

Result:

KAFKAJS_LOG_LEVEL=debug node examples/consumer.js
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.551Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9094","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.586Z","logger":"kafkajs","message":"[Connection] Request ApiVersions(key: 18, version: 2)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":0,"expectResponse":true,"size":30}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.591Z","logger":"kafkajs","message":"[Connection] Response ApiVersions(key: 18, version: 2)","broker":"127.0.0.1:9094","clientId":"example-consumer","error":"The version of API is not supported","correlationId":0,"payload":{"type":"Buffer","data":"[filtered]"}}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.592Z","logger":"kafkajs","message":"[Connection] Request ApiVersions(key: 18, version: 1)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":1,"expectResponse":true,"size":30}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.594Z","logger":"kafkajs","message":"[Connection] Response ApiVersions(key: 18, version: 1)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":1,"size":218,"data":{"errorCode":0,"apiVersions":[{"apiKey":0,"minVersion":0,"maxVersion":3},{"apiKey":1,"minVersion":0,"maxVersion":5},{"apiKey":2,"minVersion":0,"maxVersion":2},{"apiKey":3,"minVersion":0,"maxVersion":4},{"apiKey":4,"minVersion":0,"maxVersion":0},{"apiKey":5,"minVersion":0,"maxVersion":0},{"apiKey":6,"minVersion":0,"maxVersion":3},{"apiKey":7,"minVersion":1,"maxVersion":1},{"apiKey":8,"minVersion":0,"maxVersion":3},{"apiKey":9,"minVersion":0,"maxVersion":3},{"apiKey":10,"minVersion":0,"maxVersion":1},{"apiKey":11,"minVersion":0,"maxVersion":2},{"apiKey":12,"minVersion":0,"maxVersion":1},{"apiKey":13,"minVersion":0,"maxVersion":1},{"apiKey":14,"minVersion":0,"maxVersion":1},{"apiKey":15,"minVersion":0,"maxVersion":1},{"apiKey":16,"minVersion":0,"maxVersion":1},{"apiKey":17,"minVersion":0,"maxVersion":0},{"apiKey":18,"minVersion":0,"maxVersion":1},{"apiKey":19,"minVersion":0,"maxVersion":2},{"apiKey":20,"minVersion":0,"maxVersion":1},{"apiKey":21,"minVersion":0,"maxVersion":0},{"apiKey":22,"minVersion":0,"maxVersion":0},{"apiKey":23,"minVersion":0,"maxVersion":0},{"apiKey":24,"minVersion":0,"maxVersion":0},{"apiKey":25,"minVersion":0,"maxVersion":0},{"apiKey":26,"minVersion":0,"maxVersion":0},{"apiKey":27,"minVersion":0,"maxVersion":0},{"apiKey":28,"minVersion":0,"maxVersion":0},{"apiKey":29,"minVersion":0,"maxVersion":0},{"apiKey":30,"minVersion":0,"maxVersion":0},{"apiKey":31,"minVersion":0,"maxVersion":0},{"apiKey":32,"minVersion":0,"maxVersion":0},{"apiKey":33,"minVersion":0,"maxVersion":0}],"throttleTime":0}}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.595Z","logger":"kafkajs","message":"[Broker] Verified support for SaslAuthenticate","broker":"127.0.0.1:9094","supportAuthenticationProtocol":false}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.599Z","logger":"kafkajs","message":"[Connection] Request SaslHandshake(key: 17, version: 0)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":2,"expectResponse":true,"size":37}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.601Z","logger":"kafkajs","message":"[Connection] Response SaslHandshake(key: 17, version: 0)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":2,"size":47,"data":{"errorCode":0,"enabledMechanisms":["PLAIN","SCRAM-SHA-512","SCRAM-SHA-256"]}}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.602Z","logger":"kafkajs","message":"[SASLPlainAuthenticator] Authenticate with SASL PLAIN","broker":"127.0.0.1:9094"}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.604Z","logger":"kafkajs","message":"[SASLPlainAuthenticator] SASL PLAIN authentication successful","broker":"127.0.0.1:9094"}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.610Z","logger":"kafkajs","message":"[Connection] Request Metadata(key: 3, version: 4)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":3,"expectResponse":true,"size":47}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.613Z","logger":"kafkajs","message":"[Connection] Response Metadata(key: 3, version: 4)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":3,"size":291,"data":{"throttleTime":0,"brokers":[{"nodeId":2,"host":"127.0.0.1","port":9100,"rack":null},{"nodeId":1,"host":"127.0.0.1","port":9097,"rack":null},{"nodeId":0,"host":"127.0.0.1","port":9094,"rack":null}],"clusterId":"ocWo0aD1RT-SlRNTr9R9yA","controllerId":2,"topicMetadata":[{"topicErrorCode":0,"topic":"topic-test","isInternal":false,"partitionMetadata":[{"partitionErrorCode":0,"partitionId":2,"leader":2,"replicas":[2],"isr":[2]},{"partitionErrorCode":0,"partitionId":5,"leader":2,"replicas":[2],"isr":[2]},{"partitionErrorCode":0,"partitionId":4,"leader":1,"replicas":[1],"isr":[1]},{"partitionErrorCode":0,"partitionId":1,"leader":1,"replicas":[1],"isr":[1]},{"partitionErrorCode":0,"partitionId":3,"leader":0,"replicas":[0],"isr":[0]},{"partitionErrorCode":0,"partitionId":0,"leader":0,"replicas":[0],"isr":[0]}]}]}}
{"level":"INFO","timestamp":"2019-02-27T13:03:56.615Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"test-group"}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.616Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:03:56.619Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:03:56.620Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":0,"retryTime":267}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.893Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:03:56.895Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:03:56.895Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":1,"retryTime":492}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:57.392Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:03:57.393Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:03:57.393Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":2,"retryTime":930}
{"level":"DEBUG","timestamp":"2019-02-27T13:03:58.324Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:03:58.325Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:03:58.326Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":3,"retryTime":2022}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:00.353Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:04:00.355Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:04:00.355Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":4,"retryTime":4288}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:04.649Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:04:04.650Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:04:04.651Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":5,"retryTime":8040}
{"level":"ERROR","timestamp":"2019-02-27T13:04:04.652Z","logger":"kafkajs","message":"[Consumer] Crash: KafkaJSNumberOfRetriesExceeded: Connection error: read ECONNRESET","groupId":"test-group","retryCount":5,"stack":"KafkaJSNumberOfRetriesExceeded\n  Caused by: KafkaJSConnectionError: Connection error: read ECONNRESET\n    at TLSSocket.onError (/Users/tulio.ornelas/Projects/github/kafkajs/src/network/connection.js:132:23)\n    at emitOne (events.js:116:13)\n    at TLSSocket.emit (events.js:211:7)\n    at emitErrorNT (internal/streams/destroy.js:64:8)\n    at _combinedTickCallback (internal/process/next_tick.js:138:11)\n    at process._tickCallback (internal/process/next_tick.js:180:9)"}
{"level":"INFO","timestamp":"2019-02-27T13:04:04.652Z","logger":"kafkajs","message":"[Consumer] Stopped","groupId":"test-group"}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:04.652Z","logger":"kafkajs","message":"[Consumer] consumer has stopped, disconnecting","groupId":"test-group"}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:04.652Z","logger":"kafkajs","message":"[Connection] disconnecting...","broker":"127.0.0.1:9094","clientId":"example-consumer"}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:04.653Z","logger":"kafkajs","message":"[Connection] disconnected","broker":"127.0.0.1:9094","clientId":"example-consumer"}
{"level":"ERROR","timestamp":"2019-02-27T13:04:04.653Z","logger":"kafkajs","message":"[Consumer] Restarting the consumer in 8040ms","retryCount":5,"groupId":"test-group"}
{"level":"INFO","timestamp":"2019-02-27T13:04:12.698Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"test-group"}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:12.698Z","logger":"kafkajs","message":"[Cluster] No brokers in the broker pool, refreshing metadata and trying again...","groupId":"test-group","retryCount":0,"retryTime":351}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:12.699Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9094","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:12.731Z","logger":"kafkajs","message":"[Connection] Request SaslHandshake(key: 17, version: 0)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":4,"expectResponse":true,"size":37}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:12.733Z","logger":"kafkajs","message":"[Connection] Response SaslHandshake(key: 17, version: 0)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":4,"size":47,"data":{"errorCode":0,"enabledMechanisms":["PLAIN","SCRAM-SHA-512","SCRAM-SHA-256"]}}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:12.733Z","logger":"kafkajs","message":"[SASLPlainAuthenticator] Authenticate with SASL PLAIN","broker":"127.0.0.1:9094"}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:12.735Z","logger":"kafkajs","message":"[SASLPlainAuthenticator] SASL PLAIN authentication successful","broker":"127.0.0.1:9094"}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:12.735Z","logger":"kafkajs","message":"[Connection] Request Metadata(key: 3, version: 4)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":5,"expectResponse":true,"size":47}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:12.739Z","logger":"kafkajs","message":"[Connection] Response Metadata(key: 3, version: 4)","broker":"127.0.0.1:9094","clientId":"example-consumer","correlationId":5,"size":291,"data":{"throttleTime":0,"brokers":[{"nodeId":2,"host":"127.0.0.1","port":9100,"rack":null},{"nodeId":1,"host":"127.0.0.1","port":9097,"rack":null},{"nodeId":0,"host":"127.0.0.1","port":9094,"rack":null}],"clusterId":"ocWo0aD1RT-SlRNTr9R9yA","controllerId":2,"topicMetadata":[{"topicErrorCode":0,"topic":"topic-test","isInternal":false,"partitionMetadata":[{"partitionErrorCode":0,"partitionId":2,"leader":2,"replicas":[2],"isr":[2]},{"partitionErrorCode":0,"partitionId":5,"leader":2,"replicas":[2],"isr":[2]},{"partitionErrorCode":0,"partitionId":4,"leader":1,"replicas":[1],"isr":[1]},{"partitionErrorCode":0,"partitionId":1,"leader":1,"replicas":[1],"isr":[1]},{"partitionErrorCode":0,"partitionId":3,"leader":0,"replicas":[0],"isr":[0]},{"partitionErrorCode":0,"partitionId":0,"leader":0,"replicas":[0],"isr":[0]}]}]}}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:13.092Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:04:13.093Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:04:13.094Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":0,"retryTime":324}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:13.418Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:04:13.420Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:04:13.420Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":1,"retryTime":602}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:14.023Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:04:14.024Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:04:14.024Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":2,"retryTime":1400}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:15.425Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:04:15.427Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:04:15.427Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":3,"retryTime":3238}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:18.668Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:04:18.669Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:04:18.670Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":4,"retryTime":5524}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:24.195Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true}
{"level":"ERROR","timestamp":"2019-02-27T13:04:24.196Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n    at _errnoException (util.js:992:11)\n    at TLSWrap.onread (net.js:618:25)"}
{"level":"ERROR","timestamp":"2019-02-27T13:04:24.196Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":5,"retryTime":10950}
{"level":"ERROR","timestamp":"2019-02-27T13:04:24.197Z","logger":"kafkajs","message":"[Consumer] Crash: KafkaJSNumberOfRetriesExceeded: Connection error: read ECONNRESET","groupId":"test-group","retryCount":5,"stack":"KafkaJSNumberOfRetriesExceeded\n  Caused by: KafkaJSConnectionError: Connection error: read ECONNRESET\n    at TLSSocket.onError (/Users/tulio.ornelas/Projects/github/kafkajs/src/network/connection.js:132:23)\n    at emitOne (events.js:116:13)\n    at TLSSocket.emit (events.js:211:7)\n    at emitErrorNT (internal/streams/destroy.js:64:8)\n    at _combinedTickCallback (internal/process/next_tick.js:138:11)\n    at process._tickCallback (internal/process/next_tick.js:180:9)"}
{"level":"INFO","timestamp":"2019-02-27T13:04:24.197Z","logger":"kafkajs","message":"[Consumer] Stopped","groupId":"test-group"}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:24.197Z","logger":"kafkajs","message":"[Consumer] consumer has stopped, disconnecting","groupId":"test-group"}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:24.197Z","logger":"kafkajs","message":"[Connection] disconnecting...","broker":"127.0.0.1:9094","clientId":"example-consumer"}
{"level":"DEBUG","timestamp":"2019-02-27T13:04:24.197Z","logger":"kafkajs","message":"[Connection] disconnected","broker":"127.0.0.1:9094","clientId":"example-consumer"}
{"level":"ERROR","timestamp":"2019-02-27T13:04:24.197Z","logger":"kafkajs","message":"[Consumer] Restarting the consumer in 10950ms","retryCount":5,"groupId":"test-group"}

The connection to the broker fails but the client keeps retrying:

{"level":"DEBUG","timestamp":"2019-02-27T13:03:56.616Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"127.0.0.1:9097","clientId":"example-consumer","ssl":true,"sasl":true} {"level":"ERROR","timestamp":"2019-02-27T13:03:56.619Z","logger":"kafkajs","message":"[Connection] Connection error: read ECONNRESET","broker":"127.0.0.1:9097","clientId":"example-consumer","stack":"Error: read ECONNRESET\n at _errnoException (util.js:992:11)\n at TLSWrap.onread (net.js:618:25)"} {"level":"ERROR","timestamp":"2019-02-27T13:03:56.620Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to broker, reconnecting","retryCount":0,"retryTime":267}

Eventually the consumer crashes and restarts

{"level":"ERROR","timestamp":"2019-02-27T13:04:04.652Z","logger":"kafkajs","message":"[Consumer] Crash: KafkaJSNumberOfRetriesExceeded: Connection error: read ECONNRESET","groupId":"test-group","retryCount":5,"stack":"KafkaJSNumberOfRetriesExceeded\n Caused by: KafkaJSConnectionError: Connection error: read ECONNRESET\n at TLSSocket.onError (/Users/tulio.ornelas/Projects/github/kafkajs/src/network/connection.js:132:23)\n at emitOne (events.js:116:13)\n at TLSSocket.emit (events.js:211:7)\n at emitErrorNT (internal/streams/destroy.js:64:8)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickCallback (internal/process/next_tick.js:180:9)"} {"level":"INFO","timestamp":"2019-02-27T13:04:04.652Z","logger":"kafkajs","message":"[Consumer] Stopped","groupId":"test-group"} {"level":"DEBUG","timestamp":"2019-02-27T13:04:04.652Z","logger":"kafkajs","message":"[Consumer] consumer has stopped, disconnecting","groupId":"test-group"} {"level":"DEBUG","timestamp":"2019-02-27T13:04:04.652Z","logger":"kafkajs","message":"[Connection] disconnecting...","broker":"127.0.0.1:9094","clientId":"example-consumer"} {"level":"DEBUG","timestamp":"2019-02-27T13:04:04.653Z","logger":"kafkajs","message":"[Connection] disconnected","broker":"127.0.0.1:9094","clientId":"example-consumer"} {"level":"ERROR","timestamp":"2019-02-27T13:04:04.653Z","logger":"kafkajs","message":"[Consumer] Restarting the consumer in 8040ms","retryCount":5,"groupId":"test-group"} {"level":"INFO","timestamp":"2019-02-27T13:04:12.698Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"test-group"}

But it never dies, it keeps retrying until it succeeds. If tcpkill is disabled the client will manage to reconnect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment