Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@bunste

bunste/data_node.log

Created October 14, 2021 10:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bunste/5fcc43a4d6bc37527b8a0a8eaf0aaf85 to your computer and use it in GitHub Desktop.
Save bunste/5fcc43a4d6bc37527b8a0a8eaf0aaf85 to your computer and use it in GitHub Desktop.
Download ZIP
Log files for elasticsearch crash
Raw
data_node.log
[2021-10-13T14:15:08,671][TRACE][o.e.c.s.ClusterApplierService] [es-node05-a] connecting to nodes of cluster state with version 5429193
[2021-10-13T14:15:08,671][DEBUG][o.e.c.s.ClusterApplierService] [es-node05-a] applying settings from cluster state with version 5429193
[2021-10-13T14:15:08,671][DEBUG][o.e.c.s.ClusterApplierService] [es-node05-a] apply cluster state with version 5429193
[2021-10-13T14:15:08,671][TRACE][o.e.c.s.ClusterApplierService] [es-node05-a] calling [org.elasticsearch.repositories.RepositoriesService@4a0995e7] with change to version [5429193]
[2021-10-13T14:15:08,671][TRACE][o.e.c.s.ClusterApplierService] [es-node05-a] calling [org.elasticsearch.indices.cluster.IndicesClusterStateService@c418f17] with change to version [5429193]
[2021-10-13T14:17:09,136][INFO ][o.e.i.r.PeerRecoveryTargetService] [es-node05-a] recovery of [events-2021.10.05][0] from [{es-node06-a}{R00-RxIuQGud85biVet1XA}{rPoopo8EQQCMkkaXnIu0Xg}{192.168.200.185}{192.168.200.185:19301}{cdhilrstw}{ml.machine_memory=67085619200, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.185, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es-node05-a][192.168.200.184:19301][internal:index/shard/recovery/file_chunk] disconnected]
[2021-10-13T14:17:09,185][INFO ][o.e.i.r.PeerRecoveryTargetService] [es-node05-a] recovery of [groot_news_bucket_23_v3][0] from [{es-node03-a}{O_DOHlu7QqChJNdkgZHtbQ}{6F2-uSzeSka08lRdaE2VIw}{192.168.200.182}{192.168.200.182:19301}{cdhilrstw}{ml.machine_memory=135073177600, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.182, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es-node05-a][192.168.200.184:19301][internal:index/shard/recovery/file_chunk] disconnected]
[2021-10-13T14:17:11,607][INFO ][o.e.c.c.Coordinator ] [es-node05-a] master node [{es-master02}{RLYFvvrgSCymDsgMDbx-dw}{LVDIbVeiQ4qtVEg8CaWq6w}{192.168.200.52}{192.168.200.52:9300}{ilmr}{ml.machine_memory=8376066048, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{es-master02}{RLYFvvrgSCymDsgMDbx-dw}{LVDIbVeiQ4qtVEg8CaWq6w}{192.168.200.52}{192.168.200.52:9300}{ilmr}{ml.machine_memory=8376066048, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1181) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1181) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:277) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:224) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:275) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:267) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:131) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [es-master02][192.168.200.52:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{es-node05-a}{KEWUUnwASICujt430nUhOA}{0nN2CmrJT5Wsape0a_OxgA}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}] has been removed from the cluster
Raw
data_node_config.yml
# Ansible managed
node.name: es-node05-a
node.attr.machine: 192.168.6.184
node.attr.disks: ssd
cluster.name: wilma_van_der_heel
node.master: false
node.data: true
http.port: 19201
http.host:
- 192.168.6.184
- 127.0.0.1
transport.port: 19301
transport.host:
- 192.168.200.184
- 127.0.0.1
# Don't put replicas on the same physical host (works together with node.attr.machine)
cluster.routing.allocation.awareness.attributes: machine
# Disable swapping
bootstrap.memory_lock: false # our data nodes have no swap space configured and don't need mlockall
# For security, don't allow delete operation with wildcards
action.destructive_requires_name: true
http.cors.enabled: true
http.cors.allow-origin: "*"
http.max_initial_line_length: 8kb
discovery.seed_hosts: ["192.168.200.51:9300", "192.168.200.52:9300", "192.168.200.53:9300"]
path.data:
- /var/lib/elasticsearch/es-node05-a/1
- /var/lib/elasticsearch/es-node05-a/2
- /var/lib/elasticsearch/es-node05-a/3
- /var/lib/elasticsearch/es-node05-a/4
- /var/lib/elasticsearch/es-node05-a/5
- /var/lib/elasticsearch/es-node05-a/6
path.logs: /var/log/elasticsearch/es-node05-a
gateway.recover_after_time: 5m
gateway.recover_after_master_nodes: 2
gateway.recover_after_data_nodes: 3
gateway.expected_master_nodes: 3
gateway.expected_data_nodes: 6
# Performance improvements
indices.recovery.max_bytes_per_sec: 100mb
cluster.routing.allocation.node_concurrent_recoveries: 4
cluster.routing.allocation.cluster_concurrent_rebalance: 8
cluster.routing.allocation.enable: all
# Joining queries requires to use a lot of terms
indices.query.bool.max_clause_count: 100000
# Allow regular expressions for update_by_query via Painless
# https://www.elastic.co/guide/en/elasticsearch/painless/6.7/painless-examples.html
script.painless.regex.enabled: true
search.max_buckets: 1000000
Raw
master_node.log
2021-10-14T11:03:30,026][INFO ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [9.9s] publication of cluster state version [5430424] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:03:50,053][WARN ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [30s] publication of cluster state version [5430424] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:04:12,298][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585102] overhead, spent [299ms] collecting in the last [1s]
[2021-10-14T11:04:42,444][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585132] overhead, spent [311ms] collecting in the last [1s]
[2021-10-14T11:04:53,449][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585143] overhead, spent [312ms] collecting in the last [1s]
[2021-10-14T11:05:19,461][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585169] overhead, spent [253ms] collecting in the last [1s]
[2021-10-14T11:05:20,053][WARN ][o.e.c.c.LagDetector ] [es-master02] node [{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}] is lagging at cluster state version [5430423], although publication of cluster state version [5430424] completed [1.5m] ago
[2021-10-14T11:05:23,455][INFO ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [10.1s] publication of cluster state version [5430425] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:05:23,528][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585173] overhead, spent [316ms] collecting in the last [1s]
[2021-10-14T11:05:33,556][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585183] overhead, spent [266ms] collecting in the last [1s]
[2021-10-14T11:05:43,504][WARN ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [30.1s] publication of cluster state version [5430425] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:05:43,676][INFO ][o.e.c.s.MasterService ] [es-master02] node-left[{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} reason: lagging], term: 9, version: 5430426, delta: removed {{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}}
[2021-10-14T11:05:43,896][INFO ][o.e.c.s.ClusterApplierService] [es-master02] removed {{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}}, term: 9, version: 5430426, reason: Publication{term=9, version=5430426}
[2021-10-14T11:05:43,927][INFO ][o.e.c.r.DelayedAllocationService] [es-master02] scheduling reroute for delayed shards in [59.5s] (15 delayed shards)
[2021-10-14T11:05:44,016][WARN ][o.e.c.r.a.AllocationService] [es-master02] [keyword_statistics-2021.10][0] marking unavailable shards as stale: [Icrrqj16T5iMn4JtLlUGtw]
[2021-10-14T11:05:46,744][INFO ][o.e.c.s.MasterService ] [es-master02] node-join[{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} join existing leader], term: 9, version: 5430429, delta: added {{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}}
[2021-10-14T11:05:56,919][INFO ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [10.1s] publication of cluster state version [5430429] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:06:16,919][INFO ][o.e.c.s.ClusterApplierService] [es-master02] added {{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}}, term: 9, version: 5430429, reason: Publication{term=9, version=5430429}
[2021-10-14T11:06:17,007][WARN ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [30.2s] publication of cluster state version [5430429] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:06:17,109][WARN ][o.e.c.r.a.AllocationService] [es-master02] [groot_news_bucket_18_v3][0] marking unavailable shards as stale: [yJohWFMyRNisYU7tJ1Cp5g]
[2021-10-14T11:06:27,180][INFO ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [10s] publication of cluster state version [5430430] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:06:32,957][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585242] overhead, spent [289ms] collecting in the last [1s]
[2021-10-14T11:06:43,962][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585253] overhead, spent [351ms] collecting in the last [1s]
[2021-10-14T11:06:47,213][WARN ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [30s] publication of cluster state version [5430430] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:06:51,965][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585261] overhead, spent [263ms] collecting in the last [1s]
[2021-10-14T11:06:57,358][INFO ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [10.1s] publication of cluster state version [5430431] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:06:59,968][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585269] overhead, spent [282ms] collecting in the last [1s]
[2021-10-14T11:07:02,969][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585272] overhead, spent [284ms] collecting in the last [1s]
[2021-10-14T11:07:17,390][WARN ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [30s] publication of cluster state version [5430431] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:07:27,418][INFO ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [10.1s] publication of cluster state version [5430432] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:07:47,007][WARN ][o.e.c.c.LagDetector ] [es-master02] node [{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}] is lagging at cluster state version [0], although publication of cluster state version [5430429] completed [1.5m] ago
[2021-10-14T11:07:47,032][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585316] overhead, spent [304ms] collecting in the last [1s]
[2021-10-14T11:07:47,441][WARN ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [30.1s] publication of cluster state version [5430432] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:07:47,504][INFO ][o.e.c.s.MasterService ] [es-master02] node-left[{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} reason: lagging], term: 9, version: 5430433, delta: removed {{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}}
[2021-10-14T11:07:47,864][INFO ][o.e.c.s.ClusterApplierService] [es-master02] removed {{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}}, term: 9, version: 5430433, reason: Publication{term=9, version=5430433}
[2021-10-14T11:07:47,895][INFO ][o.e.c.r.DelayedAllocationService] [es-master02] scheduling reroute for delayed shards in [59.5s] (4 delayed shards)
[2021-10-14T11:07:47,896][WARN ][o.e.c.r.a.AllocationService] [es-master02] [events-long-term-2017.11-v2][0] marking unavailable shards as stale: [LTaHQEUXT4eAWF_wIkeheA]
[2021-10-14T11:07:47,896][WARN ][o.e.c.r.a.AllocationService] [es-master02] [news-2023.17][0] marking unavailable shards as stale: [dvAibW4WRYiHc1xzEUiIlw]
[2021-10-14T11:07:47,896][WARN ][o.e.c.r.a.AllocationService] [es-master02] [news-2021.46][1] marking unavailable shards as stale: [NTBo9qWtQ9mUrt6T2JSIrA]
[2021-10-14T11:07:48,116][WARN ][o.e.c.r.a.AllocationService] [es-master02] [events-long-term-2019.10][0] marking unavailable shards as stale: [HueqjVwgSpGNNSvFQHCi1Q]
[2021-10-14T11:07:48,116][WARN ][o.e.c.r.a.AllocationService] [es-master02] [news-2023.25][0] marking unavailable shards as stale: [CaGPoZlGTRm7XpKMGZIQBQ]
[2021-10-14T11:07:49,040][WARN ][o.e.c.r.a.AllocationService] [es-master02] [prnews_news-2021.03][0] marking unavailable shards as stale: [iGuGI5Z6QEiD6shX-eJOgA]
[2021-10-14T11:07:49,474][WARN ][o.e.c.r.a.AllocationService] [es-master02] [news-2022.44][0] marking unavailable shards as stale: [MVai_gOOSSyGIRpycMznqg]
[2021-10-14T11:07:49,474][WARN ][o.e.c.r.a.AllocationService] [es-master02] [news-2023.35][1] marking unavailable shards as stale: [x_WzXOVZTKa-Z5Jwv82i5A]
[2021-10-14T11:07:50,200][INFO ][o.e.c.s.MasterService ] [es-master02] node-join[{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} join existing leader], term: 9, version: 5430443, delta: added {{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}}
[2021-10-14T11:08:00,324][INFO ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [10s] publication of cluster state version [5430443] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:08:13,212][INFO ][o.e.m.j.JvmGcMonitorService] [es-master02] [gc][23585342] overhead, spent [295ms] collecting in the last [1s]
[2021-10-14T11:08:20,324][INFO ][o.e.c.s.ClusterApplierService] [es-master02] added {{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}}, term: 9, version: 5430443, reason: Publication{term=9, version=5430443}
[2021-10-14T11:08:20,377][WARN ][o.e.c.c.C.CoordinatorPublication] [es-master02] after [29.9s] publication of cluster state version [5430443] is still waiting for {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} [SENT_APPLY_COMMIT]
[2021-10-14T11:08:22,298][WARN ][o.e.c.NodeConnectionsService] [es-master02] failed to connect to {es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [es-node05-a][192.168.200.184:19301] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:978) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:198) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.10.1.jar:7.10.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 192.168.200.184/192.168.200.184:19301
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
... 7 more
[2021-10-14T11:08:22,633][INFO ][o.e.c.s.MasterService ] [es-master02] node-left[{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true} reason: disconnected], term: 9, version: 5430445, delta: removed {{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}}
[2021-10-14T11:08:22,774][INFO ][o.e.c.s.ClusterApplierService] [es-master02] removed {{es-node05-a}{AcV5mJL5T-SPk4LiV7x2qA}{IFo3Hlz0QjaPjhlXc3xfmQ}{192.168.200.184}{192.168.200.184:19301}{cdhilrstw}{ml.machine_memory=67185217536, ml.max_open_jobs=20, xpack.installed=true, disks=ssd, machine=192.168.6.184, transform.node=true}}, term: 9, version: 5430445, reason: Publication{term=9, version=5430445}
[2021-10-14T11:08:22,815][INFO ][o.e.c.r.DelayedAllocationService] [es-master02] scheduling reroute for delayed shards in [59.5s] (4 delayed shards)
Raw
master_node_config.yml
# Ansible managed
node.name: es-master02
cluster.name: wilma_van_der_heel
node.master: true
node.data: false
http.port: 9200
http.host:
- 192.168.6.52
- 127.0.0.1
transport.port: 9300
transport.host:
- 192.168.200.52
- 127.0.0.1
# Don't put replicas on the same physical host (works together with node.attr.machine)
cluster.routing.allocation.awareness.attributes: machine
# Disable swapping
bootstrap.memory_lock: true
# For security, don't allow delete operation with wildcards
action.destructive_requires_name: true
http.cors.enabled: true
http.cors.allow-origin: "*"
http.max_initial_line_length: 8kb
discovery.seed_hosts: ["192.168.200.51:9300", "192.168.200.52:9300", "192.168.200.53:9300"]
path.data:
- /var/lib/elasticsearch/es-master02/1
path.logs: /var/log/elasticsearch/es-master02
gateway.recover_after_time: 5m
gateway.recover_after_master_nodes: 2
gateway.recover_after_data_nodes: 3
gateway.expected_master_nodes: 3
gateway.expected_data_nodes: 6
# Performance improvements
indices.recovery.max_bytes_per_sec: 100mb
cluster.routing.allocation.node_concurrent_recoveries: 4
cluster.routing.allocation.cluster_concurrent_rebalance: 8
cluster.routing.allocation.enable: all
# Joining queries requires to use a lot of terms
indices.query.bool.max_clause_count: 100000
# Allow regular expressions for update_by_query via Painless
# https://www.elastic.co/guide/en/elasticsearch/painless/6.7/painless-examples.html
script.painless.regex.enabled: true
search.max_buckets: 65535
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment