Skip to content

Instantly share code, notes, and snippets.

@sai92messy

sai92messy/logs

Created July 22, 2021 16:04
Show Gist options
  • Save sai92messy/29cc9bb3a1f9393b8bd69d18d365a01b to your computer and use it in GitHub Desktop.
Save sai92messy/29cc9bb3a1f9393b8bd69d18d365a01b to your computer and use it in GitHub Desktop.
Download ZIP
Elasticsearch Temporarily Unavailable Logs
Raw
logs
...
19:28:12.991
elasticsearch.server
[elasticsearch.server][INFO] master node [{instance-0000000000}{rzyW_Ck_S4O8DOEALh3_bA}{LpAavRpOQVGJcWH-vLz7UA}{172.25.70.164}{172.25.70.164:19610}{himrst}{logical_availability_zone=zone-0, server_name=instance-0000000000.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1b, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}] failed, restarting discovery
19:28:12.993
elasticsearch.server
[elasticsearch.server][INFO] master node changed {previous [{instance-0000000000}{rzyW_Ck_S4O8DOEALh3_bA}{LpAavRpOQVGJcWH-vLz7UA}{172.25.70.164}{172.25.70.164:19610}{himrst}{logical_availability_zone=zone-0, server_name=instance-0000000000.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1b, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}], current []}, term: 4, version: 155103, reason: becoming candidate: onLeaderFailure
19:28:13.594
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 22001658376 ns, Reaching safepoint: 212482 ns, At safepoint: 9557 ns, Total: 222039 ns
19:28:13.628
elasticsearch.server
[elasticsearch.server][INFO] master node [{instance-0000000000}{rzyW_Ck_S4O8DOEALh3_bA}{LpAavRpOQVGJcWH-vLz7UA}{172.25.70.164}{172.25.70.164:19610}{himrst}{logical_availability_zone=zone-0, server_name=instance-0000000000.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1b, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}] failed, restarting discovery
19:28:13.636
elasticsearch.server
[elasticsearch.server][INFO] master node changed {previous [{instance-0000000000}{rzyW_Ck_S4O8DOEALh3_bA}{LpAavRpOQVGJcWH-vLz7UA}{172.25.70.164}{172.25.70.164:19610}{himrst}{logical_availability_zone=zone-0, server_name=instance-0000000000.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1b, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}], current []}, term: 4, version: 155103, reason: becoming candidate: onLeaderFailure
19:28:13.650
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 13984898607 ns, Reaching safepoint: 331559 ns, At safepoint: 13361 ns, Total: 344920 ns
19:28:13.765
elasticsearch.server
[elasticsearch.server][INFO] elected-as-master ([2] nodes joined)[{tiebreaker-0000000002}{XikbwK1mSZ-WrCo4FO9rNg}{N4b0ARvpSPu-J3hn7jaLoA}{172.25.151.102}{172.25.151.102:19855}{mv}{logical_availability_zone=tiebreaker, server_name=tiebreaker-0000000002.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1e, xpack.installed=true, data=hot, instance_configuration=aws.master.r5d, transform.node=false, region=us-east-1} elect leader, {instance-0000000001}{onstqMAoT1ySGTPkCVY-Yg}{zBGoKv7uSsWPTwo3dnrkSQ}{172.25.46.129}{172.25.46.129:19051}{himrst}{logical_availability_zone=zone-1, server_name=instance-0000000001.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1a, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 5, version: 155104, delta: master node changed {previous [], current [{instance-0000000001}{onstqMAoT1ySGTPkCVY-Yg}{zBGoKv7uSsWPTwo3dnrkSQ}{172.25.46.129}{172.25.46.129:19051}{himrst}{logical_availability_zone=zone-1, server_name=instance-0000000001.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1a, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}]}
19:28:13.822
elasticsearch.server
[elasticsearch.server][INFO] master node changed {previous [], current [{instance-0000000001}{onstqMAoT1ySGTPkCVY-Yg}{zBGoKv7uSsWPTwo3dnrkSQ}{172.25.46.129}{172.25.46.129:19051}{himrst}{logical_availability_zone=zone-1, server_name=instance-0000000001.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1a, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}]}, term: 5, version: 155104, reason: ApplyCommitRequest{term=5, version=155104, sourceNode={instance-0000000001}{onstqMAoT1ySGTPkCVY-Yg}{zBGoKv7uSsWPTwo3dnrkSQ}{172.25.46.129}{172.25.46.129:19051}{himrst}{logical_availability_zone=zone-1, server_name=instance-0000000001.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1a, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}}
19:28:14.594
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 1000145334 ns, Reaching safepoint: 196433 ns, At safepoint: 5801 ns, Total: 202234 ns
19:28:14.651
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 1000397444 ns, Reaching safepoint: 368177 ns, At safepoint: 9245 ns, Total: 377422 ns
19:28:15.651
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 1000270272 ns, Reaching safepoint: 255598 ns, At safepoint: 24877 ns, Total: 280475 ns
19:28:17.652
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 2000383310 ns, Reaching safepoint: 210625 ns, At safepoint: 10231 ns, Total: 220856 ns
19:28:20.653
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 3000557404 ns, Reaching safepoint: 191654 ns, At safepoint: 6779 ns, Total: 198433 ns
19:28:23.776
elasticsearch.server
[elasticsearch.server][INFO] after [10s] publication of cluster state version [155104] is still waiting for {instance-0000000000}{rzyW_Ck_S4O8DOEALh3_bA}{LpAavRpOQVGJcWH-vLz7UA}{172.25.70.164}{172.25.70.164:19610}{himrst}{logical_availability_zone=zone-0, server_name=instance-0000000000.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1b, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1} [SENT_PUBLISH_REQUEST]
19:28:24.654
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 4000785910 ns, Reaching safepoint: 263379 ns, At safepoint: 14923 ns, Total: 278302 ns
19:28:26.654
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 2000392318 ns, Reaching safepoint: 238708 ns, At safepoint: 10245 ns, Total: 248953 ns
19:28:31.655
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 5000823654 ns, Reaching safepoint: 295631 ns, At safepoint: 18358 ns, Total: 313989 ns
19:28:34.656
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 3000545580 ns, Reaching safepoint: 226338 ns, At safepoint: 13013 ns, Total: 239351 ns
19:28:36.657
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 2000378969 ns, Reaching safepoint: 257955 ns, At safepoint: 18730 ns, Total: 276685 ns
19:28:39.658
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 3000612322 ns, Reaching safepoint: 225065 ns, At safepoint: 6750 ns, Total: 231815 ns
19:28:41.658
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 2000331442 ns, Reaching safepoint: 192200 ns, At safepoint: 14256 ns, Total: 206456 ns
19:28:43.597
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 29002317321 ns, Reaching safepoint: 135333 ns, At safepoint: 6589 ns, Total: 141922 ns
19:28:43.659
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 2000413315 ns, Reaching safepoint: 241429 ns, At safepoint: 15764 ns, Total: 257193 ns
19:28:43.778
elasticsearch.server
[elasticsearch.server][INFO] master node changed {previous [], current [{instance-0000000001}{onstqMAoT1ySGTPkCVY-Yg}{zBGoKv7uSsWPTwo3dnrkSQ}{172.25.46.129}{172.25.46.129:19051}{himrst}{logical_availability_zone=zone-1, server_name=instance-0000000001.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1a, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}]}, term: 5, version: 155104, reason: Publication{term=5, version=155104}
19:28:43.862
elasticsearch.server
[elasticsearch.server][INFO] scheduling snapshot lifecycle job [cloud-snapshot-policy-1]
19:28:43.871
elasticsearch.server
[elasticsearch.server][WARN] after [30.2s] publication of cluster state version [155104] is still waiting for {instance-0000000000}{rzyW_Ck_S4O8DOEALh3_bA}{LpAavRpOQVGJcWH-vLz7UA}{172.25.70.164}{172.25.70.164:19610}{himrst}{logical_availability_zone=zone-0, server_name=instance-0000000000.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1b, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1} [SENT_PUBLISH_REQUEST]
19:28:44.660
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 1000289489 ns, Reaching safepoint: 429236 ns, At safepoint: 21280 ns, Total: 450516 ns
19:28:45.660
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 1000286755 ns, Reaching safepoint: 202395 ns, At safepoint: 25973 ns, Total: 228368 ns
19:28:45.798
elasticsearch.server
[elasticsearch.server][INFO] updating number_of_replicas to [0] for indices [.ds-.slm-history-5-2021.05.07-000002, metrics-endpoint.metadata_current_default, .ds-ilm-history-5-2021.06.06-000003, .kibana-event-log-7.12.0-000002, .security-tokens-7, .security-7, .apm-custom-link, .kibana_7.12.0_001, .kibana-event-log-7.12.0-000004, .tasks, .transform-notifications-000002, .kibana_security_session_1, .ds-ilm-history-5-2021.07.06-000004, .kibana-event-log-7.12.0-000001, .transform-internal-006, .ds-.slm-history-5-2021.04.07-000001, .ds-ilm-history-5-2021.05.07-000002, .kibana_task_manager_7.12.0_001, .kibana-event-log-7.12.0-000003, .ds-.slm-history-5-2021.07.06-000004, .apm-agent-configuration, .ds-.slm-history-5-2021.06.06-000003, .ds-ilm-history-5-2021.04.07-000001]
19:28:45.803
elasticsearch.server
[elasticsearch.server][INFO] node-left[{instance-0000000000}{rzyW_Ck_S4O8DOEALh3_bA}{LpAavRpOQVGJcWH-vLz7UA}{172.25.70.164}{172.25.70.164:19610}{himrst}{logical_availability_zone=zone-0, server_name=instance-0000000000.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1b, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1} reason: followers check retry count exceeded], term: 5, version: 155105, delta: removed {{instance-0000000000}{rzyW_Ck_S4O8DOEALh3_bA}{LpAavRpOQVGJcWH-vLz7UA}{172.25.70.164}{172.25.70.164:19610}{himrst}{logical_availability_zone=zone-0, server_name=instance-0000000000.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1b, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}}
19:28:45.835
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "ICBufferFull", Time since last: 174238285 ns, Reaching safepoint: 418060 ns, At safepoint: 18519 ns, Total: 436579 ns
19:28:45.844
elasticsearch.server
[elasticsearch.server][INFO] removed {{instance-0000000000}{rzyW_Ck_S4O8DOEALh3_bA}{LpAavRpOQVGJcWH-vLz7UA}{172.25.70.164}{172.25.70.164:19610}{himrst}{logical_availability_zone=zone-0, server_name=instance-0000000000.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1b, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}}, term: 5, version: 155105, reason: ApplyCommitRequest{term=5, version=155105, sourceNode={instance-0000000001}{onstqMAoT1ySGTPkCVY-Yg}{zBGoKv7uSsWPTwo3dnrkSQ}{172.25.46.129}{172.25.46.129:19051}{himrst}{logical_availability_zone=zone-1, server_name=instance-0000000001.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1a, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}}
19:28:45.846
elasticsearch.server
[elasticsearch.server][INFO] removed {{instance-0000000000}{rzyW_Ck_S4O8DOEALh3_bA}{LpAavRpOQVGJcWH-vLz7UA}{172.25.70.164}{172.25.70.164:19610}{himrst}{logical_availability_zone=zone-0, server_name=instance-0000000000.901115ae4294436f999506fdfa6048d1, availability_zone=us-east-1b, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-east-1}}, term: 5, version: 155105, reason: Publication{term=5, version=155105}
19:28:45.989
elasticsearch.server
[elasticsearch.server][INFO] [.siem-signals-default-000004][0] primary-replica resync completed with 0 operations
19:28:45.989
elasticsearch.server
[elasticsearch.server][INFO] [.security-7][0] primary-replica resync completed with 0 operations
19:28:45.989
elasticsearch.server
[elasticsearch.server][INFO] [metrics-endpoint.metadata_current_default][0] primary-replica resync completed with 0 operations
19:28:45.989
elasticsearch.server
[elasticsearch.server][INFO] [.kibana-event-log-7.12.0-000003][0] primary-replica resync completed with 0 operations
19:28:45.990
elasticsearch.server
[elasticsearch.server][INFO] [.security-tokens-7][0] primary-replica resync completed with 0 operations
19:28:45.990
elasticsearch.server
[elasticsearch.server][INFO] [.ds-.slm-history-5-2021.04.07-000001][0] primary-replica resync completed with 0 operations
19:28:45.990
elasticsearch.server
[elasticsearch.server][INFO] [.kibana_task_manager_7.12.0_001][0] primary-replica resync completed with 0 operations
19:28:45.990
elasticsearch.server
[elasticsearch.server][INFO] [.siem-signals-default-000003][0] primary-replica resync completed with 0 operations
19:28:45.990
elasticsearch.server
[elasticsearch.server][INFO] [.apm-custom-link][0] primary-replica resync completed with 0 operations
19:28:45.990
elasticsearch.server
[elasticsearch.server][INFO] [.apm-agent-configuration][0] primary-replica resync completed with 0 operations
19:28:45.994
elasticsearch.server
[elasticsearch.server][INFO] [.kibana-event-log-7.12.0-000004][0] primary-replica resync completed with 0 operations
19:28:46.000
elasticsearch.server
[elasticsearch.server][INFO] [.ds-ilm-history-5-2021.04.07-000001][0] primary-replica resync completed with 0 operations
19:28:46.007
elasticsearch.server
[elasticsearch.server][INFO] [.transform-internal-006][0] primary-replica resync completed with 0 operations
19:28:46.009
elasticsearch.server
[elasticsearch.server][INFO] [.transform-notifications-000002][0] primary-replica resync completed with 0 operations
19:28:46.011
elasticsearch.server
[elasticsearch.server][INFO] [.kibana_security_session_1][0] primary-replica resync completed with 0 operations
19:28:46.014
elasticsearch.server
[elasticsearch.server][INFO] [.kibana_7.12.0_001][0] primary-replica resync completed with 0 operations
19:28:46.018
elasticsearch.server
[elasticsearch.server][INFO] [.kibana-event-log-7.12.0-000002][0] primary-replica resync completed with 0 operations
19:28:46.021
elasticsearch.server
[elasticsearch.server][INFO] [.ds-.slm-history-5-2021.07.06-000004][0] primary-replica resync completed with 0 operations
19:28:46.023
elasticsearch.server
[elasticsearch.server][INFO] [idx-service-events-prod-v1][0] primary-replica resync completed with 0 operations
19:28:46.074
elasticsearch.server
[elasticsearch.server][INFO] [.tasks][0] primary-replica resync completed with 0 operations
19:28:46.083
elasticsearch.server
[elasticsearch.server][INFO] [.ds-ilm-history-5-2021.07.06-000004][0] primary-replica resync completed with 0 operations
19:28:46.088
elasticsearch.server
[elasticsearch.server][INFO] [.ds-.slm-history-5-2021.06.06-000003][0] primary-replica resync completed with 0 operations
19:28:46.091
elasticsearch.server
[elasticsearch.server][INFO] [.ds-.slm-history-5-2021.05.07-000002][0] primary-replica resync completed with 0 operations
19:28:46.091
elasticsearch.server
[elasticsearch.server][INFO] [.ds-ilm-history-5-2021.06.06-000003][0] primary-replica resync completed with 0 operations
19:28:46.095
elasticsearch.server
[elasticsearch.server][INFO] [.siem-signals-default-000002][0] primary-replica resync completed with 0 operations
19:28:46.101
elasticsearch.server
[elasticsearch.server][INFO] [.ds-ilm-history-5-2021.05.07-000002][0] primary-replica resync completed with 0 operations
19:28:46.107
elasticsearch.server
[elasticsearch.server][INFO] [.siem-signals-default-000001][0] primary-replica resync completed with 0 operations
19:28:46.107
elasticsearch.server
[elasticsearch.server][INFO] [.kibana-event-log-7.12.0-000001][0] primary-replica resync completed with 0 operations
19:28:46.114
elasticsearch.server
[elasticsearch.server][WARN] failed to retrieve shard stats from node [rzyW_Ck_S4O8DOEALh3_bA]: [instance-0000000000][172.25.70.164:19610][indices:monitor/stats[n]] disconnected
19:28:46.119
elasticsearch.server
[elasticsearch.server][WARN] failed to retrieve stats for node [rzyW_Ck_S4O8DOEALh3_bA]: [instance-0000000000][172.25.70.164:19610][cluster:monitor/nodes/stats[n]] disconnected
19:28:46.178
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "ICBufferFull", Time since last: 341983275 ns, Reaching safepoint: 672493 ns, At safepoint: 62886 ns, Total: 735379 ns
19:28:46.184
elasticsearch.server
[elasticsearch.server][INFO] scheduling reroute for delayed shards in [59.6s] (6 delayed shards)
19:28:46.184
elasticsearch.server
[elasticsearch.server][INFO] scheduling reroute for delayed shards in [59.6s] (6 delayed shards)
19:28:46.403
elasticsearch.server
[elasticsearch.server][WARN] [0] marking unavailable shards as stale: [cT6Flvz-RTKpj8ExW5lVTA]
19:28:46.501
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "ICBufferFull", Time since last: 322791460 ns, Reaching safepoint: 451233 ns, At safepoint: 45005 ns, Total: 496238 ns
19:28:46.514
elasticsearch.server
[elasticsearch.server][INFO] updating state for transform to [{"task_state":"started","indexer_state":"stopped","checkpoint":1,"should_stop_at_checkpoint":false}].
19:28:46.597
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 3000246010 ns, Reaching safepoint: 127447 ns, At safepoint: 6818 ns, Total: 134265 ns
19:28:46.683
elasticsearch.server
[elasticsearch.server][INFO] successfully completed and scheduled task in node operation
19:28:46.684
elasticsearch.server
[elasticsearch.server][WARN] [0] marking unavailable shards as stale: [DUKq3ea8QXWBSjvNAS0m3Q]
19:28:47.503
elasticsearch.gc
[elasticsearch.gc][INFO] Safepoint "Cleanup", Time since last: 1001336488 ns, Reaching safepoint: 735587 ns, At safepoint: 22569 ns, Total: 758156 ns
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment