Skip to content

Instantly share code, notes, and snippets.

@wduncanfraser

wduncanfraser/0700_crash.txt Secret

Created Feb 18, 2021
Embed
What would you like to do?
$ sudo journalctl --unit sensu-backend --since "2021-02-18 06:30" --until "2021-02-18 07:30" | cat
-- Logs begin at Mon 2020-02-10 16:53:57 UTC, end at Thu 2021-02-18 15:34:23 UTC. --
Feb 18 06:30:27 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:6 size:2091\" took too long (105.880394ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:30:27Z"}
Feb 18 06:30:27 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/silenced/\\\" range_end:\\\"/sensu.io/silencee\\\" serializable:true \" with result \"range_response_count:0 size:6\" took too long (106.899425ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:30:27Z"}
Feb 18 06:31:03 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (135.517112ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:31:03Z"}
Feb 18 06:31:24 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/assets/shared-services/\\\" range_end:\\\"/sensu.io/assets/shared-services0\\\" serializable:true \" with result \"range_response_count:4 size:3532\" took too long (278.990635ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:31:24Z"}
Feb 18 06:31:24 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" range_end:\\\"/sensu.io/rings/metricsd/backends/items/\\\\377\\\" limit:2 \" with result \"range_response_count:1 size:108\" took too long (309.386898ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:31:24Z"}
Feb 18 06:31:29 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (117.374689ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:31:29Z"}
Feb 18 06:31:48 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (262.673634ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:31:48Z"}
Feb 18 06:31:48 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:6 size:2091\" took too long (288.056677ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:31:48Z"}
Feb 18 06:31:48 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/namespaces/\\\" range_end:\\\"/sensu.io/namespacet\\\" serializable:true \" with result \"range_response_count:2 size:124\" took too long (301.340603ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:31:48Z"}
Feb 18 06:31:50 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/shared-services/<snip>-ciwebtest01.ad.mta-telco.com\\\" limit:1 serializable:true \" with result \"range_response_count:1 size:356\" took too long (367.287658ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:31:50Z"}
Feb 18 06:31:50 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/keepalive\\\" limit:1 \" with result \"range_response_count:1 size:125\" took too long (633.27595ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:31:50Z"}
Feb 18 06:31:50 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (314.59746ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:31:50Z"}
Feb 18 06:32:25 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"request \"header:\u003cID:6390458008197954912 \u003e txn:\u003ccompare:\u003cresult:GREATER target:CREATE key:\\\"/sensu.io/namespaces/shared-services\\\" create_revision:0 \u003e success:\u003crequest_put:\u003ckey:\\\"/sensu.io/api/internal/metricsd/v1/metrics/shared-services/entity_gauges\\\" value_size:172 \u003e\u003e failure:\u003crequest_range:\u003ckey:\\\"/sensu.io/namespaces/shared-services\\\" \u003e \u003e\u003e\" with result \"size:18\" took too long (111.761639ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:32:25Z"}
Feb 18 06:33:33 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/keepalive\\\" limit:1 \" with result \"range_response_count:1 size:125\" took too long (168.298574ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:33:33Z"}
Feb 18 06:34:08 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/shared-services/linux/items/<snip>-ciworkertest01.ad.mta-telco.com\\\" \" with result \"range_response_count:1 size:115\" took too long (183.691623ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:34:08Z"}
Feb 18 06:34:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 417098 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T06:34:12Z"}
Feb 18 06:34:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 417098","pkg":"compactor","time":"2021-02-18T06:34:12Z"}
Feb 18 06:34:27 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/keepalive\\\" limit:1 \" with result \"range_response_count:1 size:125\" took too long (156.903709ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:34:27Z"}
Feb 18 06:35:25 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" range_end:\\\"/sensu.io/rings/metricsd/backends/items/\\\\377\\\" limit:2 \" with result \"range_response_count:1 size:108\" took too long (157.858023ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:35:25Z"}
Feb 18 06:35:29 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (183.938992ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:35:29Z"}
Feb 18 06:36:42 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"sync duration of 1.013986002s, expected less than 1s","pkg":"wal","time":"2021-02-18T06:36:42Z"}
Feb 18 06:37:27 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" range_end:\\\"/sensu.io/rings/metricsd/backends/items/\\\\377\\\" limit:2 \" with result \"range_response_count:1 size:108\" took too long (196.462813ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:37:27Z"}
Feb 18 06:39:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 417818 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T06:39:12Z"}
Feb 18 06:39:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 417818","pkg":"compactor","time":"2021-02-18T06:39:12Z"}
Feb 18 06:41:20 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/events/shared-services/<snip>-ciwebtest01.ad.mta-telco.com/cpu-load-linux\\\" range_end:\\\"/sensu.io/events/shared-services/<snip>-ciwebtest01.ad.mta-telco.com/cpu-load-linuy\\\" serializable:true \" with result \"range_response_count:1 size:1117\" took too long (107.375074ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:41:20Z"}
Feb 18 06:41:24 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (224.381053ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:41:24Z"}
Feb 18 06:41:26 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/events/shared-services/<snip>-bastiontest02.ad.mta-telco.com/memory\\\" range_end:\\\"/sensu.io/events/shared-services/<snip>-bastiontest02.ad.mta-telco.com/memorz\\\" serializable:true \" with result \"range_response_count:1 size:1063\" took too long (130.456526ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:41:26Z"}
Feb 18 06:41:26 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/events/shared-services/<snip>-ciwebtest01.ad.mta-telco.com/memory\\\" range_end:\\\"/sensu.io/events/shared-services/<snip>-ciwebtest01.ad.mta-telco.com/memorz\\\" serializable:true \" with result \"range_response_count:1 size:1052\" took too long (133.055752ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:41:26Z"}
Feb 18 06:41:26 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/events/shared-services/<snip>-sensutest01.ad.mta-telco.com/memory\\\" range_end:\\\"/sensu.io/events/shared-services/<snip>-sensutest01.ad.mta-telco.com/memorz\\\" serializable:true \" with result \"range_response_count:1 size:1070\" took too long (118.946343ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:41:26Z"}
Feb 18 06:41:26 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/events/shared-services/<snip>-pgsqltest02.ad.mta-telco.com/memory\\\" range_end:\\\"/sensu.io/events/shared-services/<snip>-pgsqltest02.ad.mta-telco.com/memorz\\\" serializable:true \" with result \"range_response_count:1 size:1055\" took too long (128.900841ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:41:26Z"}
Feb 18 06:44:09 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (136.551363ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:44:09Z"}
Feb 18 06:44:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 418538 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T06:44:12Z"}
Feb 18 06:44:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 418538","pkg":"compactor","time":"2021-02-18T06:44:12Z"}
Feb 18 06:44:47 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (131.50404ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:44:47Z"}
Feb 18 06:44:47 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/namespaces/\\\" range_end:\\\"/sensu.io/namespacet\\\" serializable:true \" with result \"range_response_count:2 size:124\" took too long (170.763242ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:44:47Z"}
Feb 18 06:44:47 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:6 size:2091\" took too long (157.818164ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:44:47Z"}
Feb 18 06:44:47 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:6 size:2091\" took too long (187.141216ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:44:47Z"}
Feb 18 06:44:47 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/silenced/\\\" range_end:\\\"/sensu.io/silencee\\\" serializable:true \" with result \"range_response_count:0 size:6\" took too long (188.170916ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:44:47Z"}
Feb 18 06:44:47 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/shared-services/<snip>-sensutest01.ad.mta-telco.com\\\" limit:1 serializable:true \" with result \"range_response_count:1 size:372\" took too long (432.383149ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:44:47Z"}
Feb 18 06:49:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 419258 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T06:49:12Z"}
Feb 18 06:49:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 419258","pkg":"compactor","time":"2021-02-18T06:49:12Z"}
Feb 18 06:50:03 <snip>-sensutest01 sensu-backend[17071]: {"addr":"192.168.100.163:46212","agent":"<snip>-ciworkertest01.ad.mta-telco.com","component":"agentd","error":"Connection closed: websocket: close 1001 (going away): bye","level":"warning","msg":"stopping session","time":"2021-02-18T06:50:03Z"}
Feb 18 06:50:33 <snip>-sensutest01 sensu-backend[17071]: {"check":"users-linux","component":"eventd","entity":"<snip>-ciworkertest01.ad.mta-telco.com","level":"warning","msg":"check TTL expired","namespace":"shared-services","previous_status":"alive","status":"dead","time":"2021-02-18T06:50:33Z"}
Feb 18 06:54:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 420003 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T06:54:12Z"}
Feb 18 06:54:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 420003","pkg":"compactor","time":"2021-02-18T06:54:12Z"}
Feb 18 06:55:20 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (122.965656ms) to execute","pkg":"etcdserver","time":"2021-02-18T06:55:20Z"}
Feb 18 06:55:38 <snip>-sensutest01 sensu-backend[17071]: {"addr":"192.168.100.170:45482","agent":"<snip>-ciwebtest01.ad.mta-telco.com","component":"agentd","error":"Connection closed: websocket: close 1001 (going away): bye","level":"warning","msg":"stopping session","time":"2021-02-18T06:55:38Z"}
Feb 18 06:59:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 420745 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T06:59:12Z"}
Feb 18 06:59:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 420745","pkg":"compactor","time":"2021-02-18T06:59:12Z"}
Feb 18 07:00:50 <snip>-sensutest01 sensu-backend[17071]: {"level":"warn","ts":"2021-02-18T07:00:50.849Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-7bfeee7e-ca17-4752-a55f-259bdf1192c2/localhost:2379","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: request timed out"}
Feb 18 07:00:50 <snip>-sensutest01 sensu-backend[17071]: {"component":"keepalived","error":"internal error: etcdserver: request timed out","level":"error","msg":"error updating event","time":"2021-02-18T07:00:50Z"}
Feb 18 07:00:50 <snip>-sensutest01 sensu-backend[17071]: {"component":"backend","error":"error from keepalived: internal error: etcdserver: request timed out","level":"error","msg":"backend stopped working and is restarting","time":"2021-02-18T07:00:50Z"}
Feb 18 07:00:50 <snip>-sensutest01 sensu-backend[17071]: {"component":"replicatord","level":"warning","msg":"shutting down replicatord","time":"2021-02-18T07:00:50Z"}
Feb 18 07:00:50 <snip>-sensutest01 sensu-backend[17071]: {"component":"secrets-providers","level":"error","msg":"the secrets providers watcher is no longer running","time":"2021-02-18T07:00:50Z"}
Feb 18 07:00:50 <snip>-sensutest01 sensu-backend[17071]: {"component":"auth-providers","level":"error","msg":"the authentication providers watcher is no longer running","time":"2021-02-18T07:00:50Z"}
Feb 18 07:00:51 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"timed out waiting for read index response (local node might have slow network)","pkg":"etcdserver","time":"2021-02-18T07:00:51Z"}
Feb 18 07:00:51 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"error:etcdserver: request timed out\" took too long (7.000444905s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:51Z"}
Feb 18 07:00:51 <snip>-sensutest01 sensu-backend[17071]: {"level":"warn","ts":"2021-02-18T07:00:51.752Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-7bfeee7e-ca17-4752-a55f-259bdf1192c2/localhost:2379","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: request timed out"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"sync duration of 9.52066036s, expected less than 1s","pkg":"wal","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"request \"header:\u003cID:6390458008197964556 \u003e delete_range:\u003ckey:\\\"/sensu.io/keepalives/default/shared-services/<snip>-ciworkertest01.ad.mta-telco.com\\\" \u003e \" with result \"size:6\" took too long (104.112392ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:6 size:2135\" took too long (5.778797558s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (5.724717005s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (2.624435412s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/assets/shared-services/\\\" range_end:\\\"/sensu.io/assets/shared-services0\\\" serializable:true \" with result \"range_response_count:4 size:3532\" took too long (6.305946737s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (4.723122843s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/shared-services/<snip>-sensutest01.ad.mta-telco.com\\\" limit:1 serializable:true \" with result \"range_response_count:1 size:372\" took too long (6.022648497s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/silenced/\\\" range_end:\\\"/sensu.io/silencee\\\" serializable:true \" with result \"range_response_count:0 size:6\" took too long (5.779554407s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:6 size:2135\" took too long (5.749720078s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/namespaces/\\\" range_end:\\\"/sensu.io/namespacet\\\" serializable:true \" with result \"range_response_count:2 size:124\" took too long (5.762859643s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/shared-services/<snip>-bastiontest02.ad.mta-telco.com\\\" limit:1 serializable:true \" with result \"range_response_count:1 size:360\" took too long (5.472895954s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/shared-services/<snip>-pgsqltest02.ad.mta-telco.com\\\" limit:1 serializable:true \" with result \"range_response_count:1 size:356\" took too long (4.1476862s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/api/enterprise/secrets/v1/secrets\\\" range_end:\\\"/sensu.io/api/enterprise/secrets/v1/secrett\\\" count_only:true \" with result \"range_response_count:0 size:6\" took too long (7.956389069s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/bb543e23-4268-4535-ba03-1c9704dfee41\\\" \" with result \"range_response_count:1 size:108\" took too long (1.694361209s) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"level":"warn","ts":"2021-02-18T07:00:53.511Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-7bfeee7e-ca17-4752-a55f-259bdf1192c2/localhost:2379","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"level":"warn","ts":"2021-02-18T07:00:53.511Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-7bfeee7e-ca17-4752-a55f-259bdf1192c2/localhost:2379","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"level":"warn","ts":"2021-02-18T07:00:53.511Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-7bfeee7e-ca17-4752-a55f-259bdf1192c2/localhost:2379","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
Feb 18 07:00:53 <snip>-sensutest01 sensu-backend[17071]: {"level":"warn","ts":"2021-02-18T07:00:53.512Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-7bfeee7e-ca17-4752-a55f-259bdf1192c2/localhost:2379","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
Feb 18 07:02:18 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"sync duration of 4.517774521s, expected less than 1s","pkg":"wal","time":"2021-02-18T07:02:18Z"}
Feb 18 07:02:18 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/silenced/\\\" range_end:\\\"/sensu.io/silencee\\\" serializable:true \" with result \"range_response_count:0 size:6\" took too long (562.878534ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:02:18Z"}
Feb 18 07:02:18 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:6 size:2135\" took too long (563.04134ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:02:18Z"}
Feb 18 07:02:18 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/namespaces/\\\" range_end:\\\"/sensu.io/namespacet\\\" serializable:true \" with result \"range_response_count:2 size:124\" took too long (546.95458ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:02:18Z"}
Feb 18 07:02:18 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:6 size:2135\" took too long (533.903046ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:02:18Z"}
Feb 18 07:02:18 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/assets/shared-services/\\\" range_end:\\\"/sensu.io/assets/shared-services0\\\" serializable:true \" with result \"range_response_count:4 size:3532\" took too long (182.041411ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:02:18Z"}
Feb 18 07:04:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 421441 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:04:12Z"}
Feb 18 07:04:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 421441","pkg":"compactor","time":"2021-02-18T07:04:12Z"}
Feb 18 07:05:26 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"request \"header:\u003cID:6390458008197965885 \u003e txn:\u003ccompare:\u003cresult:GREATER key:\\\"/sensu.io/namespaces/shared-services\\\" version:0 \u003e success:\u003crequest_put:\u003ckey:\\\"/sensu.io/events/shared-services/<snip>-ciworkertest01.ad.mta-telco.com/memory\\\" value_size:980 \u003e\u003e failure:\u003c\u003e\u003e\" with result \"size:18\" took too long (280.735138ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:05:26Z"}
Feb 18 07:05:28 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"sync duration of 1.632041454s, expected less than 1s","pkg":"wal","time":"2021-02-18T07:05:28Z"}
Feb 18 07:05:28 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/namespaces/\\\" range_end:\\\"/sensu.io/namespacet\\\" serializable:true \" with result \"range_response_count:2 size:124\" took too long (414.45203ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:05:28Z"}
Feb 18 07:05:28 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:6 size:2135\" took too long (401.67362ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:05:28Z"}
Feb 18 07:05:28 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/shared-services/<snip>-bastiontest02.ad.mta-telco.com\\\" limit:1 serializable:true \" with result \"range_response_count:1 size:360\" took too long (126.202468ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:05:28Z"}
Feb 18 07:05:28 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (1.358777283s) to execute","pkg":"etcdserver","time":"2021-02-18T07:05:28Z"}
Feb 18 07:05:28 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/silenced/\\\" range_end:\\\"/sensu.io/silencee\\\" serializable:true \" with result \"range_response_count:0 size:6\" took too long (431.700778ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:05:28Z"}
Feb 18 07:05:28 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/shared-services/<snip>-sensutest01.ad.mta-telco.com\\\" limit:1 serializable:true \" with result \"range_response_count:1 size:372\" took too long (675.078585ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:05:28Z"}
Feb 18 07:05:28 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:6 size:2135\" took too long (431.307123ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:05:28Z"}
Feb 18 07:05:48 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/shared-services/linux/items/<snip>-bastiontest02.ad.mta-telco.com\\\" \" with result \"range_response_count:1 size:114\" took too long (153.433903ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:05:48Z"}
Feb 18 07:09:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 422101 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:09:12Z"}
Feb 18 07:09:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 422101","pkg":"compactor","time":"2021-02-18T07:09:12Z"}
Feb 18 07:14:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 422761 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:14:12Z"}
Feb 18 07:14:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 422761","pkg":"compactor","time":"2021-02-18T07:14:12Z"}
Feb 18 07:19:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 423421 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:19:12Z"}
Feb 18 07:19:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 423421","pkg":"compactor","time":"2021-02-18T07:19:12Z"}
Feb 18 07:20:20 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (117.519447ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:20:20Z"}
Feb 18 07:24:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 424081 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:24:12Z"}
Feb 18 07:24:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 424081","pkg":"compactor","time":"2021-02-18T07:24:12Z"}
Feb 18 07:29:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 424741 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:29:12Z"}
Feb 18 07:29:12 <snip>-sensutest01 sensu-backend[17071]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 424741","pkg":"compactor","time":"2021-02-18T07:29:12Z"}
$ sudo journalctl --unit sensu-backend --since "2021-02-18 07:00" --until "2021-02-18 08:00" | cat
-- Logs begin at Mon 2020-02-10 16:53:57 UTC, end at Thu 2021-02-18 15:39:22 UTC. --
Feb 18 07:00:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 537626 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/shared-services/linux/items/<snip>-ciproxy01.ad.mta-telco.com\\\" \" with result \"range_response_count:1 size:108\" took too long (255.474857ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:00:53Z"}
Feb 18 07:00:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 537626","pkg":"compactor","time":"2021-02-18T07:00:53Z"}
Feb 18 07:04:04 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/6c78293a-0614-419f-a55c-4463a22a199c\\\" \" with result \"range_response_count:1 size:108\" took too long (743.207224ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:04:04Z"}
Feb 18 07:04:14 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/shared-services/linux/items/<snip>-ciworker01.ad.mta-telco.com\\\" \" with result \"range_response_count:1 size:109\" took too long (131.533888ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:04:14Z"}
Feb 18 07:05:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 538567 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:05:53Z"}
Feb 18 07:05:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 538567","pkg":"compactor","time":"2021-02-18T07:05:53Z"}
Feb 18 07:06:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (111.392617ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:06:24Z"}
Feb 18 07:10:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 539507 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:10:53Z"}
Feb 18 07:10:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 539507","pkg":"compactor","time":"2021-02-18T07:10:53Z"}
Feb 18 07:14:38 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/keepalive\\\" limit:1 \" with result \"range_response_count:1 size:125\" took too long (102.683382ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:14:38Z"}
Feb 18 07:15:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 540449 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:15:53Z"}
Feb 18 07:15:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 540449","pkg":"compactor","time":"2021-02-18T07:15:53Z"}
Feb 18 07:20:20 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (107.145151ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:20:20Z"}
Feb 18 07:20:20 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (107.163374ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:20:20Z"}
Feb 18 07:20:20 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (107.322279ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:20:20Z"}
Feb 18 07:20:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 541389 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:20:53Z"}
Feb 18 07:20:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 541389","pkg":"compactor","time":"2021-02-18T07:20:53Z"}
Feb 18 07:22:03 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (143.714523ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:22:03Z"}
Feb 18 07:22:03 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (143.778625ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:22:03Z"}
Feb 18 07:22:03 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (143.738071ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:22:03Z"}
Feb 18 07:22:03 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (143.833307ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:22:03Z"}
Feb 18 07:22:03 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/microsoft-teams\\\" limit:1 \" with result \"range_response_count:1 size:473\" took too long (144.595428ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:22:03Z"}
Feb 18 07:22:03 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (143.773107ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:22:03Z"}
Feb 18 07:22:03 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (143.887382ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:22:03Z"}
Feb 18 07:25:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 542329 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:25:53Z"}
Feb 18 07:25:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 542329","pkg":"compactor","time":"2021-02-18T07:25:53Z"}
Feb 18 07:28:50 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:8 size:2763\" took too long (1.507346573s) to execute","pkg":"etcdserver","time":"2021-02-18T07:28:50Z"}
Feb 18 07:28:50 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/silenced/\\\" range_end:\\\"/sensu.io/silencee\\\" serializable:true \" with result \"range_response_count:0 size:6\" took too long (1.508812507s) to execute","pkg":"etcdserver","time":"2021-02-18T07:28:50Z"}
Feb 18 07:28:50 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/namespaces/\\\" range_end:\\\"/sensu.io/namespacet\\\" serializable:true \" with result \"range_response_count:2 size:124\" took too long (1.502941918s) to execute","pkg":"etcdserver","time":"2021-02-18T07:28:50Z"}
Feb 18 07:28:50 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:8 size:2763\" took too long (1.49526197s) to execute","pkg":"etcdserver","time":"2021-02-18T07:28:50Z"}
Feb 18 07:28:50 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/6c78293a-0614-419f-a55c-4463a22a199c\\\" \" with result \"range_response_count:1 size:108\" took too long (1.477036597s) to execute","pkg":"etcdserver","time":"2021-02-18T07:28:50Z"}
Feb 18 07:28:50 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/6c78293a-0614-419f-a55c-4463a22a199c\\\" \" with result \"range_response_count:1 size:108\" took too long (481.040182ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:28:50Z"}
Feb 18 07:30:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 543271 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:30:53Z"}
Feb 18 07:30:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 543271","pkg":"compactor","time":"2021-02-18T07:30:53Z"}
Feb 18 07:35:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 544211 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:35:53Z"}
Feb 18 07:35:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 544211","pkg":"compactor","time":"2021-02-18T07:35:53Z"}
Feb 18 07:36:05 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:8 size:2763\" took too long (1.967538749s) to execute","pkg":"etcdserver","time":"2021-02-18T07:36:05Z"}
Feb 18 07:36:05 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/namespaces/\\\" range_end:\\\"/sensu.io/namespacet\\\" serializable:true \" with result \"range_response_count:2 size:124\" took too long (1.975692664s) to execute","pkg":"etcdserver","time":"2021-02-18T07:36:05Z"}
Feb 18 07:36:05 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"request \"header:\u003cID:6390458007455718087 \u003e put:\u003ckey:\\\"/sensu.io/switchsets/eventd/shared-services/users-linux/<snip>-ciworker02.ad.mta-telco.com\\\" value_size:2 lease:6390458007455592089 prev_kv:true \u003e\" with result \"size:126\" took too long (1.949192011s) to execute","pkg":"etcdserver","time":"2021-02-18T07:36:05Z"}
Feb 18 07:36:05 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (1.98500278s) to execute","pkg":"etcdserver","time":"2021-02-18T07:36:05Z"}
Feb 18 07:36:05 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"sync duration of 1.976370748s, expected less than 1s","pkg":"wal","time":"2021-02-18T07:36:05Z"}
Feb 18 07:36:05 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (2.010129979s) to execute","pkg":"etcdserver","time":"2021-02-18T07:36:05Z"}
Feb 18 07:36:05 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (2.010300514s) to execute","pkg":"etcdserver","time":"2021-02-18T07:36:05Z"}
Feb 18 07:36:05 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (2.010630294s) to execute","pkg":"etcdserver","time":"2021-02-18T07:36:05Z"}
Feb 18 07:36:05 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/6c78293a-0614-419f-a55c-4463a22a199c\\\" \" with result \"range_response_count:1 size:108\" took too long (980.645275ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:36:05Z"}
Feb 18 07:36:05 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/6c78293a-0614-419f-a55c-4463a22a199c\\\" \" with result \"range_response_count:1 size:108\" took too long (1.981955733s) to execute","pkg":"etcdserver","time":"2021-02-18T07:36:05Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"level":"warn","ts":"2021-02-18T07:40:24.031Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-cd5770da-2e55-4a15-8a54-a27c0795b781/localhost:2379","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: request timed out"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"keepalived","error":"internal error: etcdserver: request timed out","level":"error","msg":"error updating event","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"backend","error":"error from keepalived: internal error: etcdserver: request timed out","level":"error","msg":"backend stopped working and is restarting","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"replicatord","level":"warning","msg":"shutting down replicatord","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"secrets-providers","level":"error","msg":"the secrets providers watcher is no longer running","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"auth-providers","level":"error","msg":"the authentication providers watcher is no longer running","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"sync duration of 7.826943139s, expected less than 1s","pkg":"wal","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/namespaces/\\\" range_end:\\\"/sensu.io/namespacet\\\" serializable:true \" with result \"range_response_count:2 size:124\" took too long (6.217369232s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:8 size:2763\" took too long (6.221937617s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/assets/shared-services/\\\" range_end:\\\"/sensu.io/assets/shared-services0\\\" serializable:true \" with result \"range_response_count:4 size:3532\" took too long (6.818533126s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/assets/shared-services/\\\" range_end:\\\"/sensu.io/assets/shared-services0\\\" serializable:true \" with result \"range_response_count:4 size:3532\" took too long (6.706786796s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/shared-services/<snip>-sensu01.ad.mta-telco.com\\\" limit:1 serializable:true \" with result \"range_response_count:1 size:364\" took too long (6.359017391s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/silenced/\\\" range_end:\\\"/sensu.io/silencee\\\" serializable:true \" with result \"range_response_count:0 size:6\" took too long (6.223334638s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/assets/shared-services/\\\" range_end:\\\"/sensu.io/assets/shared-services0\\\" serializable:true \" with result \"range_response_count:4 size:3532\" took too long (382.299106ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:8 size:2763\" took too long (6.209730157s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/assets/shared-services/\\\" range_end:\\\"/sensu.io/assets/shared-services0\\\" serializable:true \" with result \"range_response_count:4 size:3532\" took too long (4.765296572s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/assets/shared-services/\\\" range_end:\\\"/sensu.io/assets/shared-services0\\\" serializable:true \" with result \"range_response_count:4 size:3532\" took too long (841.802043ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/6c78293a-0614-419f-a55c-4463a22a199c\\\" \" with result \"range_response_count:1 size:108\" took too long (5.188651889s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/6c78293a-0614-419f-a55c-4463a22a199c\\\" \" with result \"range_response_count:1 size:108\" took too long (6.189784407s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/6c78293a-0614-419f-a55c-4463a22a199c\\\" \" with result \"range_response_count:1 size:108\" took too long (858.836777ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/rings/metricsd/backends/items/6c78293a-0614-419f-a55c-4463a22a199c\\\" \" with result \"range_response_count:1 size:108\" took too long (4.188620933s) to execute","pkg":"etcdserver","time":"2021-02-18T07:40:24Z"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"level":"warn","ts":"2021-02-18T07:40:24.897Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-cd5770da-2e55-4a15-8a54-a27c0795b781/localhost:2379","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
Feb 18 07:40:24 <snip>-sensu01 sensu-backend[16172]: {"level":"warn","ts":"2021-02-18T07:40:24.897Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-cd5770da-2e55-4a15-8a54-a27c0795b781/localhost:2379","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
Feb 18 07:40:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 545150 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:40:53Z"}
Feb 18 07:40:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 545150","pkg":"compactor","time":"2021-02-18T07:40:53Z"}
Feb 18 07:45:26 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/microsoft-teams\\\" limit:1 \" with result \"range_response_count:1 size:473\" took too long (104.119066ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:45:26Z"}
Feb 18 07:45:26 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (104.238442ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:45:26Z"}
Feb 18 07:45:26 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/handlers/shared-services/default\\\" limit:1 \" with result \"range_response_count:1 size:121\" took too long (104.316153ms) to execute","pkg":"etcdserver","time":"2021-02-18T07:45:26Z"}
Feb 18 07:45:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 546015 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:45:53Z"}
Feb 18 07:45:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 546015","pkg":"compactor","time":"2021-02-18T07:45:53Z"}
Feb 18 07:50:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 546875 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:50:53Z"}
Feb 18 07:50:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 546875","pkg":"compactor","time":"2021-02-18T07:50:53Z"}
Feb 18 07:55:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Starting auto-compaction at revision 547749 (retention: 2 revisions)","pkg":"compactor","time":"2021-02-18T07:55:53Z"}
Feb 18 07:55:53 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"Finished auto-compaction at revision 547749","pkg":"compactor","time":"2021-02-18T07:55:53Z"}
Feb 18 07:57:30 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"sync duration of 6.911766797s, expected less than 1s","pkg":"wal","time":"2021-02-18T07:57:30Z"}
Feb 18 07:57:30 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/assets/shared-services/\\\" range_end:\\\"/sensu.io/assets/shared-services0\\\" serializable:true \" with result \"range_response_count:4 size:3532\" took too long (6.471117313s) to execute","pkg":"etcdserver","time":"2021-02-18T07:57:30Z"}
Feb 18 07:57:30 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/assets/shared-services/\\\" range_end:\\\"/sensu.io/assets/shared-services0\\\" serializable:true \" with result \"range_response_count:4 size:3532\" took too long (4.773833628s) to execute","pkg":"etcdserver","time":"2021-02-18T07:57:30Z"}
Feb 18 07:57:30 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/silenced/\\\" range_end:\\\"/sensu.io/silencee\\\" serializable:true \" with result \"range_response_count:0 size:6\" took too long (2.315294444s) to execute","pkg":"etcdserver","time":"2021-02-18T07:57:30Z"}
Feb 18 07:57:30 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:8 size:2763\" took too long (2.314146612s) to execute","pkg":"etcdserver","time":"2021-02-18T07:57:30Z"}
Feb 18 07:57:30 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/namespaces/\\\" range_end:\\\"/sensu.io/namespacet\\\" serializable:true \" with result \"range_response_count:2 size:124\" took too long (2.309740007s) to execute","pkg":"etcdserver","time":"2021-02-18T07:57:30Z"}
Feb 18 07:57:30 <snip>-sensu01 sensu-backend[16172]: {"component":"etcd","level":"warning","msg":"read-only range request \"key:\\\"/sensu.io/entity_configs/\\\" range_end:\\\"/sensu.io/entity_configt\\\" serializable:true \" with result \"range_response_count:8 size:2763\" took too long (2.301990798s) to execute","pkg":"etcdserver","time":"2021-02-18T07:57:30Z"}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment