Skip to content

Instantly share code, notes, and snippets.

@mvines

mvines/bs-leader.log

Last active January 29, 2020 17:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mvines/142cff34f6f0aa0cf4bb3c0ba425fd39 to your computer and use it in GitHub Desktop.
Save mvines/142cff34f6f0aa0cf4bb3c0ba425fd39 to your computer and use it in GitHub Desktop.
Download ZIP
Why did bs leader not vote for slot 1031288 or 1031291
Raw
bs-leader.log
[2020-01-29T00:08:56.186968315Z INFO solana_core::replay_stage] vote bank: Some(1031280) reset bank: 1031280
[2020-01-29T00:08:56.187006657Z INFO solana_core::replay_stage] new fork:1031281 parent:1031280 (leader) root:1031271
[2020-01-29T00:08:56.416855574Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002048 reached, clearing working_bank 1031281
[2020-01-29T00:08:56.417832835Z INFO solana_runtime::bank] bank frozen: 1031281 hash: iABt4wQH4iddkMqnCr6kJcyh5q716FFHMqzjGXF7czp accounts_delta: BankHash bb93496ad824018014b78831c37987c2a7bf5ef316a67873e3c609c5c8ba8f4d signature_count: 2 last_blockhash: 2cZfuGbAiVmusF2qZdc3hRJgGczun1pu6NUdy5iooLLo
[2020-01-29T00:08:56.417873592Z INFO solana_runtime::bank] accounts hash slot: 1031281 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 1015298080, total_data_len: 2070, num_executable_accounts: 0 }
[2020-01-29T00:08:56.417945006Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031281i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=229837i total_entries=0i total_shreds=0i
[2020-01-29T00:08:56.418513046Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031281i shredding_time=8254i insertion_time=4216i broadcast_time=9643i receive_time=202248i seed=3i num_shreds=65i slot_broadcast_time=227984i
[2020-01-29T00:08:56.419192295Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031281i weight="C940E3822FE5AEDD9EC74"
[2020-01-29T00:08:56.419186354Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031281 15206282374332371344944244 46692588969473521228041308 1031280
[2020-01-29T00:08:56.419241788Z INFO solana_core::replay_stage] voting: 1031281 46692588969473521228041308
[2020-01-29T00:08:56.419528789Z INFO solana_core::poh_recorder] reset poh from: 3dWrTDdbyPAdepczBxhFiNPB9xHjDtsV6xbi54pZA7qh,66002048,1031281 to: 2cZfuGbAiVmusF2qZdc3hRJgGczun1pu6NUdy5iooLLo,1031281
[2020-01-29T00:08:56.419547123Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002048 (within slot 1031281). My next leader slot is 1031282
[2020-01-29T00:08:56.419554838Z INFO solana_core::replay_stage] vote bank: Some(1031281) reset bank: 1031281
[2020-01-29T00:08:56.419587379Z INFO solana_core::replay_stage] new fork:1031282 parent:1031281 (leader) root:1031271
[2020-01-29T00:08:56.653932724Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002112 reached, clearing working_bank 1031282
[2020-01-29T00:08:56.654466078Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031282i shredding_time=7642i insertion_time=3792i broadcast_time=8923i receive_time=193961i seed=4i num_shreds=61i slot_broadcast_time=231711i
[2020-01-29T00:08:56.654866342Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031282i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=233586i total_entries=0i total_shreds=0i
[2020-01-29T00:08:56.654863096Z INFO solana_runtime::bank] bank frozen: 1031282 hash: 2ipjsXzwzdyW6w985xAX6fXExmvSdUYJ5X6WmZPBk8Nm accounts_delta: BankHash 7cea85cc7256ca7ce9852afb714a470a12821de151dee1bb7ba3ec8ab361cbaf signature_count: 2 last_blockhash: DwBnZ8QHGTxD5knj6kFpv5vez2TJRdQQFtUyVjWHJUZx
[2020-01-29T00:08:56.654888444Z INFO solana_runtime::bank] accounts hash slot: 1031282 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 1015298080, total_data_len: 2070, num_executable_accounts: 0 }
[2020-01-29T00:08:56.662460253Z INFO solana_runtime::bank] bank frozen: 1031272 hash: 9VLa3mnkCz9HTzqgxqmWcQfZwarePEntMxHNK9uGxCXJ accounts_delta: BankHash 44f094d14485cf965cb323b3b14e43ee3d8b35fb057897b9ce5ac9e5cb05270c signature_count: 142 last_blockhash: CrBUpSQfAPcNhYxBhehatxaop3EW2opzRrydf2STv9Qv
[2020-01-29T00:08:56.662487514Z INFO solana_runtime::bank] accounts hash slot: 1031272 stats: BankHashStats { num_removed_accounts: 130, num_added_accounts: 0, num_lamports_stored: 32123491410082, total_data_len: 122130, num_executable_accounts: 0 }
[2020-01-29T00:08:56.663187362Z INFO solana_core::replay_stage] validator fork confirmed 1031271 2684ms
[2020-01-29T00:08:56.663189055Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031272i fetch_entries_time=3991i fetch_entries_fail_time=0i entry_verification_time=5882i replay_time=8665i replay_total_elapsed=1459570i total_entries=78i total_shreds=90i
[2020-01-29T00:08:56.663221476Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031272 16280024225320719962931744 32560048448129498441618324 1031271
[2020-01-29T00:08:56.663235102Z WARN solana_metrics::metrics] datapoint: validator-confirmation duration_ms=2684i
[2020-01-29T00:08:56.663246644Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031272i weight="D776DE1B4801D824D0220"
[2020-01-29T00:08:56.663534165Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031282i weight="C940E3831345579DD2874"
[2020-01-29T00:08:56.663529026Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031282 15206282378332371226011764 61898871347805892454053072 1031281
[2020-01-29T00:08:56.663579951Z INFO solana_core::replay_stage] voting: 1031282 61898871347805892454053072
[2020-01-29T00:08:56.663876520Z INFO solana_core::poh_recorder] reset poh from: 8gw3WGjh7MnrkK1pkkCnzR8HSe3mW8CRpRMbsJp5bu75,66002114,1031282 to: DwBnZ8QHGTxD5knj6kFpv5vez2TJRdQQFtUyVjWHJUZx,1031282
[2020-01-29T00:08:56.663893191Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002112 (within slot 1031282). My next leader slot is 1031283
[2020-01-29T00:08:56.663900355Z INFO solana_core::replay_stage] vote bank: Some(1031282) reset bank: 1031282
[2020-01-29T00:08:56.663946211Z INFO solana_core::replay_stage] new fork:1031283 parent:1031282 (leader) root:1031271
[2020-01-29T00:08:56.664602507Z INFO solana_core::replay_stage] new fork:1031273 parent:1031272 root:1031271
[2020-01-29T00:08:56.693417160Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031273i fetch_entries_time=1077i fetch_entries_fail_time=0i entry_verification_time=26116i replay_time=771i replay_total_elapsed=27982i total_entries=67i total_shreds=67i
[2020-01-29T00:08:56.693415798Z INFO solana_runtime::bank] bank frozen: 1031273 hash: 6j8gngANmhjUmp4xS3H1ZcswZtLT1hMqrcEKWUDnB2mX accounts_delta: BankHash 34e2ce3a9bcdd1a2e1ca61d635571ad1a4fca57af9003d33f4045194659953a5 signature_count: 6 last_blockhash: EhE3hHtK2LMbFn2QAkhn1YNjekVz3dbmKYzjwNqKeCrm
[2020-01-29T00:08:56.693454430Z INFO solana_runtime::bank] accounts hash slot: 1031273 stats: BankHashStats { num_removed_accounts: 6, num_added_accounts: 0, num_lamports_stored: 1500001781760, total_data_len: 6210, num_executable_accounts: 0 }
[2020-01-29T00:08:56.693951026Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031273 16280024225040615634843484 48840072673170114076461808 1031272
[2020-01-29T00:08:56.693955434Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031273i weight="D776DE1B3815CA353375C"
[2020-01-29T00:08:56.694028852Z INFO solana_core::replay_stage] new fork:1031274 parent:1031273 root:1031271
[2020-01-29T00:08:56.902385024Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002176 reached, clearing working_bank 1031283
[2020-01-29T00:08:56.902719554Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031283i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=237336i total_entries=0i total_shreds=0i
[2020-01-29T00:08:56.902715497Z INFO solana_runtime::bank] bank frozen: 1031283 hash: 65mauUWdo2LZbSGQdJj4EnPdN2vXfcthZJTDU5KrHURU accounts_delta: BankHash 34540b122f9be372f3563c5e410985dd393085ce044179dd5a1a380d5e835d15 signature_count: 2 last_blockhash: 8zdWq2J8aWguMi8PvKURHrRpggP8fGMVY7UodvVK4hPC
[2020-01-29T00:08:56.902914421Z INFO solana_runtime::bank] accounts hash slot: 1031283 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 1015298080, total_data_len: 2070, num_executable_accounts: 0 }
[2020-01-29T00:08:56.902864698Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031283i shredding_time=8567i insertion_time=4167i broadcast_time=9094i receive_time=206572i seed=5i num_shreds=65i slot_broadcast_time=234780i
[2020-01-29T00:08:56.903877515Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031283i weight="D776DE0F1681579DD2874"
[2020-01-29T00:08:56.903872696Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031283 16280024170406676724000884 78178895518212569178053956 1031282
[2020-01-29T00:08:56.903945713Z INFO solana_core::replay_stage] voting: 1031283 78178895518212569178053956
[2020-01-29T00:08:56.904264945Z INFO solana_core::poh_recorder] reset poh from: E2MSahpMeVAedojXXWSWYdLiT3D5LXJYRs2M87GVubUZ,66002176,1031283 to: 8zdWq2J8aWguMi8PvKURHrRpggP8fGMVY7UodvVK4hPC,1031283
[2020-01-29T00:08:56.904285884Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002176 (within slot 1031283). My next leader slot is 1031284
[2020-01-29T00:08:56.904295081Z INFO solana_core::replay_stage] vote bank: Some(1031283) reset bank: 1031283
[2020-01-29T00:08:56.904338693Z INFO solana_core::replay_stage] new fork:1031284 parent:1031283 (leader) root:1031271
[2020-01-29T00:08:57.142185943Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002240 reached, clearing working_bank 1031284
[2020-01-29T00:08:57.142615191Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031284i shredding_time=8295i insertion_time=3948i broadcast_time=9790i receive_time=208452i seed=10i num_shreds=64i slot_broadcast_time=234045i
[2020-01-29T00:08:57.143846610Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031284i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=238361i total_entries=0i total_shreds=0i
[2020-01-29T00:08:57.143844987Z INFO solana_runtime::bank] bank frozen: 1031284 hash: 3rxAnwyXaJxnyMdPL91jk4P76bhqRUj2CpSsTEhXYY9n accounts_delta: BankHash 262d4e0698b5f6934ed01c3f12b1da8b7aedb8cadec6df8e827a0402b64c3f07 signature_count: 2 last_blockhash: CF6GuqGU16EEfCRvQ3uL5ETYRpArVALKrYRHTz1C1fPj
[2020-01-29T00:08:57.143899941Z INFO solana_runtime::bank] accounts hash slot: 1031284 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 1015298080, total_data_len: 2070, num_executable_accounts: 0 }
[2020-01-29T00:08:57.144397899Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031284 16280024170406676724000884 94458919688619245902054840 1031283
[2020-01-29T00:08:57.144405663Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031284i weight="D776DE0F1681579DD2874"
[2020-01-29T00:08:57.144421723Z INFO solana_core::replay_stage] voting: 1031284 94458919688619245902054840
[2020-01-29T00:08:57.144641728Z INFO solana_core::poh_recorder] reset poh from: 658UMfFKw4bTzb9o8qXCkwytpSGcRfz7BCvJQZsSUMGy,66002240,1031284 to: CF6GuqGU16EEfCRvQ3uL5ETYRpArVALKrYRHTz1C1fPj,1031284
[2020-01-29T00:08:57.144655704Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002240 (within slot 1031284). My next leader slot is 1031285
[2020-01-29T00:08:57.144661255Z INFO solana_core::replay_stage] vote bank: Some(1031284) reset bank: 1031284
[2020-01-29T00:08:57.144692924Z INFO solana_core::replay_stage] new fork:1031285 parent:1031284 (leader) root:1031271
[2020-01-29T00:08:57.366057122Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002304 reached, clearing working_bank 1031285
[2020-01-29T00:08:57.366556072Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031285i shredding_time=8344i insertion_time=4185i broadcast_time=9785i receive_time=189196i seed=2i num_shreds=65i slot_broadcast_time=217793i
[2020-01-29T00:08:57.367109765Z INFO solana_runtime::bank] bank frozen: 1031285 hash: A4vg1vvXg6z2ASbqvJev5vRzBr72wDPk9XM4s3UprN2w accounts_delta: BankHash 4fbeaac243dede9d5bf4f60243304137a083886c158f1ad8c82e58dd5ab5f921 signature_count: 2 last_blockhash: 6jsrXTVTUY5iYE5HEYAiHxp6vopXHt5kvX37kDiz4AaH
[2020-01-29T00:08:57.367140112Z INFO solana_runtime::bank] accounts hash slot: 1031285 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 1015298080, total_data_len: 2070, num_executable_accounts: 0 }
[2020-01-29T00:08:57.367261491Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031285i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=221498i total_entries=0i total_shreds=0i
[2020-01-29T00:08:57.367645854Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031285 16280024170406676724000884 110738943859025922626055724 1031284
[2020-01-29T00:08:57.367654280Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031285i weight="D776DE0F1681579DD2874"
[2020-01-29T00:08:57.367677925Z INFO solana_core::replay_stage] voting: 1031285 110738943859025922626055724
[2020-01-29T00:08:57.367971368Z INFO solana_core::poh_recorder] reset poh from: 7VMzSHXqhKXRFufqACuf8vk3jrbbdoWsokyjdJMAKG9L,66002304,1031285 to: 6jsrXTVTUY5iYE5HEYAiHxp6vopXHt5kvX37kDiz4AaH,1031285
[2020-01-29T00:08:57.367990053Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002304 (within slot 1031285). My next leader slot is 1031286
[2020-01-29T00:08:57.367997687Z INFO solana_core::replay_stage] vote bank: Some(1031285) reset bank: 1031285
[2020-01-29T00:08:57.368042662Z INFO solana_core::replay_stage] new fork:1031286 parent:1031285 (leader) root:1031271
[2020-01-29T00:08:57.586535834Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002368 reached, clearing working_bank 1031286
[2020-01-29T00:08:57.587046896Z INFO solana_runtime::bank] bank frozen: 1031286 hash: CC4o6nDoWfFciUpqp39u6e6Fm42Fv5X5b2n3f5KUiwHQ accounts_delta: BankHash 828cdf2869d7dfc52d76b7b4ec46fd75bfddb9269a1556f1d107d8aa9fa6909d signature_count: 2 last_blockhash: AZ8cG9fNiYxnWv11mDLTz6FUekQfwGVhVgFTp3Dswx8Q
[2020-01-29T00:08:57.587094035Z INFO solana_runtime::bank] accounts hash slot: 1031286 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 1015298080, total_data_len: 2070, num_executable_accounts: 0 }
[2020-01-29T00:08:57.587612802Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031286i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=217941i total_entries=0i total_shreds=0i
[2020-01-29T00:08:57.587735092Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031286i shredding_time=8147i insertion_time=3614i broadcast_time=9984i receive_time=189038i seed=7i num_shreds=65i slot_broadcast_time=216142i
[2020-01-29T00:08:57.587875947Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031286 16280024170406676724000884 127018968029432599350056608 1031285
[2020-01-29T00:08:57.587886988Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031286i weight="D776DE0F1681579DD2874"
[2020-01-29T00:08:57.587923386Z INFO solana_core::replay_stage] voting: 1031286 127018968029432599350056608
[2020-01-29T00:08:57.588352805Z INFO solana_core::poh_recorder] reset poh from: 4BDdjkCKUqDg95jg6m3cnrg1gEhJtjLCrhSrnTP9UGLm,66002368,1031286 to: AZ8cG9fNiYxnWv11mDLTz6FUekQfwGVhVgFTp3Dswx8Q,1031286
[2020-01-29T00:08:57.588376560Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002368 (within slot 1031286). My next leader slot is 1031287
[2020-01-29T00:08:57.588386899Z INFO solana_core::replay_stage] vote bank: Some(1031286) reset bank: 1031286
[2020-01-29T00:08:57.588437505Z INFO solana_core::replay_stage] new fork:1031287 parent:1031286 (leader) root:1031271
[2020-01-29T00:08:57.708601175Z WARN solana_metrics::counter] COUNTER:{"name": "poh_recorder-tick_lock_contention", "counts": 85, "samples": 1000, "now": 1580256537708, "events": 0}
[2020-01-29T00:08:57.708618958Z WARN solana_metrics::counter] COUNTER:{"name": "poh_recorder-tick_overhead", "counts": 2196, "samples": 1000, "now": 1580256537708, "events": 2}
[2020-01-29T00:08:57.794202337Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002432 reached, clearing working_bank 1031287
[2020-01-29T00:08:57.794447719Z INFO solana_runtime::bank] bank frozen: 1031287 hash: 4ydG7QPBo2RhK7u2NwHmxRQVJdCm7g9PZ8kqht7wb7ZK accounts_delta: BankHash d0090810c6be5e80beab6a26624211602db8f197b1391cd37b3dcbda1c83b930 signature_count: 2 last_blockhash: 38HcWAUVbyZqfSpiHSPokkCEfgZmkJXXDDChxAZKebFP
[2020-01-29T00:08:57.794490129Z INFO solana_runtime::bank] accounts hash slot: 1031287 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 1015298080, total_data_len: 2070, num_executable_accounts: 0 }
[2020-01-29T00:08:57.794633359Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031287i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=205062i total_entries=0i total_shreds=0i
[2020-01-29T00:08:57.795596593Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031287i weight="D776DE0F1681579DD2874"
[2020-01-29T00:08:57.795597905Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031287 16280024170406676724000884 143298992199839276074057492 1031286
[2020-01-29T00:08:57.795650795Z INFO solana_core::replay_stage] voting: 1031287 143298992199839276074057492
[2020-01-29T00:08:57.796865152Z INFO solana_core::poh_recorder] reset poh from: 7nd6xXKhHVRLFnjDyxYYMZM4ckQWPvhxma6WuS7LMJrD,66002432,1031287 to: 38HcWAUVbyZqfSpiHSPokkCEfgZmkJXXDDChxAZKebFP,1031287
[2020-01-29T00:08:57.796885420Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002432 (within slot 1031287). My next leader slot is 1031288
[2020-01-29T00:08:57.796889207Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031287i shredding_time=8049i insertion_time=4051i broadcast_time=11885i receive_time=176433i seed=9i num_shreds=65i slot_broadcast_time=203185i
[2020-01-29T00:08:57.796894236Z INFO solana_core::replay_stage] vote bank: Some(1031287) reset bank: 1031287
[2020-01-29T00:08:57.796962365Z INFO solana_core::replay_stage] new fork:1031288 parent:1031287 (leader) root:1031271
[2020-01-29T00:08:58.006552420Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002496 reached, clearing working_bank 1031288
[2020-01-29T00:08:58.006888814Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031288i shredding_time=7587i insertion_time=4429i broadcast_time=11510i receive_time=182758i seed=10i num_shreds=65i slot_broadcast_time=206392i
[2020-01-29T00:08:58.014764595Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031288i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=216683i total_entries=0i total_shreds=0i
[2020-01-29T00:08:58.014767140Z INFO solana_runtime::bank] bank frozen: 1031288 hash: GHnPyZHXD8pa3fbeEK7QyZYTdH5xzjsiFHvfdS79ajVC accounts_delta: BankHash c12fb76ae93aa00496ea543694afb97ef9bdf2d4339ce82ccfbae77bad5177a5 signature_count: 2 last_blockhash: GtKBatH9K7moK4giyQFxVfeEgZ74SaC9GvbgAqu5mdFy
[2020-01-29T00:08:58.014798419Z INFO solana_runtime::bank] accounts hash slot: 1031288 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 1015298080, total_data_len: 2070, num_executable_accounts: 0 }
[2020-01-29T00:08:58.015545716Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031288i weight="D776DE0F1681579DD2874"
[2020-01-29T00:08:58.015541859Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031288 16280024170406676724000884 159579016370245952798058376 1031287
[2020-01-29T00:08:58.015602032Z INFO solana_core::poh_recorder] reset poh from: HTD7YAX84ahP1yt6fzreok11nDRmit3eEbKoaggaJiq9,66002497,1031288 to: GtKBatH9K7moK4giyQFxVfeEgZ74SaC9GvbgAqu5mdFy,1031288
[2020-01-29T00:08:58.015615969Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002496 (within slot 1031288). My next leader slot is 1031289
[2020-01-29T00:08:58.015626188Z INFO solana_core::replay_stage] vote bank: None reset bank: 1031288
[2020-01-29T00:08:58.015629895Z WARN solana_core::replay_stage] PARTITION DETECTED waiting to join fork: 1031288 last vote: Vote { slots: [1031287], hash: 4ydG7QPBo2RhK7u2NwHmxRQVJdCm7g9PZ8kqht7wb7ZK, timestamp: None }
[2020-01-29T00:08:58.015653459Z INFO solana_metrics::metrics] datapoint: replay_stage-partition slot=1031288i
[2020-01-29T00:08:58.015687594Z INFO solana_core::replay_stage] new fork:1031289 parent:1031288 (leader) root:1031271
[2020-01-29T00:08:58.226335591Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002560 reached, clearing working_bank 1031289
[2020-01-29T00:08:58.226616290Z INFO solana_runtime::bank] bank frozen: 1031289 hash: D7MQj5bqYic2gdCCjDhB6NZ8xigDGLDz22Xr2X2tT6wp accounts_delta: BankHash c12fb76ae93aa00496ea543694afb97ef9bdf2d4339ce82ccfbae77bad5177a5 signature_count: 0 last_blockhash: 52uKMEbeP1Jj55PecZtg332WTXfxErb95icgcNBj4uHW
[2020-01-29T00:08:58.226643170Z INFO solana_runtime::bank] accounts hash slot: 1031289 stats: BankHashStats { num_removed_accounts: 0, num_added_accounts: 0, num_lamports_stored: 0, total_data_len: 0, num_executable_accounts: 0 }
[2020-01-29T00:08:58.227385749Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031289i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=210264i total_entries=0i total_shreds=0i
[2020-01-29T00:08:58.227658432Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031289 16280024170406676724000884 175859040540652629522059260 1031288
[2020-01-29T00:08:58.227693438Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031289i shredding_time=8083i insertion_time=4692i broadcast_time=9895i receive_time=182981i seed=5i num_shreds=64i slot_broadcast_time=207466i
[2020-01-29T00:08:58.227715660Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031289i weight="D776DE0F1681579DD2874"
[2020-01-29T00:08:58.227721481Z INFO solana_core::poh_recorder] reset poh from: DshvVrTDqD32ETAYKgUmNskrAwWHez6eCzyRYQTdDmf5,66002560,1031289 to: 52uKMEbeP1Jj55PecZtg332WTXfxErb95icgcNBj4uHW,1031289
[2020-01-29T00:08:58.227737772Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002560 (within slot 1031289). My next leader slot is 1031290
[2020-01-29T00:08:58.227746288Z INFO solana_core::replay_stage] vote bank: None reset bank: 1031289
[2020-01-29T00:08:58.227860453Z INFO solana_core::replay_stage] new fork:1031290 parent:1031289 (leader) root:1031271
[2020-01-29T00:08:58.450862015Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002624 reached, clearing working_bank 1031290
[2020-01-29T00:08:58.451182518Z INFO solana_runtime::bank] bank frozen: 1031290 hash: EXmHrp2dLQfV8AoHc1GfgBTYMHBgGhM92WXJC4FmgkN2 accounts_delta: BankHash a1cb6bb85940b0136a139badac9ccd1c65f27ae7f8f30ed1efc799e1c906f89a signature_count: 2 last_blockhash: 6fFJeUYuqjb1sE3c1S6sAYmiEXAYiTN86aaX9YdRvTiW
[2020-01-29T00:08:58.451210711Z INFO solana_runtime::bank] accounts hash slot: 1031290 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 500000890880, total_data_len: 2070, num_executable_accounts: 0 }
[2020-01-29T00:08:58.451560270Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031290i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=222408i total_entries=0i total_shreds=0i
[2020-01-29T00:08:58.452505189Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031290 15206282378332371226011764 191065322918985000748071024 1031289
[2020-01-29T00:08:58.452546357Z INFO solana_core::replay_stage] voting: 1031290 191065322918985000748071024
[2020-01-29T00:08:58.452854306Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031290i weight="C940E3831345579DD2874"
[2020-01-29T00:08:58.452868143Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031290i shredding_time=7815i insertion_time=4815i broadcast_time=9910i receive_time=194960i seed=17i num_shreds=65i slot_broadcast_time=220678i
[2020-01-29T00:08:58.453000722Z INFO solana_core::poh_recorder] reset poh from: 3XMeHiTKTQ2oyw45XtD18agJAzebgGnyoUUre36Zf8sP,66002624,1031290 to: 6fFJeUYuqjb1sE3c1S6sAYmiEXAYiTN86aaX9YdRvTiW,1031290
[2020-01-29T00:08:58.453022002Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002624 (within slot 1031290). My next leader slot is 1031291
[2020-01-29T00:08:58.453030097Z INFO solana_core::replay_stage] vote bank: Some(1031290) reset bank: 1031290
[2020-01-29T00:08:58.453034776Z WARN solana_core::replay_stage] PARTITION resolved fork: 1031290 last vote: Vote { slots: [1031290], hash: EXmHrp2dLQfV8AoHc1GfgBTYMHBgGhM92WXJC4FmgkN2, timestamp: None }
[2020-01-29T00:08:58.453111861Z INFO solana_core::replay_stage] new fork:1031291 parent:1031290 (leader) root:1031271
[2020-01-29T00:08:58.673650886Z INFO solana_core::poh_recorder] poh_record: max_tick_height 66002688 reached, clearing working_bank 1031291
[2020-01-29T00:08:58.674018378Z INFO solana_runtime::bank] bank frozen: 1031291 hash: Ed7mXPAzLpoTuyxHCNbEqed1CpKRfyNxLjFrg882n4UP accounts_delta: BankHash 1a07e3b2faec61a925ada67ab1eb2f2aadcff3066c9236fc2a6bff876578e6b4 signature_count: 4 last_blockhash: 5i6UWdABy5DhEGhUS8KN5VQuN231g4D5cwugPAJWpdw3
[2020-01-29T00:08:58.674050107Z INFO solana_runtime::bank] accounts hash slot: 1031291 stats: BankHashStats { num_removed_accounts: 4, num_added_accounts: 0, num_lamports_stored: 501016188960, total_data_len: 4140, num_executable_accounts: 0 }
[2020-01-29T00:08:58.674122794Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=1031291i fetch_entries_time=0i fetch_entries_fail_time=0i entry_verification_time=0i replay_time=0i replay_total_elapsed=219986i total_entries=0i total_shreds=0i
[2020-01-29T00:08:58.674533057Z INFO solana_metrics::metrics] datapoint: broadcast-bank-stats slot=1031291i shredding_time=9086i insertion_time=4509i broadcast_time=9161i receive_time=189080i seed=3i num_shreds=66i slot_broadcast_time=217760i
[2020-01-29T00:08:58.675403216Z WARN solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH slot_weight: 1031291 16280024170406676724000884 207345347089391677472071908 1031290
[2020-01-29T00:08:58.675453420Z INFO solana_core::poh_recorder] reset poh from: J768HhP4Z9FPv2WC6C7Rzj7awjhWhifwNaUwufs7WCf2,66002688,1031291 to: 5i6UWdABy5DhEGhUS8KN5VQuN231g4D5cwugPAJWpdw3,1031291
[2020-01-29T00:08:58.675473478Z INFO solana_core::replay_stage] 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH reset PoH to tick 66002688 (within slot 1031291). My next leader slot is 1031300
[2020-01-29T00:08:58.675483247Z INFO solana_core::replay_stage] vote bank: None reset bank: 1031291
[2020-01-29T00:08:58.675488997Z WARN solana_core::replay_stage] PARTITION DETECTED waiting to join fork: 1031291 last vote: Vote { slots: [1031290], hash: EXmHrp2dLQfV8AoHc1GfgBTYMHBgGhM92WXJC4FmgkN2, timestamp: None }
[2020-01-29T00:08:58.675554741Z WARN solana_metrics::metrics] datapoint: bank_weight slot=1031291i weight="D776DE0F1681579DD2874"
[2020-01-29T00:08:58.675571342Z INFO solana_metrics::metrics] datapoint: replay_stage-partition slot=1031291i
Raw
leader-schedule
Slot Leader
1031280 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
1031281 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
1031282 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
1031283 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
1031284 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
1031285 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
1031286 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
1031287 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
1031288 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH !!!! PROBLEM !!!! No leader vote
1031289 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
1031290 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
1031291 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH !!!! PROBLEM !!!! No leader vote
1031292 4vPqTnfH2ud6hp1yFSFRy9t9xhm8sGDwU4amcZGr2gT7 <-- this node was too slow,
1031293 4vPqTnfH2ud6hp1yFSFRy9t9xhm8sGDwU4amcZGr2gT7 <-- and didn't produce at 1031292
1031294 4vPqTnfH2ud6hp1yFSFRy9t9xhm8sGDwU4amcZGr2gT7 <-- until well after 1031296 was produced
1031295 4vPqTnfH2ud6hp1yFSFRy9t9xhm8sGDwU4amcZGr2gT7 <--
1031296 2tvTYUajoBgeAd66Zhd5Nc2BgKGCgdmasR94fpBokzop <== this node was broken
1031297 2tvTYUajoBgeAd66Zhd5Nc2BgKGCgdmasR94fpBokzop <==
1031298 2tvTYUajoBgeAd66Zhd5Nc2BgKGCgdmasR94fpBokzop <==
1031299 2tvTYUajoBgeAd66Zhd5Nc2BgKGCgdmasR94fpBokzop <==
1031300 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH Skipped slot
1031301 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH Skipped slot
1031302 5KTouxSFVt5d6DyN4oCMASQXzvaeXdwqLNuptRfWAJGH OK, leader vote present
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment