Over a roughly 2 hour window, the max profit relay delivered 9 blocks that failed to be added to chain due to wrong block hash. The relay’s auto demotion check was not working correctly to demote this builder.

missedSlotBids.md

[
{
"slot": "8366820",
"parent_hash": "0x27dcb717d5921af93e2b56a82f546e1c11b33619238510cbe36e4d8c43f446a8",
"block_hash": "0x6868f8e474a163bb17f39d05847aa8e3f1b38db3cca0dfc9bd139db1da8dcde2",
"builder_pubkey": "0x8e6df6e0a9ca3fd89db2aa2f3daf77722dc4fbcd15e285ed7d9560fdf07b7d69ba504add4cc12ac999b8094ff30ed06c",
"proposer_pubkey": "0xb6d5424e28a738d002c96a19db7434fff22877272649e0ca38b579bb44398f3977f43af6c055414b7a71ec2bd7cb8480",
"proposer_fee_recipient": "0x72FDdC41CA177551Ce5949C75a92b945eFa04141",
"gas_limit": "30000000",
"gas_used": "9713903",
"value": "25974747686278448",
"num_tx": "120",
"block_number": "19169473"
},
{
"slot": "8366796",
"parent_hash": "0x4713ace81757839b3c0bf137889c4d4568f8eae563e37a0a8410d1258f25a139",
"block_hash": "0x19d8a7520dfbe419c11213dad07c011a64b850bb5cf838df918164540d9b9363",
"builder_pubkey": "0x8e6df6e0a9ca3fd89db2aa2f3daf77722dc4fbcd15e285ed7d9560fdf07b7d69ba504add4cc12ac999b8094ff30ed06c",
"proposer_pubkey": "0x84e86c0987ab3f27856a0733f530bbafbfeff34de922475851de8173ebc284a8fd5693ce88660b97eb96911bf9dfcb20",
"proposer_fee_recipient": "0x4675C7e5BaAFBFFbca748158bEcBA61ef3b0a263",
"gas_limit": "30000000",
"gas_used": "9942654",
"value": "20614487402014643",
"num_tx": "123",
"block_number": "19169451"
},
{
"slot": "8366782",
"parent_hash": "0x0d54e50b1dda70dea62653c4b338eed13156bd8763c3d52cd34117dab7a28875",
"block_hash": "0xa40dad4111142b21148383baf366bee083e6a4688779502561a01be3fff78e42",
"builder_pubkey": "0x8e6df6e0a9ca3fd89db2aa2f3daf77722dc4fbcd15e285ed7d9560fdf07b7d69ba504add4cc12ac999b8094ff30ed06c",
"proposer_pubkey": "0x94c42e1d66accbb73d4406e7c8d85bbdff6446d915f2745c609145201aa67374f6a681bf9dd2c1fb9b75a6f78a831199",
"proposer_fee_recipient": "0xA8C62111e4652b07110A0FC81816303c42632f64",
"gas_limit": "30000000",
"gas_used": "24848080",
"value": "156182547611636384",
"num_tx": "203",
"block_number": "19169438"
},
{
"slot": "8366772",
"parent_hash": "0x05f8d8dc22c9c0073d9226ef778f11f7453eed0a073a81313cfc89bb96edf411",
"block_hash": "0xfa2d5f53f24420ba488f067497183d015c5cfe15c698e81d863ebabf61f1681d",
"builder_pubkey": "0x8e6df6e0a9ca3fd89db2aa2f3daf77722dc4fbcd15e285ed7d9560fdf07b7d69ba504add4cc12ac999b8094ff30ed06c",
"proposer_pubkey": "0x87a4d97172106189068a63d7c2482145dab0f39889a702bec6aae4f57c3458c5dfbabf0b62f6feaaf5840a64aadb8317",
"proposer_fee_recipient": "0xa3d8D276f6eaB243cB22f4D1A5e3Cf08476346EF",
"gas_limit": "30000000",
"gas_used": "16928838",
"value": "59797900937841359",
"num_tx": "238",
"block_number": "19169429"
},
{
"slot": "8366661",
"parent_hash": "0xda93ed9a5e250e1dbb822f5fd6bf4bd208293b81907a88e533c6df04209de467",
"block_hash": "0x8c83c2a629b43cf1f715704f7837bb62beb03359d3d9c28a1272e15c8d6667c1",
"builder_pubkey": "0x8e6df6e0a9ca3fd89db2aa2f3daf77722dc4fbcd15e285ed7d9560fdf07b7d69ba504add4cc12ac999b8094ff30ed06c",
"proposer_pubkey": "0x95ac90ba252023e8ad9ff5d4196aa413cc2d2dcf299c7a3d931838bb5f6f8cff95daf693cb21418b5ab2769fd8ec4c9b",
"proposer_fee_recipient": "0x94f7b863d39e81bB17AEbB998AB5f3a48e775dE8",
"gas_limit": "30000000",
"gas_used": "19856929",
"value": "118768445676649612",
"num_tx": "190",
"block_number": "19169319"
},
{
"slot": "8366513",
"parent_hash": "0x59ff31be054c93fdf9abffc2885106883301a9ce16d3ab5fb44316385361cf68",
"block_hash": "0x1ed00179426eb222bbcc302a6b3dac5940afa246455b57d3fc40ee7ecee01614",
"builder_pubkey": "0x8e6df6e0a9ca3fd89db2aa2f3daf77722dc4fbcd15e285ed7d9560fdf07b7d69ba504add4cc12ac999b8094ff30ed06c",
"proposer_pubkey": "0x8bae3b8c3c9311d36c824462c0172ecfcb5c840381d0b92e22c3fed2f0f9b48b2042899ee24276311196fb4e54a71658",
"proposer_fee_recipient": "0x388C818CA8B9251b393131C08a736A67ccB19297",
"gas_limit": "30000000",
"gas_used": "16777893",
"value": "117021094633455503",
"num_tx": "151",
"block_number": "19169173"
},
{
"slot": "8366508",
"parent_hash": "0xf313526a8c2054c32a6e5c6c26bd1656c7775be4d680b553d9d2ea1502804d39",
"block_hash": "0x5e84feb5ace45c5433d735c8d5c5aa7f786f306d376d1037c090660d82e144c6",
"builder_pubkey": "0x8e6df6e0a9ca3fd89db2aa2f3daf77722dc4fbcd15e285ed7d9560fdf07b7d69ba504add4cc12ac999b8094ff30ed06c",
"proposer_pubkey": "0x8aeed47c257aae56b70d8388678415a12ff686f87242c2df89df06acb3ee130f7da655bd16d5368aebb6a19a747feee0",
"proposer_fee_recipient": "0xC42f80913B33Ba0B6002F3B5B8Ad335ED9A137e3",
"gas_limit": "30000000",
"gas_used": "14156242",
"value": "92101569672087368",
"num_tx": "150",
"block_number": "19169169"
},
{
"slot": "8366383",
"parent_hash": "0x3a099d9389b7c9731b0f68c16095563bed4fc45c86533b5479cc9af82fabd684",
"block_hash": "0xe54a219967603710a59f35f841fd2ec10e16481167ee9f889044c9273acdc9d5",
"builder_pubkey": "0x8e6df6e0a9ca3fd89db2aa2f3daf77722dc4fbcd15e285ed7d9560fdf07b7d69ba504add4cc12ac999b8094ff30ed06c",
"proposer_pubkey": "0x95b68304a7353d0286e5c1d48f23443f51e3153c01c2554cb01172c718d60779e90682c19df38becbc683d4fcdf0cde8",
"proposer_fee_recipient": "0x5b3F88f027C6d115713Df8e1f57D996666B88F7b",
"gas_limit": "30000000",
"gas_used": "11603480",
"value": "40151294531800951",
"num_tx": "171",
"block_number": "19169048"
},
{
"slot": "8366319",
"parent_hash": "0xe83def4b19ce5362d09036fc0308f715c3557c5585fd53fd5996276ba52dbf4f",
"block_hash": "0xd543680bc98b32be616c8b24f97bfd4e3da3f3721f80d5f252a0832a8960b09e",
"builder_pubkey": "0x8e6df6e0a9ca3fd89db2aa2f3daf77722dc4fbcd15e285ed7d9560fdf07b7d69ba504add4cc12ac999b8094ff30ed06c",
"proposer_pubkey": "0xa2c653dfc954ec571ef3e5da21ac693a90f72453d93d407f5775f6755905eea1d07ddc0576f7b168beeb4502c2f7750d",
"proposer_fee_recipient": "0xF135B9eD84E0AB08fdf03A744947cb089049bd79",
"gas_limit": "30000000",
"gas_used": "29997666",
"value": "336627698167516219",
"num_tx": "211",
"block_number": "19168985"
}
]

postMortem.md

Overview

During the morning of Feb 6th, a builder released a change using a custom gRPC implementation that was submitting improperly built blocks. From these blocks 9 won the auction but failed to be broadcasted due to the improper formatting of the blocks. The relay's auto-demotion logic had faulty checks for treating timeouts and failures differently which resulted in this builder not getting demoted automatically.

We were first notified of the problem at 08:52 AM EST and disabled the optimistic submissions for that builder at 9:06 AM EST.

Impact

Of the 600 blocks proposed in the 2 hours window, 9 blocks were missed with the following values below:

slot, value, pubkey
8366820, 25974747686278448, "0x72FDdC41CA177551Ce5949C75a92b945eFa04141",
8366796, 20614487402014643, "0x4675C7e5BaAFBFFbca748158bEcBA61ef3b0a263",
8366782, 156182547611636384, "0xA8C62111e4652b07110A0FC81816303c42632f64",
8366772, 59797900937841359, "0xa3d8D276f6eaB243cB22f4D1A5e3Cf08476346EF",
8366661, 118768445676649612, "0x94f7b863d39e81bB17AEbB998AB5f3a48e775dE8",
8366513, 117021094633455503, "0x388C818CA8B9251b393131C08a736A67ccB19297",
8366508, 92101569672087368, "0xC42f80913B33Ba0B6002F3B5B8Ad335ED9A137e3",
8366383, 40151294531800951, "0x5b3F88f027C6d115713Df8e1f57D996666B88F7b",
8366319, 336627698167516219, "0xF135B9eD84E0AB08fdf03A744947cb089049bd79",

Totaling 0.967 ETH

Root Cause

Ultimately the relay should have caught these and demoted this builder immediately after the first failure. The underlying bug that caused this to surface was due to some false positives bringing up specific edge cases for successfully delivered blocks. This logic was modified to ignore these false positives but in the case of this builder the logic was not catching these failures properly and thus did not demote the builder automatically.

Resolution

The faulty builder had optimistic submissions disabled as soon as we were alerted and we began work on bug fixing the post simulation code. We had since added increased test coverage to weed out this type of failure that was being handled improperly and improved our post simulation logic to correctly handle this.

BloXroute holds collateral for builders using optimism for their block submissions but with this incident since it is related to our post simulation demotion logic we will be handling the payment for missed slots while working with the builder to fix their implementation.