A bug caused by Implicit memory aliasing in go’s for loops caused the blob commitment values to be all set to the last value during encoding from go struct to a gRPC encoded value. This bug surfaced when a block that was received via a gRPC submission built using the erroneous code was the top or selected bid. The error seen would be mismatched KZG commitment.
A total of 16 slots were missed causing about 3.21 ETH missed block values.
All missed slots can be found here: https://gist.github.com/benhenryhunter/6feed90aff1a93106d629fe85a88d778
We released a fix to our regional relays and saw the bug resolved from our end but found out builders who previously did not use our library for gRPC began to use it. This caused a few more blocks to slip through requiring us to shut off getHeaders for the remainder of the night on the 14th.
Here is a commit that addresses the implicit memory aliasing error happening during encoding: https://github.com/bloXroute-Labs/relay-grpc/commit/77140b8234cfc3389b6ee7127252b88272f2b0d3
After seeing some post-simulation failures for blocks that successfully landed on chain we began to investigate the bug. Due to these blocks landing on chain successfully the demotion was automatically lifted when seeing that the slot was not missed. After recognizing the bug could cause missed slots from a notice in the Boost Relays telegram group we began to shut off gRPC usage from our regional relay.
When we identified the cause for the bug we added separate blob commitment verification to all block submissions, isolated it to gRPC submissions, and then saw our hot fix resolve the problem.
[2024-03-15 5:40 PM EST] bloXroute is in the process of refunding all ETH lost during missed slots.
would you say its valid to characterize this bug as a "blob verification bug"?