On March 27-28 the Ethereum network suffered from extremely high rate of miss slots. Most of these slots were first relayed from the bloXroute relays. We identified that the bloXroute relays worked properly throughout the incident, publishing blocks and blobs correctly, however they propagated the blocks fast thru the BDN while the blobs sidecar propagated through the p2p more slowly (the sidecar is expected to propagate slower, and is allowed to be accepted until t=8 sec) this uncovered a specific CL behavior which caused clients to reject these blocks and cause missed slots. In the current Lighthouse version, the node is expecting the peer that first provided the block to also provide the blobs. The BDN does not propagate blobs and that caused the BDN connected consensus nodes to ignore blocks that were first received from the BDN. A recent release of the BDN improved the speed of gossiped blocks without blobs, relying on the rest of the p2p network to propagate blobs as needed which caused the significant increase of the missed slots. The BDN relies heavily on Lighthouse, which makes up the majority of our beacon nodes at bloXroute, due to its performance and speed. Post release we witnessed successful block propagation through our BDN and made the assumption this release was valid. This also showcased itself mainly on the bloXroute relays due to their tight coupling with the BDN. The BDNs speed of providing the beacon nodes with the block caused this behavior even in scenarios where other relays were publishing blocks that bloXroute did not have.
Throughout this time the bloXroute relays were providing blocks with blobs back to validators and also publishing blocks with blobs to our BDN and to our network of beacon nodes. These publish requests would return a 202 response due to the beacon nodes already seeing that block from the BDN.
This issue was able to be resolved after a series of tests were done isolating this issue to lighthouse’s behavior after seeing a block first through the BDN and then slowly migrating our relay away from using the BDN for block publishing and then disabling the BDN’s block propagation of any blocks containing blobs.
sidecars are the same size magnitude as blocks. you expect the to propagate at roughly the same speed and there to commonly be a race between which a particular node might get first at any give slot (the block or an associated blob).
There is not a strict acceptance cut-off at 8s. It is unlikely under normal conditions that a block with late blobs (4s***) would garner attestations and be accepted in the canonical chain but under adverse network conditions, a block whose blobs are found after 4, 8, or even 12+ seconds can be seen as valid and available and integrated into the block tree.
This not accurately explaining the lighthouse issue. Lighthouse is highly well tested to expect race conditiosn between blocks and blobs on the p2p and has no assumptions about which peers send which.
This issue you ran into is only via the HTTP api and not a p2p rule/issue. Thus this is not in relation to "peers" as the HTTP API is not a p2p API. This is in relation to advanced publishing techniques in which you were trying to share blobs directly to nodes you control. These were thus dropped before ever hitting the p2p.