Mining improvement proposal

MiningImprovement.md

Mining improvement proposal

Simple Summary

It is cheap for GPU miner to switch the mining block, so we can dynamically adjust the mining block in a single mining round to make sure the mining block almost includes the transactions with highest fee price and includes as much valid uncle blocks as possible.

Motivation

• A transaction that the user pays for a very high fee price may need to wait for a round of mining time before it can be processed.
• The time window for a round of mining is too large, so that the miner’s fee-based transaction selection strategy is not always optimal.
• The uncle blocks can not be included as soon as possible, so that the uncle block miner may loss or reduce their block reward and current miner also may loss uncle block reward.

Mining process

1. get a state S1 from parent state.
2. Select a batch of valid transactions with the highest fees from txpool.
3. Pack these transactions to the pending state S1(S1 -> S2).
4. Select no more than 2 valid uncle blocks from the possible uncle block list.
5. Pack selected uncle blocks into the mining block and apply reward (S2 -> S3).
6. Run ethash algorithm to find a valid solution.

Optimization

1. streaming uncle blocks to mining block.

   • Once there is a new uncle block arrives, if our mining block containes less than 2 uncle blocks, add it to our uncle block list, apply the uncle block reward (S3 -> S4) and rerun 6.

2. Post uncle block event as soon as possible.

   • Miner needs to get an event for new possible uncles as soon as a new side block inserted . Currently the ChainSideEvent is fired after a full block import.

3. Regenerate mining block every 3 second.

   • Collect a batch of transaction with highest fee price every 3 second can make the mining block's reward keeping the best to some extent.
   • Rerun 2-6, fired by a time ticker.

4. Stratum mining protocol.

   • Post new mining work by websocket connection proactive.

   ​

I'll recap the optimizations we did in ethminer. The first was to optimize the switching cost (ethereum-mining/ethminer#4 cleaned up in ethereum-mining/ethminer#217). Another was to continue mining on the same head (to find uncle blocks) after a solution is found ethereum-mining/ethminer#179. And another was to submit all solutions ethereum-mining/ethminer#125. Previously ethminer would submit solutions that were one header old, but occasionally you would get a header update and then find two quick solutions on the old header (so you'd find two uncles but ethminer would only submit the first one).

Back then, I was testing the ethminer optimizations against a patched geth (patched to support uncle solutions and do frequent work updates). Here's the profiling numbers I wrote down at the time:

i'm doing more profiling, and when ethminer finds a share/block, it is idle during the round trip time to submit the solution and get new work (two round trips actually). that's around 300-500ms on my local network (network latency)
when the work update is due to a new block arriving (rather than finding and submitting a solution), there's less idle time, about 20ms
the 20ms is because ethminer stops and starts the GPU thread on every work update (the kernel loop itself is about 15ms, according to my profile). so with one work update per second you'd lose 20ms per 1000ms or ~2% efficiency
the optimized ethminer is much better. it does a work update by writing the new header to the GPU buffer without stopping the thread, so profile is ~0.2ms (200 microseconds) to update the work. and it doesn't idle after finding a solution (it just keeps searching for another nonce on the same header)

Now with ethminer optimized for frequent work updates, and geth supporting work solutions for "stale" (uncle) blocks, support in the mining pool will still be necessary.

Unless I'm mistaken, the only open source mining pool will reject "stale" solutions: https://github.com/sammy007/open-ethereum-pool/blob/bcfc0eb0e683993ad0c51bacad8d4b354e769393/proxy/handlers.go#L71-L72

EDIT: dug deeper and I was mistaken (thought I was forgetting something). it accepts solutions for the past 3 block heights. uncles are valid for up to 7 block heights, so that would be a better default: https://github.com/sammy007/open-ethereum-pool/blob/3ccd90ca1aaeb22a1679434eefc772aa8dce9124/proxy/blocks.go#L78-L86

For the big pools (which all run custom closed-source pool software), who knows? We'd have to investigate to find out. Either by asking them, probing them (register as a miner and submit "stale" shares), or my preference, attempting an analysis on uncle stats (does the pool ever find two blocks at the same height? what is its uncle rate compared to other pools, corrected for pool size? what is its rate including uncles in main blocks?).

Because of the network latency between the pool and the miners (even with Stratum), a lot of stale shares are certainly solved. When I looked around last year, I noticed some pools listed "we pay for uncle blocks" as a selling point. But that could mean they pay for fresh shares that become uncle blocks (not stale shares that become uncle blocks). If pools are rejecting stale shares, it could be a significant amount of revenue that they're losing (or not sharing with their miners). But we don't know until someone does the research.

p.s. about Stratum versions: sammy007/open-ethereum-pool#42