Design sketch: "reject" p2p message

reject.md

Design sketch: "reject" p2p message

Motivation

Mike Hearn has been lobbying for an "error" message in the Bitcoin p2p protocol for years (at least since the "ban peers if they send us garbage" denial-of-service mitigation code was pull-requested). This came up again with my proposed "smartfee" changes, which would drop low-priority or low-fee transactions.

In short, giving peers feedback about why their blocks or transactions are dropped or why they are being banned should help interoperability between different implementations, and will give SPV (simplified payment verification) clients feedback when their transactions are rejected due to insufficient priority or fees.

error" versus "reject"

I've gone with "reject" as the message, because "error" implies that the peer did something wrong-- and doing something wrong is only one of several possible reasons a connection, transaction, or block may be rejected.

##Specification

reject

One new message type "reject" is introduced. It is sent directly to a peer in response to a "version", "tx" or "block" message.

For example, the message flow between two peers for a relayed transaction that is rejected for some reason would be:

--> "inv"
<-- "getdata"
--> "tx"
<-- "reject"

common payload

Every "reject" message begins with the following fields. Some messages append extra, message-specific data.

Field Size	Description	Data type	Comments
?	response-to-msg	var_str	Message that triggered the reject
1	reject-code	uint8_t
?	reason	var_str	Human-readable message for debugging

The human-readable string is intended only for debugging purposes; in particular, different implementations may use different strings. The string should not be shown to users or used for anthing besides diagnosing interoperability problems.

The following reject code categories are used; in the descriptions below, "server" is the peer generating the reject message, "client" is the peer that will receive the message.

Range	Category
0x01-0x0f	Protocol syntax errors
0x10-0x1f	Protocol semantic errors
0x40-0x4f	Server policy rule

rejection codes common to all message types

Code	Description
0x01	Message could not be decoded

reject version codes

Codes generated during the intial connection process in response to a "version" message:

Code	Description
0x11	Client is an obsolete, unsupported version
0x12	Duplicate version message received

reject tx payload, codes

Transaction rejection messages extend the basic message with the transaction id:

Field Size	Description	Data type	Comments
32	txid	char[32]	transaction that is rejected

The following codes are used:

Code	description
0x10	Transaction is invalid for some reason (invalid signature, output value greater than input, etc.)
0x40	Not mined/relayed because it is "non-standard" (type or version unknown by the server)
0x41	One or more output amounts are below the 'dust' threshold
0x42	Transaction does not have enough fee/priority to be relayed or mined

payload, reject block

Block rejection messages extend the basic message with the block header hash:

Field Size	Description	Data type	Comments
32	hash	char[32]	block (hash of block header) that is rejected

Rejection codes:

code	description
0x10	Block is invalid for some reason (invalid proof-of-work, invalid signature, etc)
0x11	Block's version is no longer supported
0x43	Inconsistent with a compiled-in checkpoint

Note: blocks that are not part of the server's idea of the current best chain, but are otherwise valid, should not trigger "reject" messages.

Implementation notes

Sending "reject" messages to old nodes does no harm-- unknown commands are ignored for extensibility (in the reference implementation, at least-- other implementations should do the same). So there is no need to bump the protocol version.

Implementors should consider what happens if an attacker either sends them "reject" messages for valid transactions/blocks or sends them random "reject" messages, and should beware of possible denial-of-service attacks (e.g. blindly notifying the user of every "reject" message received would definitely be the wrong thing to do, and even blindly writing every "reject" message to a debug.log could open up a fill-up-disk DoS attack).

Block rejection strings are a bad idea: whether or not a block is valid isn't a matter of opinion, it's defined by the protocol specification and current state of the blockchain. If a peer isn't willing to accept a block, that's because either you or they are forked; put another way, a block being rejected is an error by one or the other party except in the case of forks. (note how almost all of the rejection messages are things you should have been able to validate yourself) Giving our peers reasons gives opportunities to take action in response, which creates a whole new kind of consensus through rejection messages passing around information that we certainly don't want to make possible.

Having said that, responding to a block that passes the context independent AcceptBlock() tests, but is rejected because it wouldn't be the best blockchain tip, with what we do consider to be that tip is probably reasonable and would let that peer more quickly find out they are out of date.

ACK other reject strings.

Hm. I'm not a fan of the block messages, I feel like they'll complicate already risky block validation code... and give non-deterministic results due to permissible differences in what order the tests are run in. WRT PT's "what we consider best"... you wouldn't use a reject message there: You'd INV your best block at the peer and let it pull it if its interested.

I'm wondering if the reject messages shouldn't have a some kind of simple field that indicates if the rejection is because the peer considers it invalid vs doesn't like it due to policy reasons... though I suspect that would run into order-non-determinism as its perfectly reasonable to apply stateless policy checks before stateful validity checks.

@gmaxwell Quite correct.

Another issue with reject messages is if some alt-implementation maintainer sees blocks it's producing getting rejected, we want them to at bare minimum dig out a Bitcoin node and figure out why on that side, rather than making some hack that seems to work.

After all, if you really do have what is the valid best block, why do you care the other side rejected it? That's their problem. All you want to know is if every peer rejected it, so you can find new peers that aren't broken. You also want to see if your peers actually propagate to other peers, which suggests we shouldn't have any kind of block reject message at all - broadcast to most of your peers, and check if the ones you don't relay the block back to you.

You can apply the same logic to tx reject messages to of course, but unlike with blocks it is reasonable to change your tx's in response to a reject.

@petertodd: Which block rejection do you object to? I did not propose any reject message response corresponding to "not on the best block chain".

@gmaxwell: Switched to a HTTP-like status code, which should address your concerns (all of the "block is not valid" are same status code; people could still switch on human-readable string, we could discourage that by changing them every once in a while and making it clear that the human readable strings are not standardized.)

@gavinandresen "not on the best block chain" would be the only informational rejection I wouldn't object to for now. Again, the issue is that whether or not a block is valid has a strict definition and we don't want people to start relying on their peers in any way about that.

@petertodd: I'm still not following. Are you worried that people will implement partially-validating nodes, go to all the work of computing a valid proof-of-work, and then rely on their peers to tell them "Nope, not a valid block, try again" ???

Really?

If you have a valid block (from your pov of the chain), but a peer sends a reject maliciously that it is invalid (although it is), what does the node do? Send the block to other peers?

@gavinandresen As I commented on the pull-request, put it another way: What actions do you expect implementations to take in response to a block rejection message?

As for the "it's just for debugging" argument: again, we really don't want to encourage implementations to do anything other than investigate blocks being rejected in any way other than delving into the code deeply. It's critical that we get behavior absolutely exact for consensus, so making it easy to play "whack-a-mole" and fix bugs as they come up with patches that seem to fix the problem just invites behavior that we don't want.

@simondlr : the reference implementation does nothing by default when it receives "reject" messages. If you turn on debug logging, then it will log the first 111 bytes of the reject message in debug.log. It has no effect on what blocks are or are not relayed to peers.

@petertodd: objection duly noted, we simply disagree about whether this will encourage lazy re-implementors.

So, with this implementation is it possible now, that if a message is perfectly valid, say the transactions inputs are all good and the fee is included, that there still exists a possibility that the nodes may decide to reject a message.

What if one of the big pools or a few of them got together and decide to reject all the messages that didn't come from one of them?

The motivation for this use case seems to be to avoid spamming (denial of service). I'm curious... has Bitcoin had to worry about DoS in this way in the past. Or are you guys pre-emptively trying to solve for something that has never happened.

PS: https://en.bitcoin.it/wiki/BIP_0061 is blank and I would have added this entire gist to the Wiki but it was locked.

Can someone refer me to examples/test vectors of reject messages. Its a pain to generate them the hard way.

Still this Protocol Running and work with Virtacoin VTA currency - Protocol 70002 as Sub-version : VirtaCoin:2.9.1
https://github.com/virtacoin/VirtaCoinProject