Author: Roy Badami
This document started out life as an intended post to the BUIP004 thread on the Bitcoin Unlimited forums, but grew too large to be posted there.
You can find the thread and proposal that this is in response to at https://bitco.in/forum/threads/buip004-rbf-double-spending-support.694/
As most people reading this presumably already know, there are a range of views on zeroconf transactions. At one extreme of the spectrum is the idea that zeroconf is so intrinsically unsafe that no one should be using zeroconf transactions at all - so it would actually a good thing to essentially remove the option of using zeroconf transactions. On the other end of the spectrum is of course the view (and I think the majority view in the community) that zeroconf transactions are useful – important, even - and we shouldn't undermine them. Hopefully everyone from BU reading this is on side with the latter viewpoint, since it's in the Articles.
(It's not clear that all views clearly lie on this spectrum, though. Peter Todd's 'scorched earth' approach although relying on full RBF, which makes zeroconf useless at a technical level, uses a game theoretic approach to discourage double spend attempts, thus ironically making zeroconf useable in practice - essentially, 'we had to destroy the village in order to save it'. I'm not going to discuss scorched earth further here, though, since there doesn't seem to be any significant support for it either within Core or in the wider community.)
So, what's the main problem that people want to solve? Transactions that don't pay a large enough fee to get mined into a block in a timely manner (if at all). When this happens, the transaction will just sit in the mempool, but never make it into a block. Mainly this happens at the moment due to many wallets being pretty poor at determining suitable fees – if I'm not mistaken then many wallets were quite recently still using flat rate fees not even taking into account transaction size. But even if wallets calculate sensible fees, there's always the risk of a sudden influx in transactions (e.g. a so called 'stress test') causing full blocks and pushing up confirmation times.
Currently when this happens it's somewhat problematic. Since the original transaction will be in the mempool of most/all nodes, the wallet can't just retransmit it with a higher fee, because the new transaction will be seen as a double spend and rejected. It will neither be relayed nor (if you manage to get it a pool) will it be mined. The traditional approach is to clear it from your client's mempool, wait 72 hours for it to (hopefully) expire from the mempools of other nodes on the network, and then send a new transaction – hopefully being careful to ensure that it spends at least one of the same outputs as the old transaction, so that there's no possibility that both transactions get mined (which would mean you'd end up making the payment twice). Apart from the problem that no wallet I'm aware of automates this – so it's currently a highly manual process requiring a fair amount of knowledge about how bitcoin transactions work – if a transaction takes longer than you want/need to confirm, there's nothing you can do about it for 72 hours. Worse, AIUI there's nothing to stop someone else resubmitting the transaction to the network, either maliciously or (I've seen some suggestions) possibly inadvertently as part of the normal operation of bitcoind. So there's no guarantee as to how long you'll have to wait before you can resubmit the transaction successfully – it might be a long time.
Now, how much of a problem are stuck transactions at the moment? Certainly, they happen, and although I don't get the sense that they're a pressing problem for most Bitcoin users, it seems like a problem that does need to be solved. Certainly, too, many of those in favour of creating a fee market will presumably feel an urgent need to solve this before we start to see fees start to rise – as stuck transactions will very probably become quite a normal occurrence in thei world - but it doesn't seem to me that there's any real disagreement that this is a problem that should be solved. We only disagree about the urgency, and perhaps the approach.
It should also be noted that there are all sorts of smart contracts envisaged where there might be a deadline by which to send a transaction, and sending it late could have serious consequences, including loss of funds. This applies to simple traditional payment channels as descrribed at https://en.bitcoin.it/wiki/Contract#Example_7:_Rapidly-adjusted_.28micro.29payments_to_a_pre-determined_party If the channel isn't closed before it expires the refund transaction becomes spendable and the sender can steal the money back. I believe Lightning has similar problems, as I suspect do all payment channel protocols, and very probably many other kinds of contracts, too.
So any application of contracts that has the requirement to send transactions in a timely manner is probably going to have a good way of handling stuck transactions before it's ready for produciton deployment.
Some people argue that increasing the block size limit removes the stuck transaction problem, and hence we don't need to solve it, but I don't really buy that; even if the limit were entirely removed tomorrow, individual pools will still apply limit the size of block they create (i.e. soft limts) just as they have throughout Bitcoin's history. So it's still possible that transaction backlogs could build up at times. Worse, if people have smart contracts that rely on timely execution, then it's eminently plausible that fraudsters might attempt to mount a 'stress test' to deliberately create a backlog to cause a contract to fail to execture correctly.
Since the problem with a stuck transaction is generally that the transaction doesn't pay a high enough fee to make it into a block, the solution is to offer – by some means or other – to pay a higher fee to pools in order to persuade them to include it into a block. A number of ways to accomplish this have been proposed, namely, CPFP, RBF, FSS-RBF, opt-in RBF, which I discuss below.
“Child pays for parent” or CFPF follows from the fact that the bitcoin protocol allows spending of outputs of unconfirmed transactions. Wallets don't generally do this (except, in some cases, to spend change outputs if no other coins are available) but it's perfectly possible to construct such transactions. So imagine that transaction A is stuck in the mempool, not being mined because its fee is too low. We can construct a new transaction B that spends one of the outputs of A with a suitably high fee. CFPF relies on the observation that a pool will optimise their fee revenue by considering A and B together. Since B spends one of A's outputs, it can't be mined without mining A as well, but if the combined fee is high enough it's in the pools interests to mine it; essentially an excess fee on transaction B can make up for a shortfall on transaction A. If this still isn't enough, longer chains can be constructed: a further transaction C could spend one of the outputs of B. pools will then, presumably, consider A, B and C together, and as long as the combined fee is high enough, pools will mine all three transactions.
CFPF is neat. I like it. It requires no changes to bitcoin (except to pools), and it allows either the sender or the recipient to increase the fees. So a sender can generate a new transaction spending one of their change outputs to clear a stuck transaction; alternatively a merchant could generate a transaction to unstick a payment to them – indeed, a merchant could advertise the fact that they will pay the fees for all transactions. The Eligius pool implements CFPF, but I don't know if any others do. Unfortunately it's not implemented in Core – and although that's not deeply important to non-mining nodes, implementing it might encourage more pools to adopt the practice.
I can see why the proponents of a fee market don't see it as a good solution, though: it's expensive, both in terms of fees, and in terms of block space usage. If, as in my example above, I end up having to increase my fee twice before the transaction gets mined, then I end up paying the fee for three transactions (A, B and C) instead of just one. Not only that, but three transactions end up in the blockchain when one would have surficed.
So CPFP is great for emergency unsticking of transactions, but not so great if you think our wallets will spend all their time in the brave new world continually adjusting fees to try and get the transactions mined.
Replace by fee – here refered to as full RBF to distinguish it from the modified forms of RBF discussed below – involves completely ditching the traditional 'first seen' policy of the mempool. Traditionally, if a received transaction conflicts with a transaction already in the mempool – that is, it tries to spend the same inputs – then it is dropped by the node. Full RBF changes the rules: if a transaction conflicts with an existing transaction and, loosely speaking, it pays a higher fee, then it replaces the existing transaction; that is, the existing transaction is dropped from the mempool and the new transaction enters the mempool.
Now, if you believe that zeroconf is of no value or, perhaps more charitably, you believe that the security model of zeroconf is so broken that regretably we must abandon it, then this approach presents itself as a perfectly simple solution to the stuck transaction problem. If a transaction is stuck, just broadcast a new transaction – essentially the transaction you wish you had broadcast in the first place. Nodes will see this conflicts with the original transaction – essentially a double spend – but rather than drop it, because the fee is higher.
Proponents argue that this is rational pool behaviour since it nets them more fees, and hence ultimately unavoidable. Furthermore, there's little to nothing that non-mining nodes can do to stop pools behaving in this way, since a pool that chose to implement full RBF can (and, in the presence of a hostile P2P network presumably would) provide a means to submit the transaction directly to the pool.
Full RBF is highly controversial, because it completely breaks zeroconf transactions: although it can be used simply to bump the fee, the replacement transaction could instead send the funds to a completely different destination. So I could broadcast a transaction to pay a merchant, and then before that transaction is mined, I could broadcast a replacement transaction (with a higher fee) which instead sends the funds back to me.
Full RBF as described above is not currently implemented in Bitcoin Core, and AIUI is not planned for Bitcoin Core 0.12. Although a number of prominent members of the Bitcoin community are in favour of full RBF, to date all pull requests proposing to include this in Core have been blocked. Pieter Wuille has recently expressed opposition to including this functionality in Core and Jeff Garzik has expressed opposition in the past; if Jeff's position is unchanged on this then that means that at least two of the four Core committers are against implementing this. Still, it's impossible to say what decisions Core will make in future (e.g. for a 0.13 release) and there is certainly pressure on Core's leaders from various prominent figures to include this functionality, at least as an optional feature that can be enabled via a configuration option.
No pool currently implements full RBF. F2Pool briefly did, but promptly removed RBF due to the public outcry. They stated they would replace it with FSS-RBF (see below) although I don't know if they actually did.
First seen safe RBF is an attempt to fix the double spend problem inherent in full RBF while still enabling its core purpose of allowing fees to be bumped. Simply put, FSS-RBF will only accept a replacement transaction if it includes all the same outputs, paying at least as much, as the original transaction did. A replacement transaction can increase the fee, it can increase the payment to an output, and it can add additional outputs – but what it can't do is remove or reduce an output. Prospective replacement transactions that don't meet these criteria are treated just like they are now, and simply dropped.
FSS-RBF is safe to zeroconf since transaction replacement can't be used to cancel or reduce a zeroconf payment. Unfortunately FSS-RBF, at least as currently specified, also adds cost and complexity.
Since you're not allowed to decrease *any* output, including the change output – remember that the change output is not special in a bitcoin transaction but just a normal output – in order to increase the fee you're going to have to add another input. This increases the size of the transaction a little (and therefore the required fee) each time you bump the fee. Furthermore this also adds complexity to wallets – and a wallet could even conceivably run out of available outputs to use.
When F2Pool abandoned full RBF they stated they were replacing it with FSS-RBF but I don't know if they actually did or if they are still running it.
Opt-in RBF was proposed by Peter Todd in Bitcoin Core pull request #6871. It was accepted by Core and is expected to be included in Core 0.12.0. It is this decision that gives rise for the need by BU (and other implementations) to decide how to respond.
Opt-in RBF allows the sender of the transaction to specify whether or not the transaction should be subject to RBF processing. The criterion is chosen in such a way that existing normal transactions will not be affected; however a transaction can opt in to RBF be setting nSequence on at least one of the inputs to 0xfffffffd or less.
Existing wallets set all inputs' sequence number to 0xffffffff which means their transactions will be unaffected (unless they choose to deliberately generate transactions that opt in). Note that if a transaction does opt in then it is subject to full RBF processing, not FSS-RBF processing. The rationale for this is to avoid requiring an additional input every time the fee is increased (see FSS-RBF, above).
This allows anyone who has a use for RBF to send transactions subject to RBF processing while allowing the rest of us continue to send zeroconf transactions. However, it comes with a cost: anyone who wishes to accept zeroconf transactions is going to have to check whether the transaction has opted in to RBF. This means changes to wallets and merchant processing software. Moreover, this cost is born in the main not by those who want to use RBF, but by those who don't.
The change does come with some complexity – e.g. it's necessary to check the entire chain of unconfirmed transactions, if there is one, since replacing one will also kill its descendents. However it would be perfectly possible for wallets to simply not display incoming RBF transactions to the user until they get mined into a block. It is a bit of work, admittedly, but it can certainly be done.
So, Bitcoin Core has made the decision to give us opt-in RBF, and since they are currently the dominant node type on the network that's a significant change to the Bitcoin network. Granted, I'm not happy with how they did this – they made a change that requires action by ever wallet and merchant processor without, as far as I am aware, any outreach to the community. But in any case, this is where we find ourselves.
My starting point has always been that, in the spirit of permissionless innovation, if there is a new transaction type that some people believe will be useful to them, then the wider community should allow them to experiment with it. In many ways it's no different from introducing new opcodes such as OP_CTLV or new sequence number semantics, both of which increase the flexibility of Bitcoin and are therefore a good thing. The only argument I've seen against opt-in RBF is that it requires changes to existing code to recognise opt-in transactions.
That in itself is not a convincing argument to me; few of us, I think, would argue that Bitcoin is somehow 'finished', and we should stop making changes to the protocol just to give the developers an easy life. We know that there are bound to be many changes over the coming years that are going to require changes to existing infrastructure. One recent case in point: the network now requires low S signatures. Of course, this was an uncontroversial transaction malleability fix, but it still required every wallet to update their code or be unable to reliably send transactions.
My preferred way for wallets to handle receiving an opt-in transaction would be not to show it to the user at all until it is confirmed. This would be compatible with current user expectations; the opt-in transaction would appear to be take longer to arrive at the recipient's wallet, but would be just as safe once it does.
Merchant payment processors are affected too, of course, although I think the cost is not great. Many merchant processors do (or can) accept zeroconf transactions, but they are also already capable of being configured to require confirmed transactions (at least, I believe Bitpay allows the merchant to configure this). Given this, requiring at least one confirmation for opt-in transactions shouldn't be that great.
Of course, that doesn't mean we can't disagree as to whether the cost in developer time justifies the benefits of this new transaction type, but in principle, the necessary development work can be done by all participants, and then there will be no damage to existing functionality and we will gain the ability to use a (perhaps not very useful) new type of transaction.
My starting position, therefore, was that given Core's decision will essentially force the necessary developer work to be done by wallets and merchant payment processors in order to protect against the negative consequences that would otherwise ensue, we may as well at least have to ability to use this new type of transaction. I discussed the possible use in smart contracts, including Lightning and other payment channels in PART I; I also mention another possible use for high value transactions in PART VI.
In fact, a non-mining full node really can't do much to prevent opt-in RBF being misused for double spends: assuming that at least one major pool decides to implement opt-in RBF for whatever reason, either political or technical, then one has to assume they will provide some means of submitting transactions directly to a node under their control. Even if they don't, submitting the transaction to the relay network will in all likelihood result in pools seeing it. We have to assume therefore that someone attempting a double spend will take steps to submit the transaction to an RBF pool.
In actual fact, there's probably little we can do to prevent opt-in RBF transactions being relayed over the P2P network anyway. We'd have to have the vast majority of nodes being BU nodes before we'd be able to significantly impact the propagation of opt-in transactions. Furthermore Bitcoin XT nodes currently implement a feature known as 'double spend relay'; as a result of this they will always relay double spends. This feature is designed to allow merchants to build more reliable zeroconf payment systems by ensuring that they always are alerted to double spend attempts, but it also has the side effect of ensuring that XT nodes will always relay RBF transactions, even though they won't accept them into their own mempools. Given the current hiatus in XT maintainership, it's not clear whether this behaviour is likely to change in the short term.
So, in summary, we actually have very little capability to block the downside of opt-in RBF; we may, potentially, have some small ability to sabotage the legitimate use case, but only if the vast majority of nodes were BU.
However, the above analysis made two assumptions which I now realise may not be valid: firstly, that some non-negligible fraction of the hash power will be willing to mine opt-in RBF transactions, and secondly that wallets and merchant payment processors will modify their software to ensure that they recognise opt-in transactions and protect their users from them.
I don't know whether most pools have made any public statement regarding opt-in RBF but it is, of course, possible that no major pool will decide to run an opt-in RBF node. In that case it largely won't matter whether the P2P network supports opt-in RBF or not; as long as the original transaction propagated to the mining pool's node(s), then the pool (which doesn't support opt-in RBF) will not accept the replacement transaction into its mempool and therefore will not mine it.
Of course, there's still a small possibility that the replacement transaction might be mined, perhaps because a pool for some reason didn't see the original transaction or because its mempool had to be reset for some reason. Alternatively it might be mined by a p2pool node or other small miner running an unmodified bitcoind – but such miners make up a vanishingly small proportion of hash power these days.
If no major pool implements opt-in RBF, then of course wallets may decide its not worth their time to add any code to recognise opt-in transactions. I'd expect the major merchant payment processors will still decide to implement defensive measures against opt-in as a precaution, but wallet support is likely to be much more patchy as long as the risks remain negligible.
I had hoped to keep politics out of this discussion but unfortunately it's probably not possible. Due to the lack of effective community engagement by Core (which I suspect stems largely from the fact that there is no organisational support since ties between Core and the Bitcoin Foundation were cut) the introduction of this feature had been handled badly, leading to much misinformed debate within the broader non-technical Bitcoin community. Commonly held views include
- Full RBF has already been merged into Core
- Bitcoin Core has stated that is will merge full RBF into a future release of Core
- Opt-in RBF inevitably leads to a double spend risk
- The purpose of opt-in RBF is to force people onto the Lightning network
- The purpose of opt-in RBF is to provide an undo button in wallets
However, the negative public perception of opt-in RBF in some quarters – which does not seem to be at all in keeping with the costs of deploying it – do mean that there are inevitably political considerations for BU's response.
It is at least possible that, due to a lack of buy-in from pools, opt-in RBF might never be activated on Bitcoin mainnet. It may be that pools would not have supported it anyway, or it may be that they have been influenced by the public reaction to Core's behaviour. If this is the case then any decision that BU makes on this will have little real world effect on the Bitcoin network and will be largely symbolic.
I make my own proposals in PART V but here I just comment briefly on the options presented in BUIP004
A1. This is the status quo at the time of writing (i.e. before the release of Bitcoin Core 0.12)
A2. This needs fleshing out technically. We could either relay all double spends, as Bitcoin XT currently does, or we could choose to only relay transactions that are valid under opt-in RBF rules (i.e. that have sufficient fee and that do, in fact, opt in). It would be helpful to clarify which of the two is being proposed (or to split A2 into two sub-options).
B. Peter Todd's rationale for not including support for FSS-RBF in his opt-in patch was lack of interest from wallet developers due to the additional complexity. Note that simply implementing FSS-RBF in our code won't mean that RBF transactions will be accepted if they don't defraud the recipient – due to issues surrounding change, transactions need to be constructed in a particular way in order to be accepted by FSS-RBF nodes. So I think adding support for FSS-RBF transactions when it's not at all clear that anyone has any intention of writing code to produce such transactions doesn't make much sense. See PART I for further discussion of FSS-RBF.
C. Same comments as above apply to FSS-RBF. In addition, I am strongly against implementing full RBF, even as an option, given how damaging full RBF would be to zeroconf. When Luke Dashjr proposed that precisely option C be implemented in Bitcoin Core (pull request #7219) it caused an outcry in the community, and the proposal was dropped following opposition from Pieter Wuille. I do not think BU should offer full RBF, even as an option. I also believe Option C would be political suicide for BU.
Given the political nature of the opt-in RBF proposal and assuming that no one can point to any statement from a major pool endorsing opt-in RBF, then my proposal is as follows:
P1. Bitcoin Unlimited should defer any implementation of opt-in RBF, and should retain the status quo transaction handling behaviour as implemented in Bitcoin Core 0.11, until it is clear that a non-negligible proportion of the Bitcoin hash power is willing to mine such transactions. Bitcoin Unlimited should commit to reevaluate its position in a timely manner if and when opt-in RBF transactions become useable on Bitcoin mainnet (for which a prerequisite is support by mining pools representing a non-neglibile proportion of hash power) with a view to relaying such transactions so as to maximise their utility to the network.
N.B. This position does not preclude the implementation of Bitcoin XT's double spend relay feature, should BU members decide this is in the interests of maximising zeroconf security; however that discussion is outside the scope of the current discussion.
Although I think any decision as to what to do if and when opt-in RBF becomes useable on mainnet should take place at that time, my expectation is that we would want to at least relay valid opt-in RBF transactions, to maximise their utility to those who want to use them. (As discussed in PART II there is not much we can do to prevent their abuse.)
As for what we should accept into our mempool, it seems reasonable to aim to keep our mempools consistent with those of those maintained by the pools representing the majority of the hash power. This maximises the utility of our nodes to the network by maintaining a view of pending transactions that is conistent with what is most likelly to be accepted into future blocks.
So two cases arise
In this case, we relay valid opt-in RBF transactions (or alternatively we relay all double spends) but we don't accept such transactions into our mempool. This corresponds to proposal A2 in BUIP004.
If support is not yet present in most wallets and/or merchant payment processors, we could agree to defer implementation of A2 for a period of time to allow the wallets and merchant payment processors to implement support; however as discussed in PART II this would be nothing more than a gesture unless BU constituted the overwhelming majority of nodes.
In this case, it makes sense to not only relay valid opt-in RBF transactions, but also accept them into our mempool. This means we adopt the Bitcoin Core 0.12 behaviour. We do this in order to keep our mempools consistent with those operated by the pools. As with case 1 we could delay implementation if that was deemed desirable.
This is just a collection of miscellaneous points from my notes that didn't end up fitting naturally into the main body of text, but seemed worthy of inclusion.
As mentioned in PART II my preferred approach to wallet support would be not to present an opt-in RBF transaction to the user at all until it has been confirmed, thus not breaking existing user expectations.
We need to check how the wallet in Core handles this; it would be somewhat irresponsible for Core to have made this change _without_ implementing some suitable rehaviour for incoming transactions, but I don't know how 0.12 actually behaves. But regardless of what Core does, we will have to decide how BU should behave (at least if opt-in RBF is adopted by miners and assuming we choose to retain wallet functionality in BU).
One suggestion that's been made that I mention briefly here is that miners could specially recognize Lightning's on chain transactions, and allow these to be replaced, without implementing a more general form of RBF. I would oppose such a development because it goes against the spirit of permissionless innovation: if RBF is useful to Lightning (or even if it's not) then it might also be useful to other people. If we're going to have this functinality at all, then having the originator of the transaction declare whether it should be subject to RBF processing is the right way to go, rather than having to concinve the mining community that your use case is important enough for them to change their code to support your need for RBF.
I've seen it suggested that BU is better for miners, since they can change the block size without changing the code. While there is a grain of truth in this, it misses the fact that in practice all pools run heavily customised code already, and that's unlikely to change.
One obvious use case for opt-in RBF is high value transactions. Consider buying a house; the way this is done in the UK is that on the day of completion the buyer's solicitor transfers the money to the seller's solicitor using a funds transfer system called CHAPS, which is a same-day funds transfer system. Once the money has changed hands the buyer can then take possession of the property. Typically everything proceeds smoothly, but sometimes there are delays in the funds transfer resulting in considerable cost and inconvenience to all parties.
But what if we were paying with Bitcoin? Obviously the seller's solicitor is not going to accept zeroconf, or even a single confirmation; let's imagine they want six confirmations. In this case, it makes perfect sense for the buyer's solicitor to opt in to RBF. Zeroconf is of no value in this scenario; on the other hand, having the ability to bump the fee could come in quite handy, e.g. if we are unfortunate enough to experience a 'stress test' on the day of completion, since if this transaction isn't mined into a block in a timely manner then the consequences could be costly and inconvenient.