Making Bitcoin softforks pluggable

pluggable-softforks.txt

[bitcoin-dev] Making soft forks pluggable
    https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-October/011487.html

[Friday, October 09, 2015] [8:50:48 AM] <Luke-Jr>	CodeShark: btw, I was thinking of the softfork plugin thing a bit ago also, so sounds good to me :P
[Friday, October 09, 2015] [8:50:57 AM] <CodeShark>	:)
[Friday, October 09, 2015] [8:51:14 AM] <Luke-Jr>	maybe harder than it seems though in practice
[Friday, October 09, 2015] [8:52:04 AM] <CodeShark>	I have some ideas for how to do it...but if we're going to be doing a bunch of refactors after 0.12 is released, I figure this is a good area on which to focus :)
[Friday, October 09, 2015] [8:53:24 AM] <CodeShark>	we don't want to have to backport each individual soft fork perpetually...and it might actually be easier to backport the plugin thing
[Friday, October 09, 2015] [8:53:57 AM] <aj>	be easier to backport pluggable soft forks once the plugin thing exists too, presumably
[Friday, October 09, 2015] [8:54:14 AM] <CodeShark>	yes, of course
[Friday, October 09, 2015] [8:57:20 AM] <sipa>	shouldn't need any changes in primitives... that's only for data types, their construction, and serialization... softforks can't change those
[Friday, October 09, 2015] [8:57:30 AM] <CodeShark>	CBlock needs a new default version
[Friday, October 09, 2015] [8:57:43 AM] <CodeShark>	or could be implemented in getblocktemplate, I suppose
[Friday, October 09, 2015] [8:57:48 AM] <CodeShark>	in principle I agree
[Friday, October 09, 2015] [8:57:58 AM] <CodeShark>	CBlock should not be where the default version gets set
[Friday, October 09, 2015] [8:58:10 AM] <sipa>	yes, indeed
[Friday, October 09, 2015] [8:58:19 AM] <sipa>	those default versions there don't belong
[Friday, October 09, 2015] [8:59:14 AM] <sipa>	you say "easily merged units"... that's pretty vague
[Friday, October 09, 2015] [8:59:17 AM] <CodeShark>	right - primitives should just handle data access and serialization
[Friday, October 09, 2015] [8:59:22 AM] <sipa>	what do you mean specifically?
[Friday, October 09, 2015] [8:59:39 AM] <CodeShark>	sipa: by adding two lines to the makefile and one line to an init function
[Friday, October 09, 2015] [8:59:48 AM] <CodeShark>	and two files to the repo :)
[Friday, October 09, 2015] [9:00:11 AM] <CodeShark>	at least for starters
[Friday, October 09, 2015] [9:00:20 AM] <CodeShark>	long-term it would even be possible to deploy at runtime
[Friday, October 09, 2015] [9:00:22 AM] <sipa>	CodeShark: so you pretty much mean... changing the code location organizatiin so that soft forks are the top level?
[Friday, October 09, 2015] [9:00:51 AM] <CodeShark>	yeah, I suppose you could say that
[Friday, October 09, 2015] [9:00:55 AM] <sipa>	what if soft forks require wallet changes? or block creation code changes?
[Friday, October 09, 2015] [9:01:23 AM] <CodeShark>	we start with the consensus hooks...then worry about UI :p
[Friday, October 09, 2015] [9:01:31 AM] <sipa>	ok
[Friday, October 09, 2015] [9:02:48 AM] <sipa>	my concern is readability... hooks result in less readable execution flow, and much harder to reason about things like "whatever this hook does, the result is always soft fotk safe"
[Friday, October 09, 2015] [9:02:58 AM] <sipa>	but... perhaps
[Friday, October 09, 2015] [9:03:14 AM] <CodeShark>	well, if you take a look at how BIP65 and BIP66 are deployed, there aren't too many places really
[Friday, October 09, 2015] [9:03:32 AM] <CodeShark>	BIP65 is a tad bit more complex in the sense that it also defines an op code
[Friday, October 09, 2015] [9:04:27 AM] <CodeShark>	so we need a hook in the script interpreter
[Friday, October 09, 2015] [9:04:52 AM] <sipa>	yeah, i don't like that
[Friday, October 09, 2015] [9:04:57 AM] <sipa>	not for consensus
[Friday, October 09, 2015] [9:05:35 AM] <CodeShark>	we can keep that part separate for now, I suppose - the soft fork deployment would also modify script/interpreter.cpp if we don't add a hook
[Friday, October 09, 2015] [9:05:37 AM] <wumpus>	consensus code should be as simple and straightforward and self-contained as possible, IMO, no hooks and plugins
[Friday, October 09, 2015] [9:05:48 AM] <CodeShark>	I think this is about as simple as it gets, though
[Friday, October 09, 2015] [9:05:57 AM] <CodeShark>	it makes any change to consensus very easy to review
[Friday, October 09, 2015] [9:06:19 AM] <CodeShark>	the diff becomes trivial
[Friday, October 09, 2015] [9:06:50 AM] <wumpus>	does it? interaction/crosstalk issues tend to be problematic for plugin-like systems, what sipa says
[Friday, October 09, 2015] [9:07:14 AM] <wumpus>	needs to be easy to follow the execution flow
[Friday, October 09, 2015] [9:07:21 AM] <CodeShark>	a soft fork has well-defined abstract behavior
[Friday, October 09, 2015] [9:07:22 AM] <wumpus>	and *in which sequence* things are done
[Friday, October 09, 2015] [9:07:35 AM] <CodeShark>	the implementor just needs to specify what those behaviors actually are
[Friday, October 09, 2015] [9:09:01 AM] <CodeShark>	the execution flow is even easier to follow with this kind of architecture...because in the stable consensus code itself the specifics of the rule are encapsulated...and in the rule definition itself there's nothing else BUT the rule definition
[Friday, October 09, 2015] [9:09:24 AM] <Luke-Jr>	CodeShark: so… let's say a softfork with segregated witness..?
[Friday, October 09, 2015] [9:10:04 AM] <CodeShark>	can we do that cleanly with a soft fork?
[Friday, October 09, 2015] [9:10:13 AM] <Luke-Jr>	in theory, I don't see why not
[Friday, October 09, 2015] [9:10:20 AM] <sipa>	I don't see how.
[Friday, October 09, 2015] [9:10:23 AM] <Luke-Jr>	probably entails p2p protocol changes though
[Friday, October 09, 2015] [9:11:06 AM] <sipa>	Only in a very non-useful way could you do it, with external data blobs that blocks commit to (like extension blocks).
[Friday, October 09, 2015] [9:11:08 AM] <Luke-Jr>	well, we can't segregate the existing scripts - but P2SH-like..
[Friday, October 09, 2015] [9:11:29 AM] <sipa>	no
[Friday, October 09, 2015] [9:11:49 AM] <sipa>	segregated witness changes how transactions refer to each other
[Friday, October 09, 2015] [9:12:07 AM] <sipa>	i guess you can do it by making transactions not contain scriptSigs
[Friday, October 09, 2015] [9:12:12 AM] <Luke-Jr>	exactly
[Friday, October 09, 2015] [9:12:43 AM] <sipa>	and have separate witness structures that are relayed separately, and committed to in a block's coinbase
[Friday, October 09, 2015] [9:12:46 AM] <sipa>	ok
[Friday, October 09, 2015] [9:13:16 AM] <sipa>	yeah, that wouldn't easily fit into a hook structure, i think
[Friday, October 09, 2015] [9:13:45 AM] <CodeShark>	I'm just thinking about the consensus structures themselves - not the relay mechanisms. I see the p2p stuff as a separate layer
[Friday, October 09, 2015] [9:14:13 AM] <sipa>	yes, but even the consensus changes wouldn't easily fit in
[Friday, October 09, 2015] [9:14:16 AM] 	 * Luke-Jr , after breaking softfork plugins, runs off to bed. :P
[Friday, October 09, 2015] [9:14:50 AM] <CodeShark>	sipa: I just think of the witness structures as additional validation context
[Friday, October 09, 2015] [9:15:03 AM] <CodeShark>	they can be anything, really - any additional state required at validation time
[Friday, October 09, 2015] [9:15:45 AM] <sipa>	ok, how does that fit in?
[Friday, October 09, 2015] [9:16:03 AM] <CodeShark>	we'd probably need to rearchitect the script interpreter a bit to be more abstract
[Friday, October 09, 2015] [9:16:13 AM] <wumpus>	no...
[Friday, October 09, 2015] [9:17:10 AM] <CodeShark>	or we could just accept that not everything is solved by the plugin and some kinds of soft forks still would require additional modifications to the consensus code (but they would be far fewer than before, and the routine ones would all be covered)
[Friday, October 09, 2015] [9:18:45 AM] <CodeShark>	routine ones being stuff like nVersion changes and the kind of stuff we currently do in main.cpp
[Friday, October 09, 2015] [9:19:04 AM] <Luke-Jr>	when/if at some point we dynamically call libbitcoinconsensus, we could implement softforks as a forked lib, and have the node call libbitcoinconsensus_softfork's CheckBlock etc and also libbitcoinconsensus_original CheckBlock etc to enforce softfork behaviour; this makes the changes *simpler* and less risky
[Friday, October 09, 2015] [9:19:44 AM] <CodeShark>	yes, agreed, Luke-Jr. That's sorta what I meant by abstracting the script interpreter a bit more
[Friday, October 09, 2015] [9:19:57 AM] <Luke-Jr>	CodeShark: no, in this case we'd have entire duplication of the consensus lib ☺
[Friday, October 09, 2015] [9:20:15 AM] <Luke-Jr>	(so it can be reused for hardforks as well)
[Friday, October 09, 2015] [9:20:16 AM] <CodeShark>	no...the soft fork plugins could use the consensus lib once it's ready
[Friday, October 09, 2015] [9:20:26 AM] <Luke-Jr>	CodeShark: well, that's not what I just described
[Friday, October 09, 2015] [9:20:45 AM] <Luke-Jr>	and we'll want to do these dynamic calls anyway if we support sidechains in a single node ever
[Friday, October 09, 2015] [9:21:08 AM] <CodeShark>	I'm saying at the core level, there are a few things that are fundamental and invariant - amongst these are the block header tree, PoW, and version rules
[Friday, October 09, 2015] [9:21:39 AM] <CodeShark>	then on top of this you have core structures (blocks and transactions) which generally can only change their fields with a hard fork
[Friday, October 09, 2015] [9:21:50 AM] <CodeShark>	so let's say they are invariant as well
[Friday, October 09, 2015] [9:22:14 AM] <CodeShark>	then on top of this you have context (UTXO, timestamp)
[Friday, October 09, 2015] [9:22:41 AM] <CodeShark>	if these checks work, then you run the script
[Friday, October 09, 2015] [9:23:39 AM] <sipa>	we don't even get to do simple refactors
[Friday, October 09, 2015] [9:23:55 AM] <CodeShark>	sipa: understand this is not a one or two month plan :p
[Friday, October 09, 2015] [9:24:01 AM] <CodeShark>	this is probably out at least a year
[Friday, October 09, 2015] [9:24:18 AM] <CodeShark>	point is unless we start thinking about this kind of stuff well in advance it will probably never happen
[Friday, October 09, 2015] [9:24:20 AM] <sipa>	i'm not convinced it's very useful
[Friday, October 09, 2015] [9:24:34 AM] <CodeShark>	ok, then I have some homework to do :)
[Friday, October 09, 2015] [9:24:44 AM] <sipa>	the trivial softforks it is targetting are already simple to review
[Friday, October 09, 2015] [9:24:51 AM] <CodeShark>	for you, sipa
[Friday, October 09, 2015] [9:24:58 AM] <CodeShark>	I'd rather you spend your time doing other stuff :p
[Friday, October 09, 2015] [9:25:03 AM] <CodeShark>	for most people it's still very hard
[Friday, October 09, 2015] [9:25:19 AM] <sipa>	the non-trivial ones will become harder, because they'll need to change the plugin api...
[Friday, October 09, 2015] [9:25:39 AM] <CodeShark>	not if we abstract things well enough
[Friday, October 09, 2015] [9:26:15 AM] <CodeShark>	there's a sequence of validation steps that can be easily abstracted regardless of the specific rules
[Friday, October 09, 2015] [9:27:16 AM] <sipa>	i don't see how you could create an abstraction that keeps working with seggregated witness, for example
[Friday, October 09, 2015] [9:27:40 AM] <CodeShark>	isn't segregated witness effectively additional state required for validation?
[Friday, October 09, 2015] [9:28:03 AM] <sipa>	seggregated witness, from the point of existing code, removes script and sigScript entirely
[Friday, October 09, 2015] [9:28:08 AM] <CodeShark>	let
[Friday, October 09, 2015] [9:28:23 AM] <sipa>	and then jntroduces a new primitive data structure, to which consensus rules are applied
[Friday, October 09, 2015] [9:28:55 AM] <sipa>	i think you're biased by having seen a few simple soft forks
[Friday, October 09, 2015] [9:28:57 AM] <CodeShark>	removes script and sigScript?
[Friday, October 09, 2015] [9:29:27 AM] <sipa>	yes, transactions would end up having an empty scriptSig
[Friday, October 09, 2015] [9:29:38 AM] <CodeShark>	I get that part - what do you mean by "script"
[Friday, October 09, 2015] [9:29:39 AM] <CodeShark>	?
[Friday, October 09, 2015] [9:29:45 AM] <CodeShark>	scriptPubKey?
[Friday, October 09, 2015] [9:29:49 AM] <sipa>	by script i mean the script subdir in the code
[Friday, October 09, 2015] [9:30:04 AM] <CodeShark>	oh...so interpreter.cpp
[Friday, October 09, 2015] [9:30:15 AM] <CodeShark>	and that stuff :)
[Friday, October 09, 2015] [9:30:16 AM] <sipa>	of course, it would get reused in a new piece of code
[Friday, October 09, 2015] [9:30:45 AM] <CodeShark>	yes, as I pointed out the script interpreter could be far better abstracted to cover vastly more potential use cases...and with the benefit of hindsight
[Friday, October 09, 2015] [9:30:57 AM] <sipa>	but from the point of existing code - and that's what your abstraction framework can deal with - it just completely deletes the concept of script from bitcoin
[Friday, October 09, 2015] [9:31:31 AM] <sipa>	how do you deal with cases where more data needs to be exposed to the script interpreter?
[Friday, October 09, 2015] [9:31:42 AM] <CodeShark>	I imagine a function Validate(context, transaction)
[Friday, October 09, 2015] [9:31:50 AM] <CodeShark>	where context could be extended
[Friday, October 09, 2015] [9:32:12 AM] <CodeShark>	but for now it covers the current block tree
[Friday, October 09, 2015] [9:32:16 AM] <CodeShark>	and utxo set
[Friday, October 09, 2015] [9:32:20 AM] <sipa>	you can't just expose the entirety of consensus state to transaction validation, or it could break caches that assume certain data doesn't influence validation outcome
[Friday, October 09, 2015] [9:32:28 AM] <sipa>	no, it's not that easy
[Friday, October 09, 2015] [9:32:54 AM] <sipa>	script execution can purposefully not depend on certain data
[Friday, October 09, 2015] [9:32:59 AM] <sipa>	like height
[Friday, October 09, 2015] [9:33:26 AM] <sipa>	to avoid transactions going from valid to invalid
[Friday, October 09, 2015] [9:34:01 AM] <CodeShark>	who says the context structure needs to maintain everything? it could be pruned...and if Validate requires missing context when called it can throw an error
[Friday, October 09, 2015] [9:34:25 AM] <CodeShark>	I agree it isn't trivial
[Friday, October 09, 2015] [9:34:30 AM] <CodeShark>	it's a challenge - but I think it's doable
[Friday, October 09, 2015] [9:34:54 AM] <sipa>	i think you're underestimating the costs to others
[Friday, October 09, 2015] [9:34:56 AM] <CodeShark>	the trick here is figuring out what sorts of context can be excluded from what types of checks
[Friday, October 09, 2015] [9:35:23 AM] <CodeShark>	and what sorts of context are desirable to exclude
[Friday, October 09, 2015] [9:35:41 AM] <CodeShark>	again, this is not a one- or two-month plan
[Friday, October 09, 2015] [9:36:02 AM] <sipa>	ok: bottom line, i am very skeptical about the benefits
[Friday, October 09, 2015] [9:36:15 AM] <sipa>	now let's talk about things we can actually do
[Friday, October 09, 2015] [9:36:58 AM] <CodeShark>	but we CAN do this...it's just going to take a much longer time than one release cycle
[Friday, October 09, 2015] [9:37:33 AM] <CodeShark>	the argument that it doesn't have that many benefits is an acceptable one - but that it isn't possible I don't accept :)
[Friday, October 09, 2015] [9:37:44 AM] <CodeShark>	you might be able to convince me of the former
[Friday, October 09, 2015] [9:37:51 AM] <sipa>	it's certainly possible with high costs
[Friday, October 09, 2015] [9:38:09 AM] <CodeShark>	given our current process, I agree
[Friday, October 09, 2015] [9:38:11 AM] <sipa>	but i don't think it's the right direction to.go in
[Friday, October 09, 2015] [9:38:20 AM] <CodeShark>	ok - that's more fruitful discussion
[Friday, October 09, 2015] [9:38:28 AM] <sipa>	i would aim to have less abstractions, not more
[Friday, October 09, 2015] [9:38:40 AM] <sipa>	because they complicate reasoning and proving what code can do
[Friday, October 09, 2015] [9:38:57 AM] <CodeShark>	that depends on how the logic is chunked, though
[Friday, October 09, 2015] [9:39:09 AM] <sipa>	maybe
[Friday, October 09, 2015] [9:39:16 AM] <sipa>	i'm not saying there are no benefits
[Friday, October 09, 2015] [9:39:18 AM] <CodeShark>	the mind is capable of handling craploads of complexity as long as it's well-encapsulated with a simple interface
[Friday, October 09, 2015] [9:39:27 AM] <sipa>	but it will come at a high cost, and not just reviee cost
[Friday, October 09, 2015] [9:40:32 AM] <sipa>	CodeShark: unless the API is perfect, and the different modiles are guaeanteed to be isolated from each other, i don't think the abstraction you can introduce is sufficient for consensus purposes
[Friday, October 09, 2015] [9:40:50 AM] <sipa>	you will need to know what all plugins are doing to see whether the result is consensus safe
[Friday, October 09, 2015] [9:41:43 AM] <CodeShark>	if the plugins have interdependencies it does potentially add a significant amount of complexity, especially circular dependencies
[Friday, October 09, 2015] [9:41:51 AM] <sipa>	that's not what i mean
[Friday, October 09, 2015] [9:42:28 AM] <CodeShark>	with this architecture you could simulate whatever rules you wanted and review specific units that focus exactly on the rules in question
[Friday, October 09, 2015] [9:43:05 AM] <sipa>	in theory
[Friday, October 09, 2015] [9:43:34 AM] <CodeShark>	sipa: I just don't think bitcoin can perpetually keep on adding new features at the protocol level if each proposed change requires a handful of people such as yourself to review the same lines of code over and over again :)
[Friday, October 09, 2015] [9:43:56 AM] <CodeShark>	this is process scalability
[Friday, October 09, 2015] [9:43:58 AM] <sipa>	CodeShark: i will still review them if they are in plugins
[Friday, October 09, 2015] [9:44:12 AM] <sipa>	i don't see how this changes anything
[Friday, October 09, 2015] [9:44:13 AM] <wumpus>	at least some people are going to know those lines of code *very* well then
[Friday, October 09, 2015] [9:44:15 AM] <CodeShark>	yes, but the author can be tasked with proving them out considerably more before you even touch them
[Friday, October 09, 2015] [9:44:50 AM] <wumpus>	... which is kind of the goal, people need to understand the consensus rules and how everything fits together
[Friday, October 09, 2015] [9:44:51 AM] <CodeShark>	we can have a review process where stuff gets better screening before it gets to you
[Friday, October 09, 2015] [9:44:51 AM] <sipa>	i don't believe that
[Friday, October 09, 2015] [9:45:09 AM] <sipa>	i think more abstraction will only result in more ways a plugin could screw with consensus
[Friday, October 09, 2015] [9:45:38 AM] <sipa>	unless it is very minimal, and thus nearly useless