wallet_and_segwit.md

I'll first give some thoughts on the design of the wallet in general, before explaining current and future SegWit wallet support.

Current design

Conceptually, a wallet currently contains mapKeys, mapScripts, and setWatchOnly, which together determine which outputs are considered "ours" (IsMine), and how to solve and sign for them. Then the wallet further contains key metadata, key chains, and key pool at the wallet level that feed into this.

We determine whether outputs are ours using ad-hoc logic that mostly answers the question "Could we sign this?", but with some notable exceptions:

• In the case of multisig outputs, all private keys must be available (rather than just the multisig's threshold).
• For witness outputs, we require the P2SH-wrapped version of the witness program to be in mapScripts, even for non-P2SH witness outputs.
• Outputs that are in setWatchOnly are ours regardless.

This logic has many problems:

• It is overly complicated, doing too many things at once (it also answers whether something is solvable/signable)
• Despite doing pretty much the same as signing, no code is shared between the two.
• It is inefficient due to pattern matching all scripts and trying to recurse into them.
• It is not selective: it can't be made to match just P2PKH or just P2PK or just P2WPKH, except by marking one of them as watch-only (which then removes the ability to sign).
• It is hard to support extra key chains (for example publically derivable HD chains for hardware wallets).

Future design

In a future design I would like to move away from loose pieces of information that define what we consider ours when the puzzle pieces fit, and instead structure everything in records. Each record would correspond to an exact scriptPubKey (or set of scriptPubKeys), and contain several pieces of information that are currently spread out:

• Output description:
  • Arbitrary scriptPubKey
    • redeemscript/witnessscript if needed
  • HD chain
    • scriptPubKey type(s) (P2PK, P2PKH, P2SH-P2WPKH, P2WPKH, ...); multiple can be present simultaneously
    • Minimal/maximal hdchain position (to accomodate converting pre-split-hd-and-then-upgraded wallets without changing master key)
  • Possibly more complicated records in the future (e.g. BIP124 like things)
• Change or payment (let's not continue using the lack of a label to indicate change)
• Optional private key information
• Metadata which should be per address/chain rather than per key (birth date, purpose)

One record could be designated as default new payment address source, and one as default new change source. As public keys can't be derived without access to private keys in case of hardened derivation, a 'cache' with precomputed public keys is needed, which replaces keypools. Furthermore, there would need to be on-the-fly computed maps of public keys and scriptPubKeys (and inner scripts) based on the data in the record. Restricting the authorative information to just this list of records means wallet files can be small, and dumps can be very simple: just an encoding of all the records, one by one. For typical records a combined public/private encoding can be devised (like Electrum's yprv/zprv) even. A typical wallet dump would just be two extended private keys (one for change, one for not), with flags about what type scriptPubKeys to use.

One thing that does not fit in this model is labels, which will need to remain stored separately. Inherently, labels are not immutable data (they're added during normal wallet operation), but they're also not necessary for protecting balance or funds.

If a concept of 'watch only' remains, I think it should be independent from having the private key inside the wallet file. Just because your actual private key is on an airgapped system or on a hardware wallet does not mean the coins are any less yours. Watch-only could be a boolean inside records exclude it from normal view - something you perhaps want for keys/scripts that are involved in a multisig with other people, but don't actually consider fully yours.

Conversion of old wallet to new ones will probably be the trickiest part. It will involve a one-time operation at startup that tries to enumerate all possible scriptPubKeys we know about, and feed them through the old IsMine logic, converting the matching ones into ours map records and private keys.

In this new model the IsMine logic can go away:

• Whether an output is ours will just be a simple check whether it occurs in the precomputed set of all scriptPubKeys defined by the records - entirely outside the scope of CKeyStore. This is efficient and easy to reason about.
• For signing, CKeyStore can become a pure interface that exposes a function to get a script given its hash, and a function that exposes a key given its pubkey - nothing else. This interface gets implemented by the wallet as lookups into its records. As a result, we can still sign anything we have enough information for.
• To determine solvability we just try to sign using a dummy signer.

To summarize, I think it's important to start thinking about wallets in terms of concise and immutable description records of what outputs are ours and how to solve and sign for them - even if that isn't how things are implemented immediately. This is in contrast to today where a wallet is an unstructured collection of keys, scripts, watch scripts, keypools, HD chains, and metadata, which all interact.

SegWit wallet support

When supporting SegWit addresses by default, we're effectively extending the existing implicit chain records to also cover P2WPKH and P2SH-P2WPKH forms. Different types of compatibility have different requirements:

1. Problem
   • Scenario:
     • The user makes a backup
     • Upgrades to default-SegWit software
     • Creates a SegWit address
     • Restores the backup
   • Solutions:
     • 1a. Treat all keys (pre-existing or not) as implicitly SegWit-compatible, regardless of whether they have any form of marker.
     • 1b. First require an explicit wallet upgrade (that needs a backup) before SegWit addresses can be created.
   • My preference is solution (1a): it is easy to implement, has no impact on file size, and the alternative (1b) is not a very good user experience.
2. Problem
   • Scenario:
     • The user upgrades to default-SegWit software
     • Makes a backup
     • Downgrades the software
     • Restores the backup
   • Solutions:
     • 2a. Automatically store the P2SH-P2WPKH script in the wallet for all keypool keys at first startup. This only works as long as we upgrade again before the original keypool runs out.
     • 2b. Make the wallet file incompatible with old versions at first startup (but don't require a new backup).
     • 2c. Ignore: assume downgrading while restoring a backup is not supported
   • My preference is (2b) if we want to deal with the versioning mess now, and (2c) otherwise. (2a) requires a significant blowup of the wallet file for now, without strong guarantees.
3. Problem
   • Scenario:
     • The user upgrades to default-SegWit software
     • Creates a new address
     • Downgrades the software
   • Solutions:
     • 3a. Store the P2SH-P2WPKH script in the wallet for every newly created address.
     • 3b. Make the wallet file incompatible with old versions at first SegWit address creation (but don't require a new backup).
   • I'm fine with either solution. If (2b) is used, (3b) is implied.
4. Problem
   • Scenario:
     • The user makes a backup
     • Upgrades to default-SegWit software
     • Creates a new address
     • Downgrades the software
     • Restores the backup
   • No solutions exist, as this situation is undetectable.
5. Problem (only if 1a)
   • Scenario:
     • The user upgrades to default-SegWit software
     • Makes a backup
     • Creates a new address
     • Gets paid on new address
     • Restores backup
     • Downgrades the software
   • Solutions:
     • 5a. Automatically store P2SH-P2WPKH scripts to the wallet whenever a key is seen in a transaction

The SegWit wallet support PR (https://github.com/bitcoin/bitcoin/pull/11403)[#11403] currently implements (1a) and (3a). An alternative (which requires dealing with versioning issues) is (1a), (2b), and (3b).

Longer term, after migrating to the new design explained in the previous sections, I don't think our choices here matter much. (1a) means we'll have to keep watching for scriptPubKeys for SegWit versions of all old keys, but that's just an in-memory map that's slightly larger, with no effect on file size and presumably performance.