Python-bitcointx is a Python3 (3.6+) library to work with Bitcoin transactions, and other related things, like addresses, keys, Bitcoin script, etc.
It is based on popular python-bitcoinlib library, but with slightly different focus. Some functionality was removed to reduce the maintenance burden, while other useful functionaliy was added (full list of changes is in release notes).
This library should be viewed as a completely separate library from python-bitcoinlib, it diverges from python-bitcoinlib significantly and breaks the compatibility when it makes sense. Still, it can accomodate most of the code written for python-bitcoinlib, with minor adjustments.
I did not release the official announcement of the library until now, so here it goes.
Why create the derivative library rather than a fork ?
What is the added features ?
BIP32 extended keys
An interface to libbitcoinconsensus
Composability
More code examples
Support for more verification flags in Script interpreter
Other conveniences
What has to go ?
Support for python versions less than Python3.6
The code to work with Bitcoin blocks and network messages.
RPC Proxy wrapper class (replaced with rpc.RawProxy equivalent)
What might be added in the future ?
On the code correctness and bugs
Pull requests and bug reports welcome !
I lead a project that has its crypto payment processing engine written in Python, and that was using python-bitcoinlib. It was very convenient library of a reasonable quality. It was relatively easy to extend the library in a minor ways, like adding support for Litecoin addresses. More importantly, it was maintained, which means that I could hope for the new features, like support for segwit addresses, to be available in the library in a reasonable timerfame. That changed in October 2018, when I heard that Peter Todd, the maintainer of python-bitcoinlib, was planning to stop maintaining it Note that since then, Peter Todd said that the library is "somewhat maintained", and that he does not plan to hand over maintenance to anyone else (EDIT: Later, the maintenance was handed over to Bryan Bishop).
My project was in need of a maintained library with support of all the latest features to work with Bitcoin, that would also be composable, maintainable, and hopefully would be easier to work with. python-bitcoinlib is an old library, and while it is decent, it still had some warts, that could not be removed without significant API breakage.
My solution was to create a new library, that would take the best from its predecessor, but would not be encumbered by strict backward-compatibility requirements.
Along with support for native Segwit addresses (that was recently merged into python-bitcoinlib, too), python-bitcointx supports BIP32 extended keys, uses libsecp256k1 for signing and verifying, can use libbitcoinconsensus to verify Bitcoin scripts, and it is much more easily extendable.
import os
from bitcointx.wallet import CCoinExtKey
from bitcointx.core.key import BIP32_HARDENED_KEY_OFFSET
xpriv = CCoinExtKey.from_seed(os.urandom(32))
print("xpriv", xpriv)
print("xpub", xpriv.neuter())
print("xpub m/0h/1h/2", xpriv.derive_path("m/0h/1h/2"))
assert(xpriv.derive_path("m/0'/1'/2'/3").neuter()
== xpriv.derive_path("m/0'/1'").derive(2+BIP32_HARDENED_KEY_OFFSET).neuter().derive(3))
(click for full example code)
# will work if libbitcointconsensus library is available in the library path
ConsensusVerifyScript(scriptSig, address0.to_scriptPubKey(), txSpend, 0, # inIdx=0
(STANDARD_SCRIPT_VERIFY_FLAGS & BITCOINCONSENSUS_ACCEPTED_FLAGS),
amount=spent_amount, witness=txSpend.wit.vtxinwit[0].scriptWitness)
# arguments are the same, except flags
VerifyScript(scriptSig, address0.to_scriptPubKey(), txSpend, 0, # inIdx=0
(STANDARD_SCRIPT_VERIFY_FLAGS - UNHANDLED_SCRIPT_VERIFY_FLAGS),
spent_amount, witness=txSpend.wit.vtxinwit[0].scriptWitness)
# will work if libbitcointconsensus library is available in the library path
ConsensusVerifyScript(scriptSig, address0.to_scriptPubKey(), txSpend, 0, # inIdx=0
(STANDARD_SCRIPT_VERIFY_FLAGS & BITCOINCONSENSUS_ACCEPTED_FLAGS),
amount=spent_amount, witness=txSpend.wit.vtxinwit[0].scriptWitness)
# arguments are the same, except flags
VerifyScript(scriptSig, address0.to_scriptPubKey(), txSpend, 0, # inIdx=0
(STANDARD_SCRIPT_VERIFY_FLAGS - UNHANDLED_SCRIPT_VERIFY_FLAGS),
spent_amount, witness=txSpend.wit.vtxinwit[0].scriptWitness)
import os
from bitcointx.wallet import CCoinExtKey
from bitcointx.wallet import P2WPKHCoinAddress
from bitcointx.core import CTransaction, CTxIn, CTxOut, COutPoint, CTxInWitness
from bitcointx.core.script import (
CScript, CScriptWitness, SIGHASH_ALL, SIGVERSION_WITNESS_V0,
)
from bitcointx.core.scripteval import (
VerifyScript, STANDARD_SCRIPT_VERIFY_FLAGS, UNHANDLED_SCRIPT_VERIFY_FLAGS
)
from bitcointx.core.bitcoinconsensus import (
ConsensusVerifyScript, BITCOINCONSENSUS_ACCEPTED_FLAGS
)
xpriv = CCoinExtKey.from_seed(os.urandom(32))
random_txid = os.urandom(32)
priv0 = xpriv.derive_path("m/0h/1h/0").priv
pub0 = priv0.pub
pub1 = xpriv.derive_path("m/0h/1h/1").pub
address0 = P2WPKHCoinAddress.from_pubkey(pub0)
address1 = P2WPKHCoinAddress.from_pubkey(pub1)
scriptSig = CScript([]) # scriptSig is empty for native segwit spends
# NOTE: this is a mock transaction, so the amount is arbitrary.
# it should match the actual previous amount in the real case (with segwit).
spent_amount = 1100
# NOTE: CTransaction has nVersion=2 by default in python-bitcointx
txSpend = CTransaction([CTxIn(COutPoint(random_txid, 0), scriptSig,
nSequence=0xFFFFFFFF)],
[CTxOut(1000, address1.to_scriptPubKey())],
nLockTime=0, nVersion=1)
# NOTE: sigversion is by default SIGVERSION_BASE
sighash = address0.to_redeemScript().sighash(
txSpend, 0, SIGHASH_ALL, amount=spent_amount,
sigversion=SIGVERSION_WITNESS_V0)
# sign the sighash
sig = priv0.sign(sighash) + bytes([SIGHASH_ALL])
# make transaction mutable to be able to set fields
txSpend = txSpend.to_mutable()
# set the witness for txin 0
txSpend.wit.vtxinwit[0] = CTxInWitness(CScriptWitness([sig, pub0]))
# will work if libbitcointconsensus library is available in the library path
ConsensusVerifyScript(scriptSig, address0.to_scriptPubKey(), txSpend, 0, # inIdx=0
(STANDARD_SCRIPT_VERIFY_FLAGS & BITCOINCONSENSUS_ACCEPTED_FLAGS),
amount=spent_amount, witness=txSpend.wit.vtxinwit[0].scriptWitness)
# arguments are the same, except flags
VerifyScript(scriptSig, address0.to_scriptPubKey(), txSpend, 0, # inIdx=0
(STANDARD_SCRIPT_VERIFY_FLAGS - UNHANDLED_SCRIPT_VERIFY_FLAGS),
spent_amount, witness=txSpend.wit.vtxinwit[0].scriptWitness)
print("verification succeeded") # or else VerificationError will be raisedConsensusVerifyScript() and VerifyScript() accept the same arguments, so they are intended to be interchangeable (ConsensusVerifyScript() can in addition accept libbitcoinconsensus library handle).
(ConsensusVerifyScript() also requires the flags argument to be specified, but will not require this in the next version)
The reason to use ConsensusVerifyScript() over VerifyScript() is that, obviously, ConsensusVerifyScript() is Bitcoin consensus-compatible, because it uses the library from Bitcoin Core itself. You need to have libbitcointconsensus library in your library path, though, as it is not included in python-bitcointx.
VerifyScript() is not, and will most likely will never be fully consensus-compatible, as maintaining the full compatible script interpreter is a lot of work. It may be useful for debugging or learning purposes, or maybe some other uses not related to strict consensus-compatible script verification, so it is was not removed from the library.
python-bitcoinib had the notion of different 'chain parameters', so that you could change address prefixes, and some other parameters used by the classes in the library. This allowed to switch between Bitcoin regtest, testnet and mainnet, and allowed you to support addresses with other prefixes by subclassing the chain parameter classes, but:
- The parameters changed globally, so that after you switch, all the classes in the library would use the new parameters, and if you switched from regtest to main net parameters, all the existing address object instances would suddenly become mainnet addresses
- If you wanted to change the behavior of just one of the classes in your code external to the library, like make a subclass of
CTxOutwith certain custom properties, the library itself could not use your class, and would still create standard BitcoinCTxOutclasses internally.
So if you wanted to build a library to work with a Bitcoin sidechain, like Elements, you would need to replicate a lot of code.
Elements and Liquid network
Elements platform powers the Blockstream's Liquid network, that has several very interesting properties, and also has a very capable technical team behind it. Having an ability to add support for Liquid network into Simplexum was the primary motivation for me to develop the library to work with Elements data structures.
While developing and testing the code of python-elementstx I found and reported several bugs in Elements code, that was quickly fixed. One serious bug was found particularly because python-elementstx allowed to build the data structure that Elements RPC API did not allow, but python-elementstx ability to construct transactions in an unusual way allowed to uncover the bug.
To cleanly solve the problems listed above, I abstracted all the classes that could have different behaviour in different bitcoin-derived projects, and created a custom class dispatcher that would translate the 'frontend' class, like CTransaction, or CCoinAddress, to the 'concrete' classes, like CBitcoinTransaction and CBitcoinAddress.
The consequence of this is that regtest and testnet addresses are separate classes (CBitcoinRegtestAddress, CBitcoinTestnetAddress), and when you switch the chain parameters from 'bitcoin' to 'bitcoin/regtest', all the instances of CBitcoinAddress will maintain their Bitcoin mainnet address prefixes. When you create new CCoinAddress while regtest parameters is in effect, you will get CBitcoinRegtestAddress instance, and all currently existing instances of CBitcoinAddress or CBitcoinTestnetAddress will not change their behavior.
You can convert between the address classes with
>>> P2WPKHBitcoinTestnetAddress.from_scriptPubKey(
P2WPKHBitcoinAddress('bc1qupcdg6s4j0f8nrg7xf94dhag3qtmsk5yzz8nhp').to_scriptPubKey())
>>> P2WPKHBitcoinTestnetAddress('tb1qupcdg6s4j0f8nrg7xf94dhag3qtmsk5ygyuqvj')The custom class dispatcher also dispatches field access and method calls, so that within the methods of CBitcoinTransaction class, the bitcoin chain parameters will be in effect. Within methods of CElementsTransaction class, elements sidechain parameters will be in effect, and CTxOut will have CTxOutWitness, for example.
There is also a context manager to switch between the chain parameters. The switch is done using thread-local variable, so it will be thread-safe, when the next versions of the library support threading (as of v1.0.1 threading is not supported).
import elementstx
from bitcointx import ChainParams
from bitcointx.core import CTransaction, CBitcoinTransaction
from elementstx.core import CElementsTransaction
...
bitcoin_tx = CBitcoinTransaction.deserialze(bitcoin_transaction_data)
assert(bitcoin_tx = CTransaction.deserialze(bitcoin_transaction_data))
liquid_tx = CElementsTransaction.deserialze(liquid_transaction_data)
with ChainParams('elements/liquidv1'):
assert(liquid_tx = CTransaction.deserialze(liquid_transaction_data))
print("Number of txout witnesses:", len(tx.wit.vtxoutwit))This allowed me to implement python-elementstx library without much of the code duplication.
Note that there's also python-litecointx. Due to small differences between Bitcoin and Litecoin, it can be used as a template if you want to create the library to work with your own experimental Bitcoin-derived coin.
While python-bitcointx has only one added example to the ones already present in python-bitcoinlib (examples/derive-hd-key.py), there are several extensive code examples in python-elementstx, which include two examples of atomic swap -- between the assets within Elements, and as cross-chain atomic swap at between Bitcoin and Elements.
Although ConsensusVerifyScript() is preferred for consensus-compatible script verification, some effort was made to make VerifyScript() support more verification flags, to be closer to the original. It is unclear if we want to spend resources to extend the python script interpreter further, given that ConsensusVerifyScript() can be used with libbitcoinconsensus, but it may be a fine learning excersise.
If support in secp256k1 library is available, CKey and CPubKey has classmethods add(), combine(), sub(), negated() that allow to perform these operations on the keys. add() is implemented thorugh combine(), sub() implemented using negated(). This is used in confidential cross-chain atomic swap example to reveal the blinded privkey, rather than use HTLC-style contract: https://github.com/Simplexum/python-elementstx/blob/408746b230588bee665ecfbf8840acb7ff809233/examples/confidential-cross-chain-atomic-swap.py#L282-L285
Guard functions for script: DATA(), NUMBER(), OPCODE() - can be used to prevent accidental use of unintended values in the script where certain types of values are expected. For example, the code CScript([var]) does not communicate to the reader if var is expected to be just data, or a number, or an opcode. CScript([OPCODE(var)]) communicates this clearly.
Utility functions and methods:
- to handle multisig scripts:
standard_multisig_redeem_script,standard_multisig_witness_stack,parse_standard_multisig_redeem_script - to handle amounts:
coins_to_satoshi,satoshi_to_coins - to calculate transaction virtual size:
tx.get_virtual_size()
Python2 support is obviously not a concern, and even python-bitcoinlib is dropping support for python2. The newer version of python is needed because the custom class dispatching code relies on the new features, and making the code less clean and elegant just to support older versions did not make sense in this library, which is obviously more opinionated than its predecessor. Python3.6+ also has native support for type annotations, which will likely be used in python-bitcointx in the future.
This code is relevant if you want to interact with Bitcoin network directly, bypassing the Bitcoin Core. This is usually done to implement SPV (Simplified Payment Verification). Simplexum does not implement SPV, and relies on Bitcoin Core daemon to interact with Bitcoin network. The decision was to only leave the code that is actively used by our project -- that means that we can reasonably maintain the code to be up-to-date with current state of the technology. If required, the support for Bitcoin messages and Bitcoin blocks can be implemented in a separate library, so that maintenance of this library will be clearly separated from python-bitcointx.
The bitcoin.rpc.Proxy class was used in python-bitcoinlib to work with Bitcoin RPC in a way that the results of the calls are automatically converted to the library's class instances. When you called rpc.getrawtransaction(txid), you got CTransaction instance. The problem with this was that the API is not static, and the sets of supported RPC API commands in different BitcoinCore versions differ, as well as the parameters of those commands. The code of rpc.Proxy class has to maintain support of the latest version of the RPC API, but then it would have problems working with older Bitcoin Core versions, or with different projects. Sometimes you have to work with services that only mimic a specific subset of older Bitcoin RPC API. The level of convenience that wrapper class gave did not outweigh the increased maintenance burden and incompatibility problems. The new rpc.RPCCaller class acts like raw RPC proxy - rpc.getrawtransaction(txid) gives you the result of json.loads() on the returned data. I believe that converting JSON to higher-level objects is often belongs to the application code, that can convert only the relevant parts. And again, the universal converter from JSON into library objects can be done as a separate library, that can then be maintained separately.
Obvious candidates are:
-
PSBT (Partially signed bitcoin transactions)
As the library is aimed to work with Bitcoin transactions, along with other relevant data structures, adding support for partially-signed transactions seems appropriate. Although it may go into separate library, using the class dispatching code in python-bitcointx to support more than one types of PSBT (Elements has to have their own, for example) might be the argument against doing PSBT support as a separate library. Our Simplexum project already has code to work with PSBT, so this is not a pressing matter for us, but having more generic and composable code that can be just imported from the library would be nice.
-
Support for script descriptors
Being able to use the flexible Bitcoin Core's format for describing the output scripts might be useful if your application want to support multiple type of addresses, and it seems that this functionality will fit properly into bitcointx.wallet module (or its sub-module). There's also miniscript, that is much more expressive and is planned to be treated as "an extension to the descriptor language". But the feasibility of including that will depend on the complexity and the convenience of embedding the miniscript compiler (that is planned to be open-sourced by Blockstream, according to the previous link) into the library, or re-implementting it.
Other new features in Bitcoin, like Schnorr signatures and taproot will need to be included, after they made their way to Bitcoin Core.
The library has a lot of new code, and that means the code might contain new bugs.
That said, the library is already used in the development branch of Simplexum, that successfully passed our rigorous integration test suite, and has processed tens of thousands of random payment orders on regtest network. The multitude of wallets generated in that test use a mix of the address formats supported by Bitcoin (p2pkh, p2sh, p2sh-p2wpkh, p2sh-p2wsh, p2wpkh, p2wsh), as well as a variety of multisig configurations. The balances of the wallets on each test cycle, and the final balances, matched the expected values up to a single satoshi. So we have some confidence that at least the library code that is used by our project (and it uses a lot of this library functionality) is reasonably correct.
The possibility of bugs is not excluded, of course, as with almost all of the code. That level of correctness can only be achieved with formal verification, but even then, the bugs in the specification or the bugs in the verification tools might occur. Formally verifying the code is much more time-consuming, though, and it makes sense to use it for the most critical parts -- like the code that validates the data on the entry to the most secured part of the payment system, does transaction signing and enforces spending velocity limits inside that secured part.
On formal verification
There is a tool for static verification of Python3 - https://github.com/marcoeilers/nagini that does seem interesting, although it also seems to be more research-grade rather than production-grade tool. For the code that needs to run inside an HSM, that can have severe resource constraints and could be using exotic CPUs, SPARK2014 or Frama-C seems the most fitting formal verification tools. The SPARK2014 (that is a subset of Ada2012 specifically selected to be amendable to formal verification), is much more restricted language than C, doing formal code verification in SPARK seems to be the easier task than doing so in Frama-C, and I hope that one day there will be SPARK-bitcointx.
If you decided to use this library, and found a bug, or have a other concerns to discuss relating to the library, please file an issue at https://github.com/Simplexum/python-bitcointx/issues
If you want to contribute the code, please file a pull request at https://github.com/Simplexum/python-bitcointx/pulls
Thank for reading through this long announcement post, and I hope that you might find this library useful.