matthewdowney/segwit-xpubs.md

## segwit-xpubs.md

      
    Raw
  

              segwit-xpubs.md
            
          
    Issues with the Blockchain Info API

The blockchain.info API for XPUBs only works with Legacy Bitcoin XPUBs, whereas their web interface confusingly works
with Segwit as well.
I provide a couple solutions at the end, and we end up with three useful functions:

legacy_wallet_balance
segwit_wallet_balance
segwit_wallet_balance_hack

Check it out.

For a legacy Bitcoin XPUB
xpub6CUGRUonZSQ4TWtTMmzXdrXDtypWKiKrhko4egpiMZbpiaQL2jkwSB1icqYh2cfDfVxdx4df189oLKnC5fSwqPfgyP3hooxujYzAu3fDVmz


Web interface (link) says 0.00408747 BTC
API endpoint (link) says 0.00408747 BTC


For a segwit Bitcoin XPUB
xpub6CxXsMT2YRk1CjEPqYRRXxqXPoiVsvYz66sBnyD7rEbG4XJFkYd2FG9wP3KakpuBWC15u21zcCy3g2v6Vw2GQGAqKDxHFip3jBhskd42iE7


Web interface (link) says 0.00516644 BTC BTC
API endpoint (link) says 0 BTC


So you can either hit the API with each address until the gap limit (what segit_wallet_balance does) or scrape their
web interface (what segwit_wallet_balance_hack does).
Extended Public Keys

An XPUB

Is an elliptic curve public key + some other information
Can be used to generate other XPUBs
Can be used as a public key to generate an address
Are identified by a 'path' from the root seed, m

The typical hierarchical structure (see BIP44) is as follows
                                m/44'/0'/0'/0/0
                                XPUB 0
                m/44'/0'/0'/0  /
                 Receive XPUB -
                /              \
               /                XPUB n
 m/44'/0'/0'  /                 m/44'/0'/0'/0/n
Account XPUB -
              \                 m/44'/0'/0'/1/0
               \                XPUB 0
                \              /
                 Change  XPUB -
                m/44'/0'/0'/1  \
                                XPUB n
                                m/44'/0'/0'/1/n


The Account XPUB is what you export from the ledger. If n is the account number...

For Segwit its path is m/49'/0'/n'
For Bitcoin Legacy its path is m/44'/0'/n'


The 'leaf' XPUBs (m/44'/0'/0'/0/n and m/44'/0'/0'/1/n) are used to generate Bitcoin addresses

The way to generate these addresses is different for segwit & legacy.


The Blockchain Info API


The blockchain.info API doesn't support segwit, therefore when you give it an 'Account XPUB' that corresponds to segwit
addresses, it performs the derivations to the correct leaves but then generates the address as if it were a Bitcoin
legacy address.
Therefore you need to give the blockchain info API an already generated address rather than an XPUB if you want segwit
balances.

Solution

If the xpub is for a bitcoin legacy address, call the balance endpoint with the xpub and return the result. If it's
segwit, generate addresses for both change and receive addresses and sum their balances until more than gap_limit
addresses are seen with no transaction history.
Pseudocode

let xpub = ...
if xpub is legacy:
    return call("https://blockchain.info/balance?active=" xpub)
else if xpub is segwit:
    let balance = 0
    
    # Check balance in both change and non-change addresses
    for is_change in [true, false]:
        # We want to lazily generate all possible addresses rather than compute it all at once
        let all_addresses = lazy_generate_addresses(xpub, is_change)
        let all_balances = (call("https://blockchain.info/balance?active=" a) for a in all_addresses)
        let chunked = chunk(gap_limit, all_balances)
        
        balance += sum(flatted(take_while(lambda: sum(b) != 0, chunked)))
        
    return balance

Running solution to copy & paste

First make sure python is set up (should be python 3 & probably using a virtual environment with virtualenv).
Then install dependencies...
$ pip install git+https://github.com/matthewdowney/wallet-utils.git
$ pip install git+https://github.com/prusnak/bip32utils
$ pip install pysha3

Save & run
import codecs
import json
import time
import urllib.request
from urllib.error import HTTPError
from decimal import Decimal

from wallet_utils.crypto import *

xpub = "xpub6CUGRUonZSQ4TWtTMmzXdrXDtypWKiKrhko4egpiMZbpiaQL2jkwSB1icqYh2cfDfVxdx4df189oLKnC5fSwqPfgyP3hooxujYzAu3fDVmz"
xpub_segwit = "xpub6CxXsMT2YRk1CjEPqYRRXxqXPoiVsvYz66sBnyD7rEbG4XJFkYd2FG9wP3KakpuBWC15u21zcCy3g2v6Vw2GQGAqKDxHFip3jBhskd42iE7"
ypub_segwit = "ypub6XnoB27wh7HV42RWfuD3k3w2ZmrwpYYV1DPQaN71EEy97d7V1CnasKp5QFHAkjZ6uq7teVcZ4sKbZKXfDdSHCVrSBZehqddXzumX9B3kfUF"
GAP_LIMIT = 20


def ypub_to_xpub(ypub_string):
    """
    Sometimes extended public keys are written prefixed by 'ypub' instead of 'xpub', designating a segwit address.
    (Trezor, for example, does this.) This function converts a ypub address to an xpub address.
    :param ypub_string: The ypub address, as a string.
    :return: The corresponding xpub address, as a string.
    """
    raw = Base58.check_decode(ypub_string)
    return Base58.check_encode(codecs.decode('0488b21e', 'hex') + raw[4:])  # prefix with 'xpub' then reencode


def lazy_generate_addresses(account_xpub, is_change, is_segwit):
    """
    Return a lazy generator that will provide every possible address given the parameters.
    :param account_xpub: The xpub at the account level.
    :param is_change: If true generates change addresses, otherwise generates receive addresses.
    :param is_segwit: If true generates segwit (p2pkh) addresses, otherwise generates legacy (p2sh) addresses.
    """
    node = xpub_to_child_xpub(account_xpub, 1 if is_change else 0)
    leaf_index = 0
    while leaf_index < MAX_IDX:
        leaf_xpub = xpub_to_child_xpub(node, leaf_index)
        leaf_public_key = xpub_to_pk(leaf_xpub)
        leaf_address = (pk_to_p2wpkh_in_p2sh_addr if is_segwit else pk_to_p2pkh_addr)(leaf_public_key)
        yield leaf_address

        leaf_index += 1
    return None


def chunked(size, generator):
    """
    Chunk a generator & return a new generator such that each generator.__next__() call returns a chunk of a certain
    size. Chunks are not evaluated until their generation, so an infinite generator can be used.
    :param size: The number of evaluations per chunk.
    :param generator: A generator to be evaluated in chunks.
    :return: A new generator which evaluates & returns a list of size items for each call to __next__()
    """
    iterable = iter(generator)
    while True:
        # Build the chunk
        this_chunk = []
        for i in range(size):
            try:
                this_chunk.append(iterable.__next__())
            except StopIteration:  # Generator is tapped out
                return None

        # Yield the chunk
        yield this_chunk


def http_get(url, as_json=True, headers=None, backoff_coefficient=1.5, max_backoff=10, max_retries=10):
    """
    Send an HTTP GET. Implements automatic retries with exponential backoff in case of failure.
    :param url: The URL to request.
    :param as_json: Parse the response as JSON?
    :param headers: Headers to add to the request.
    :param backoff_coefficient: The number by which to multiply the wait period for each backoff.
    :param max_backoff: The maximum wait time.
    :param max_retries: The maximum number of retries for a failed request.
    """
    for i in range(max_retries + 1):
        rq = urllib.request.Request(url)
        SPOOF_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) ' \
                      'Chrome/39.0.2171.95 Safari/537.36 '
        rq.add_header('User-Agent', SPOOF_AGENT)
        if headers is not None:
            rq.headers.update(headers)
        try:
            with urllib.request.urlopen(rq) as page:
                print("OK HTTP GET " + url)
                data = page.read()
                if not as_json:
                    return str(data)
                else:
                    encoding = page.info().get_content_charset("utf-8")
                    js = json.loads(data.decode(encoding))
                    return js
        except HTTPError as e:
            print("FAILED HTTP GET " + url)
            time.sleep(min(max_backoff, backoff_coefficient ** i))


def segwit_wallet_balance(extended_public_key):
    """
    Given a segwit xpub (in either xpub or ypub format) at the account level, get the balance of the wallet by calling
    blockchain.info endpoints repeatedly.
    :param extended_public_key: The account extended public key.
    :return: The BTC balance as a Decimal.
    """
    # Convert from ypub if necessary
    xpub = extended_public_key
    if extended_public_key.find("ypub") == 0:
        xpub = ypub_to_xpub(extended_public_key)

    satoshi_balance = Decimal(0)
    for change in [True, False]:
        balance_for_node = Decimal(0)
        for chunk in chunked(GAP_LIMIT, lazy_generate_addresses(xpub, change, True)):
            chunk_data = http_get("https://blockchain.info/multiaddr?active=" + "|".join(chunk))['addresses']
            n_txns = sum([x['n_tx'] for x in chunk_data])

            # We're done once we hit a chunk with no transaction data
            if n_txns == 0:
                break
            else:
                balance_for_node += sum([Decimal(x['final_balance']) for x in chunk_data])
        print("Balance for {}: {} satoshis".format("change" if change else "receive", balance_for_node))
        satoshi_balance += balance_for_node
    print("Total balance: {} satoshis".format(satoshi_balance))
    return satoshi_balance.scaleb(-8)  # Get the bitcoin balance


def segwit_wallet_balance_hack(extended_public_key):
    """
    Given a segwit xpub (in either xpub or ypub format) at the account level, get the balance of the wallet by scraping
    data from the blockchain.info web interface.
    :param extended_public_key: The account extended public key.
    :return: The BTC balance as a Decimal.
    """
    # Convert from ypub if necessary
    xpub = extended_public_key
    if extended_public_key.find("ypub") == 0:
        xpub = ypub_to_xpub(extended_public_key)

    # You can do a nicer version of this with BeautifulSoup
    page = http_get("https://blockchain.info/xpub/" + xpub, as_json=False)
    balance = page.split('<td id="final_balance">')[1].split('BTC')[0].split('>')[-1]
    return Decimal(balance)


def legacy_wallet_balance(xpub):
    """
    Given a legacy xpub at the account level, get the balance of the wallet from the blockchain info api.
    :param xpub: The account extended public key.
    :return: The BTC balance as a Decimal.
    """
    return Decimal(http_get("https://blockchain.info/balance?active=" + xpub)[xpub]["final_balance"]).scaleb(-8)


print("legacy_wallet_balance(legacy xpub) => " + str(legacy_wallet_balance(xpub)))
print("")
print("segwit_wallet_balance_hack(segwit xpub) => " + str(segwit_wallet_balance_hack(xpub_segwit)))
print("")
print("segwit_wallet_balance(segwit xpub) => " + str(segwit_wallet_balance(xpub_segwit)))
The output should be
(venv) matthew@inspiron:$ python get_balance.py 
OK HTTP GET https://blockchain.info/balance?active=xpub6CUGRUonZSQ4TWtTMmzXdrXDtypWKiKrhko4egpiMZbpiaQL2jkwSB1icqYh2cfDfVxdx4df189oLKnC5fSwqPfgyP3hooxujYzAu3fDVmz
legacy_wallet_balance(legacy xpub) => 0.00408747

OK HTTP GET https://blockchain.info/xpub/xpub6CxXsMT2YRk1CjEPqYRRXxqXPoiVsvYz66sBnyD7rEbG4XJFkYd2FG9wP3KakpuBWC15u21zcCy3g2v6Vw2GQGAqKDxHFip3jBhskd42iE7
segwit_wallet_balance_hack(segwit xpub) => 0.00516644

OK HTTP GET https://blockchain.info/multiaddr?active=3CBAwo5jQggTSngqPv5vn5jBwX6nhepkPe|336SZTC3pvAGuHBhVkLpi1oLcogv1jKjUS|3EivUzBWqeZ2wYNFLdUt6TZqLyJuU4ZwKB|3A7yP51nyLJ5LijcFuqkHCYinSFyHRdhKY|3KmTMxkZL7Vc63PiTxHTqoAxAsBHxAQ1tb|3PMK2NbFw8fHYMcEWhW9zbPD5trcbP1cTb|338PA72RvcGrGZVgpELnMxfBKwECmCkNtD|3DmbuuFVsnCKxiGofTaHEne5qAb5DTX2rS|35sbZn2pcBH1HbnarqQERJNvXeGLjsb6n2|3KWBVptuSWCDDTR7pXh4y5xZpwgxy1GotY|3Qrq2KGoqCvC62RVUSdCGpQwWqH8Cxf7xS|36sLPe8AWKjBpJTUJRSJEMTQv8p97sXzS8|3L1PaY6yxwZxSrKZ2UBXuVzfwQGH931e3u|39mnJM4unbwPoc4MzaTxHAwrG96BVgzJcw|37SvvwdCiHxwbb5HZfPAhiPcZfdhhwPftT|34G8pfY9sKBy72ibNNwHn3Na6Q12mXdAMW|31rvn9sUf7SnBX7TBoiASkWFD9C89L2FhA|3HhTErMFrURP5K2cS7rcNLpjrefe5jjymC|3Gjti54yKWGJTjMVGMAeHWFKbo6Qnudbnt|37sJvECDCsfY7Zi8LafrTuLKcBgQAvSCDP
OK HTTP GET https://blockchain.info/multiaddr?active=3KPEVtRj3Avjcuc7SLz3Wa1jX9mLqhyBQd|3QqSuP174VoCXvJTYbsJn3t7vsuDgKNWGj|3MdPq1mWXwGW67YRyzcr6dnGKSyf4bC1Gd|36dNuwk9Z1CUW6kimLusMxpogkU7xkxBd9|38SSmNkMuyG7t2nmXfLdxvDqQwKYRmVsTf|31v74Ya9oWBCftxpVWhpCiDKZX7kndfANQ|3NUfcSaTMgQMZsLBit8ez7iDA3jHAGyTsE|3PH1d1EKxLddhBa67qD3KDZn6LMkuzjuTQ|3ASm7PMjvWHATmV36AWsyC6tWFaSYmVuTF|37rSDc3mCHhHQdBbLYkstzGDbs3GEokTRY|3HtBfRpy6vgAYDnYANqYzzzivcVqbha5BW|34MzRynnrBv9dQHE3anUmsTXoSjLBUD9Qj|36q39GMeAUTQfhXXS3cEREFUMNDkCWWZn9|32bZPNVQCboWxak7CYhzaSbdSRBsC2pswq|3MLbRyiCkmAHjDPgBMDmZs1bUCw8LTpaB8|3LjtrQYcKxxdpPSi8XM21eZUTe8CF5QGk4|33PhkXQdt6wevSWs4UJMwP2E1c98hkdBR4|37hiz9nhtQFovApxsjpVFnwit8Xyz9rXUz|37WUh3BPsQ1JgyxbH5R1cfP5ip5bQpQuXK|3BntzhusoRXjuambTcpeyQaDXw7ekS3WFf
Balance for change: 516644 satoshis
OK HTTP GET https://blockchain.info/multiaddr?active=3J1U4wD1Xfxtj2yQad43G2zvLVhDFsWTLW|3Q3zFEwCj3TGALmY4L21JNkxd4du6sMWmM|38LacH8qmhL1mfBGtdYRMMN4xLT477zqRU|3CXL6KbJ1jLCmcVTHy7cD34EhgifXpkwGo|3Db964zvDZjMgMt6KDt3coFhRUgxZPJXR8|32SFceKQmEDVRtTYgpqbJhExn1v3SYxMv4|3P1jbGjwQzhjgP8YndjLsooQvRR1QNLhVY|3Nb6yfJmz4mdgzAAxSPjZJcadN5vfC9kUJ|3Qxd13t5KwjPC3X55YiRkToRCF2G8gZA5V|33eeArpch3pzctiDbN9WiKPS2tRufrsvKt|3FdGnBfQDYWnfyCwPT3Wcf58GyViC6m5rM|3HfnQWht1FuaEpSdVLZpGZCauFsq7BvjqG|3EU9SibpQBLLEHYHJG3xN6AfFtsvaZmcxv|3LgieAiMWdod2PAQJymqSWymBHaHkwyxM7|3G6MDBcrEHNnRHCpY71teXDFCrL1C6pDXY|3AfogzjYe8gLmmY1CfV16PfumgicrFCVKu|36hMLyYvUM9f6YYorbJhmeRXD8u1EH7RJ5|3Ev4bGfcA6EN4c8pyQap9jzqAarViaB3nS|3FFfsBnmMs3TaPg2cQdMPxTxLhZAguQwzk|3GbcyVtLovEeAioFWwoBPyLtz6zsymZbZj
OK HTTP GET https://blockchain.info/multiaddr?active=3R1gT8DbxbTHGJkU39D4QZug2cUUmuuE7e|37pWv4fxzgaJj3h82rWyQCB5uQz6d4GoxM|35kTuvqfin5kesTEvnwwVWAK2KhPQg8aLw|31jxhq9rLCJUETms2tath9jLu2wZqSTmqs|33UFjPjUJbXPjRM57jQRxRXKrzSqnhWwky|3EdspnBdUfqrkJWfE13LgwRiPnXJfXJ6GR|3D1PSqyv6xNXqTvTYxQ6m82U5kXEcC7Zuj|3Man6Fq99zFnEQb7Z1GGR3RaCvoRpsW86y|39MjwbQichhR8phPpKJk8aeXDfPxe3hAvH|3LAkq4FZ6Tu5zdQEXti2pLckgVz9e8n4g1|31jxMrmvfGZ6tt9DK7zVxPpnE9nCGbdvnT|3KuTDdYbcuCABfFm6NrfkNo5Dr4RFma7FV|3LzYrgv3rmyNWvYKKBMUmUVMLTxDHg3uU9|3NB2DmM8uqr5ZnqX9WF5SWd729B4MuTuT9|34PMV22fXjvykaDRgh8yd8rsUAZemN8ug4|37LXsEcGJ9P4nn7U6aZSxtbTJC1JQYfJXu|3JdDCWpFbmme2sXeBR39nX5EqbGzi3wtY8|3BMtKAWr5fUHrTXdRJughjdWj8NdcYn9LA|3LVYy18APQP4PpHUrVa8T7y11Y7ZEz6dTW|3LwoWbFRCRUTEwi76pveWfyYbn3VeJSRdp
Balance for receive: 0 satoshis
Total balance: 516644 satoshis
segwit_wallet_balance(segwit xpub) => 0.00516644