Skip to content

Instantly share code, notes, and snippets.

@jalehman
Last active Sep 28, 2020
Embed
What would you like to do?
Full Node Provider & Bitcoin Wallet Bounty

Proposed Architecture

What follows is an architecture that satisfies the Project Requirements outlined in the Bitcoin Full Node Provider and Wallet bounty

The recommended way to interface with Bitcoin is through a full node, so this architecture involves the implementation of an interface between an Urbit ship and a full node. Any ship that connects to a full node in this way can provide this service to other ships that do not run their own full node, making the providing ship a full node provider (referred henceforth as just provider). Access via providers to the bitcoin network requires trusting the provider with only derived public addresses, meaning that little identifying information ever needs to be shared with the provider.

This approach has the advantage of catering to Bitcoin maximalists that want to run their own full nodes, but also to the security and privacy-conscious Bitcoin user that does not have the resources to administer their own full node. The amount of trust given to a provider is small, and the high trust within the Urbit network makes this even more palatable.

The proposed architecture is broken into two high-level components:

  • Bitcoin Full Node Provider: Provides full node access to its host ship, and possibly to other ships with a scheme determined by the host.
  • Bitcoin Wallet: Initiates payment requests and manages known addresses (ours and others').

This proposal is probably incomplete. The worker should consider this a good starting point for beginning the implementation, but know that modifications can and should be made as the project develops.

Bitcoin Full Node Provider

The full node provider allows Urbit ships to communicate with a Bitcoin full node. Any ship operator that runs a full node and connects their ship to it will be referred to as a host. A host can become a provider by providing access to the full node via gall agent(s) to other ships. At minimum, the host provides access to itself (a host is a trivial provider), but should be able to provide access to other ships as well.

The full node provider should satisfy the following user stories:

  • As a ship, I can connect to a full node that I control, becoming a host.
  • As a ship, I can connect to a full node via another ship, becoming a client.
  • As a host, I can control which other ships can access my full node.
  • As a host or client, I can check the balance of my Bitcoin addresses.
  • As a host or client, I can broadcast a signed transaction to the Bitcoin network.
  • As a host or client, I can obtain a list of transactions for a specified Bitcoin address.

bitcoin-provider

The bitcoin provider connects to the full node and uses its JSON RPC API to communicate. Its role is to talk to the bitcoin network directly, therefore all of its communication should be restricted to the host ship.

Sample API

  • connect: connect to a full node at a specific IP & port along with required authentication or other connection settings.
  • status: retrieve the connection status of the bitcoin node.
  • create-wallet: create a watch-only wallet for the provided address.
  • balance: look up the balance for the provided address.
  • transactions: produce all transactions for the provided address.
  • broadcast-transaction: broadcast a signed transaction to the bitcoin network.

bitcoin-provider-hook

The bitcoin provider hook handles the connection to the full node provider in one of two capacities:

  1. %local: the full node runs on our ship, or
  2. %remote: the full node runs on another ship.

Its role is to manage the connection between ship(s) and full node. It will define and enforce access controls, perform initial and ongoing setup when full node connection statuses change, and map ships to bitcoin addresses.

If a ship has a %local connection to a full node, the hook will communicate directly with the bitcoin-provider on that ship. If the connection is %remote on the other hand, communication will instead be proxied to the remote bitcoin-provider-hook, whereby they will be resolved on the local bitcoin-provider and then relayed back to the point of origination.

Sample API

  • connect: given a ship to connect to, attempt to establish a connection to its provider store.
  • set-access: determines which ships can interact with the hook. Can only be set to a value other than %host if the connection type is %local.
    • %host (default): only the host
    • %kids: only kids of this ship (e.g. moons, ships sponsored)
    • %ships: only specified ships
    • %all: any ship
  • access-level: current level of access
    • responds with null if the requesting ship is not within current level of access or is not running a node.
  • add-addresses: create watch-only wallets for the source ship and provided bitcoin addresses.
  • balance: look up the balance for the source ship or specific address.
    • %ship: produce the balance of each address known for source ship. Known addresses are those that watch-only wallets have been created for.
    • %address: given an address, produce the balance.
  • transactions: produce all transactions for the provided address.
  • broadcast-transaction: broadcast a signed transaction to the bitcoin network.

Summary

The bitcoin-provider is strictly an interface to a locally-connected full node. You can think of it as a wrapper around the RPC API that provides a domain-specific API to the Bitcoin network.

The bitcoin-provider-hook is how ships communicate with with a bitcoin-provider, whether locally or remotely. The hook serves primarily to manage access to a bitcoin node and field requests from foreign ships to communicate with that node.

That these two concerns be separated into different agents is less important than having the semantics around local/remote connections and permissions preserved.

Bitcoin Wallet

The wallet's responsibilities include facilitating payment between two parties and managing the addresses required to do so.

When receiving Bitcoin from another ship, a new address should be generated for just that transaction. This prevents anyone other than the owner from knowing the receiver's total balance, and also provides some beneficial accounting properties as well. See the BIP32 proposal for more detail on the motivation behind this mechanism. All that's needed to derive a new address is an extended public key (or xpub for short). An extended public key can come from any pre-existing Bitcoin wallet, e.g. a Hardware wallet.

The xpub key corresponds to a single Bitcoin “account” within the wallet, and should only be stored on its owner’s ship. The owning ship will use its xpub to derive subsequent addresses for transactions, revealing only derived addresses to the outside world. Given the set of derived addresses we can determine the overall account balance and transaction history:

  • The balance of an account is the sum of balances across all addresses derived from that account.
  • The transactions of an account are the concatenated list of transactions on each address derived from that account, sorted by timestamp.

When sending Bitcoin to another ship, the wallet should be able to use the address-level balances as inputs to a single transaction. For example, if account A has the following addresses and balances:

  • Ax: 0.45BTC
  • Ay: 0.2BTC
  • Az: 0.01BTC

Then an attempt to send 0.6 BTC to account B should use the balance of Ax and Ay as inputs to the transaction.

To initiate payment (sending or receiving), a transaction is constructed that includes the addresses of the sender and recipient, amount to send in satoshis, and transaction fee parameters--this transaction is then signed using the corresponding private key and broadcast to the Bitcoin network via the bitcoin-provider-hook.

NOTE: The Bitcoin Core implementation provides facilities for constructing transactions and performing coin selection. Ideally Bitcoin Core should be leveraged wherever possible, but the architecture of this system may not allow us to leverage Bitcoin Core for all of these functions.

The design of this wallet system closely resembles the signing-only wallet described in the Bitcoin docs. The signing-only wallet corresponds to a hardware wallet or master ticket, while the networked wallet is what is being outlined here.

The wallet should satisfy the following user stories:

  • As a ship, I can see my overall balance for an account.
  • As a ship, I can see all transactions for a specific account.
  • As a ship, I can send bitcoin to other ships, becoming a payer.
  • As a payer, I can tell when my payment has succeeded.
  • As a ship, I can receive bitcoin from other ships, becoming a payee.
  • As a payee, I can provide unique address for each transaction.
  • As a payee, I can tell when I have received payment.

bitcoin-wallet-store

The wallet store is responsible for holding extended public key(s) and keeping track of addresses that have been derived from xpubs, both ours and others'. A mapping of ship to address will be required to match transaction histories against Urbit ships, of which only the parties involved in a transaction need to be aware of.

Another possible use-case of the wallet store is as a cache of balance and transaction data. The bitcoin network should be queried for updated information regularly (a way for the bitcoin provider to publish this information to the appropriate ship(s) would be ideal), but the responses can be saved locally for quick presentation while awaiting updated information from the Bitcoin network.

Sample API

  • create-account: register a new extended public key from which to derive addresses from. Other metadata can be provided, like a name, that can be used to identify this account in the future.
    • The wallet can be implemented such that only one xpub can be saved at a time, or so that more can be stored. Storing multiple xpubs has implications for the API that will need to be thought through.
  • derive-address: given an account, derive a new address for their transaction.
  • addresses: produce all addresses derived from an account.
  • accounts/account: produce all known accounts.
  • construct-transaction: given an account, destination address and amount (and transaction fee inputs), determine the appropriate address(es) to use to send payment to the destination address.

bitcoin-wallet-hook

The wallet hook combines the bitcoin-provider-hook and bitcoin-wallet-store.

When a foreign ship wishes to pay our ship, it will poke the bitcoin-wallet-hook with the request, at which point a new address is derived using the bitcoin-wallet-store and saved in application state along with relevant metadata. Then, the hook will create a watch-only wallet using the bitcoin-provider-hook to enable efficient fetching of transaction history and balance information.

Sample API

  • request-address: request a new address to be derived (remote)
    • in the case of multiple known accounts, one will have to be chosen as a "default"
  • derive-address: derive a new address given a specific account (local)
@matildepark
Copy link

matildepark commented Sep 9, 2020

You will have the support of the Urbit Foundation in the development of this project, and will work closely with its director to refine requirements and test the resulting system.

I love this phrasing, even if it sounds sort of job-like, because it makes the UF sound like an agency or an institute.

I don't know the details of Bitcoin implementations but I think the spec here is pretty good? — If only because you have user stories, and I have yet to see those in a bounty.

@jalehman
Copy link
Author

jalehman commented Sep 14, 2020

Thanks! Doubled down on the user stories since I agree that those are good.

@dpc
Copy link

dpc commented Sep 25, 2020

The way to go is for some ship to host https://github.com/lightninglabs/neutrino server API , and then full client side light wallet node. Better privacy and all other properties.

The ships that do voluntarily provide that server API would be just bridges between urbit networking and Bitcoin fullnode.

I'm also quite confident that such bridges don't even have to run a fullnode themselves. They could just connect to a normal remote Bitcoin fullnode(s) via http and act as a proxy.

@timlucmiptev
Copy link

timlucmiptev commented Sep 26, 2020

dpc, can you expand a bit on how/whether that would help with deriving addresses? The only un-straightforward thing I see in this proposal currently is how to derive addresses from an xpub within Hoon, since we won't want to send the xpub to the node-hosting server.

Ideally even having a light client wouldn't be necessary: you'd just use the full nodes as dumb proxies and do all address derivation and management natively inside each Urbit ship's wallet.

@dpc
Copy link

dpc commented Sep 27, 2020

The design here leaks which addresses belong to me to other ships (providers).

Neutrino works by light-wallet (client) following blockchain but receiving only a "compact block filter header" instead of the whole block. This allows the light-wallet to check if any transaction is affecting addresses that belong to them, and only then requesting the whole block from the provider (server).

Then the light-wallet scans the whole block and looks for transactions that belong to them. This way provider never knows which addresses belong to each client, and since "compact block filter header" is tiny and blocks affecting a particular user infrequent, keeps the bandwidth and storage requirements very small.

@timlucmiptev
Copy link

timlucmiptev commented Sep 27, 2020

That makes sense. The primary difficulty is that adding dependencies to the client ship ("hey, also go follow this setup guide to make a Neutrino node for yourself") totally destroys the ease-of-use/integration part of the value proposition.

Would it make the most sense to make a local light client a pluggable option?

@timlucmiptev
Copy link

timlucmiptev commented Sep 27, 2020

I think there also may be alternate privacy-preserving solutions if we

  1. isolate the exact address leaking scenarios
  2. route messages to the provider through multiple ships in the provider's approved set of providees for plausible deniability

@dpc
Copy link

dpc commented Sep 27, 2020

https://bitcoinops.org/en/topics/compact-block-filters/

One would not make a neutrino node themselves. Just a handful of volountiers would expose neutriono-like API, and other users would query them like a Neutrino server.

Built-in onion routing in Urbit would be great for many projects BTW. Neutrino still can't address broadcasting privacy IIUC.

@timlucmiptev
Copy link

timlucmiptev commented Sep 28, 2020

Yeah, re onion routing: there is this whole class of P2P solutions that work really well in an Urbit context. My only hesitation is the well-known issues with rolling your own security anything, but I think as long as large amounts of BTC weren't sent through wallets initially, a strong solution could be iterated to.

A Neutrino API is a good idea and I think early in the feature list of "things to add right after stuff works."

Tbh the most annoying initial part is just re-implementing address derivation from an xpub, since that shouldn't be sent to providers OR neutrino nodes. But I've looked through the zuse.hoon crypto primitives and there seems to be everything I'd need to implement BIP32+44/49/84.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment