Skip to content

Instantly share code, notes, and snippets.

@jcnelson
Last active February 2, 2023 05:09
Show Gist options
  • Save jcnelson/c982e52075337ba75e00b79942164e31 to your computer and use it in GitHub Desktop.
Save jcnelson/c982e52075337ba75e00b79942164e31 to your computer and use it in GitHub Desktop.
Appchains

Revisions

  • Oct 19 2021: deployed MVP v2 of the appchain mining contract and updated the links and examples in this document to use it instead.

Introduction

Blockchains don't scale. The fact that all nodes process all transactions means that the blockchain only goes as fast as the slowest node allowed on the network. If that's something like a Raspberry Pi, then that's as fast as the blockchain goes.

And that's okay! The upside is that the more people can run nodes, the more resilient the blockchain will be. It's much harder to break a 10,000-node blockchain where most nodes run on home computers all across the world than a 10,000-node blockchain where most nodes run in a few datacenters.

But the demand for cheap transactions isn't going anywhere. What's a distributed systems hacker to do?

A Composite Blockchain

A single blockchain can't handle unbound transaction volumes, but many blockchains can. Just like how the world's most expensive high-powered mainframe cannot match the capacity of a cloud made out of cheap servers, a single high-powered blockchain cannot match the combined capacity of a linked network of many small blockchains. This is especially true if adding new blockchains to the network is a permissionless act, since this would let the network grow with user demand.

Enter appchains. The PoX consensus algorithm in Stacks lets it use the hashpower of Bitcoin to securely host its chain structure. With a couple modifications, it also allows on instance of the Stacks blockchain to use another instance of the Stacks blockchain to host its chain structure as well! PoX generalizes: many, many L1 blockchains may run at varying degress of separation between themselves and Bitcoin, but Bitcoin will commit to the chain structures of all of them.

So, now the world looks like this at each Bitcoin block:

                                                                              Degree-2 Appchain blocks
                                               Degree-1 Appchain blocks      /.---------------.
                                               /.----------------.          / |               |
Bitcoin blocks           Stacks blocks        / |====== tx ===== | <-------{  |               |
.--------------.      /.--------------.      /  |                |          \ |               |
|              |     / |              |     /   |                |           \`---------------`
|==== tx ======| <--{  |===== tx =====| <---\---`----------------`                    ^
|              |     \ |===== tx =====| <-----------/---.------------------.          |
`--------------`      \`--------------`             \   |                  |          |
       ^                      ^                      \  |                  |          |
       |                      |                       \ |                  |          |
       |                      |                        \`------------------`          |
       |                      |                                  ^                    |
.--------------.       .--------------.                          |                    |
|              |       |              |                          |                    |
|              |       |              |                          |                    |
     . . .                   . . .                             . . .                . . .

Each time a Bitcoin block is mined, it causes a Stacks block to be mined. Now, each time a Stacks block is mined, it causes one or more appchain blocks to get mined, which in turn can trigger their own subsequent appchain blocks to be mined.

The appchain approach makes Stacks a composite blockchain. You can add more capacity to the Stacks blockchain by adding more appchains. Instantiating an appchain is just a matter of deploying a specially-crafted smart contract that can store the relevant mining state. Just as with Stacks and Bitcoin, each appchain's blocks are hashed to a single transaction in the host chain, so to host chain nodes, mining an appchain just looks like doing a particular contract-call? transaction.

App chains are classified by degress of separation from Bitcoin. This is the number of host chains between the chain and Bitcoin. For Stacks, this is 1 -- Bitcoin is its host chain. For appchains mining on Stacks, this is 2 -- appchain state is separated from Bitcoin by two blockchains. For appchains running on those appchains, this is 3, and so on and so forth. However, because each host chain contains the hashes of its client chains' blocks, it still ends up being the case that Bitcoin commits to the state of all appchains, everywhere.

The reason this scales is because the host chain doesn't need to know or care about the state of its client chains (just like how Bitcoin doesn't know or care about Stacks). Only the people who want to use your smart contracts need to run nodes for your appchain; everyone else can ignore it entirely.

When to use Appchains

Appchains are one of several possible scalability solutions for Stacks being developed, but they are some trade-offs. If you're a developer, you'd use an appchain if all of the following are true:

  • Your app has its own token that is distinct from STX.
  • The smart contracts for your app don't really interact much with smart contracts for other apps.
  • Users need lots of block capacity to use your smart contracts, and/or need to send lots of transactions.

If these are all true, then it might make sense for you to start your own appchain. Your appchain will be instantiated with your smart contracts built-in, and it will have its own distinct token separate from STX. Your appchain will have its own miners, who will send transactions on the host chain to mine blocks in your appchain and in doing so, earn a block reward in your appchain's token.

Because appchains are simply instances of Stacks, they come with all the features Stacks currently has. In particular:

  • Your appchain comes with its own PoX, its own cost-voting, and its own BNS pre-installed.
  • Your users can Stack your appchain's token and earn a yield in STX (or whatever token powers the host chain).
  • You can choose your own PoX parameters and block capacity limits.

In addition, appchains have the following new features enabled by mining on Stacks as a host chain:

  • You're free to make any changes to the appchain consensus rules without affecting Stacks (or whatever host chain).
  • You can implement your own rules on who gets to mine and under what circumstances via the mining contract on the host chain. If you can write them in Clarity, you can make them apply to your appchain.

Limitations

Because appchains are independently mined from their host chains, the security budget for resisting a reorg is dependent on how much miners of that appchain are willing to spend to keep the chain safe. This budget is going to be less than the budget for Stacks, which in turn is less than the budget for Bitcoin. However, the set of blocks that exist in your appchain across all its forks (even non-canonical ones) is secured by the host chain's security budget. So even if your appchain gets reorged, it would be temporary, since the old canonical fork is still visible in the host chain.

App chain P2P messages and transactions are made non-replayable on other networks by a 4-byte chain ID field. However, there is currently no convention or means of forcing all appchains to have globally-unique chain IDs. Someone could clone your appchain and use its chain ID, thereby making your chain's transactions replayable on their network. This will need to be solved later by a dedicated appchain registry smart contract, which will first need to be spec'ed out and ratified via the SIP process. We're still in the very early days!

I don't know off the top of my head if the distinct chain ID makes it impossible to use a hardware wallet at this time. I'd imagine that adding support would be trivial, though.

Setting Up an Appchain

The code for this is currently in a feature branch on the Stacks blockchain. You can build a copy yourself as follows:

$ git clone https://github.com/blockstack/stacks-blockchain appchains
$ cd appchains
$ git checkout feat/app-chain-mvp
$ cd testnet/stacks-node
$ cargo build --release

First, you'll need to deploy a Stacks mainnet or testnet node from this feature branch (depending on where you want to run your appchain). I highly recommend using only testnet for now. This step is required because the new Stacks node software includes RPC endpoints that appchains need, such as querying Stacks headers and data var state.

After you have that running, you will need to create a mining contract for your appchain. The mining contract has a few required define-data-var and define-data-map definitions that will be needed for the appchain to query it and boot up, in addition to having a few required define-public functions. I've reproduced the one I'm testing with here, with source comments. Feel free to adapt to your needs. Note in particular that the mining contract has a way to list the bootstrap nodes for your appchain.

Once the contract is deployed, you'll need to spin up a boot node for your appchain at the IP and port listed in the mining contract. To do so, you'll first need to figure out the contract's address, as well as the appchain's genesis hash.

Let's say the contract address is ST17ABWV76GQCGWKFQR4G5N23HDXGZ3D3869A8Z5N.appchains-mvp-v2. You will want to add that to your config file as follows:

[burnchain]
chain = "stacks"
mining_contract = "ST17ABWV76GQCGWKFQR4G5N23HDXGZ3D3869A8Z5N.appchains-mvp-v2"
# since the address is a testnet address, you'll also want this:
mode = "testnet"
# Fill this in with the corechain Stacks node you deployed earlier
peer_host = "44.199.104.134"
rpc_port = 30443
peer_port = 30444

To get the genesis hash, you run the following:

$ stacks-node appchain-genesis --config /path/to/your/config.toml
Appchain genesis state instantiated in /tmp/appchain-genesis-1632538390
Appchain genesis hash: a32a3ff4b42a3d87d771396ca5ffab850afbc1376927ad2789726a43c881fe72

You'll put that hash into your config as follows:

[burnchain]
genesis_hash = "a32a3ff4b42a3d87d771396ca5ffab850afbc1376927ad2789726a43c881fe72"

So, your [burnchain] section should now have four fields:

  • chain = "stacks"
  • mining_contract = "ST17ABWV76GQCGWKFQR4G5N23HDXGZ3D3869A8Z5N.appchains-mvp-v2"
  • mode = "testnet"
  • genesis_hash = "a32a3ff4b42a3d87d771396ca5ffab850afbc1376927ad2789726a43c881fe72"

Now you're ready to go. Simply start up your appchain boot node as follows:

$ stacks-node appchain --config /path/to/your/config.toml

As long as you tell your other users and miners the mining_contract and genesis_hash values, they can go and do likewise and join your appchain network.

Adding Bootcode

One thing you can do with appchains that you can't do with Stacks is add additional boot code. This is code that will live under the boot address ST000000000000000000002AMW42H on testnet, along with .pox, .bns, and .costs. If you want to make sure your custom smart contracts are available the minute your chain boots up, you can ship them as boot code.

Shipping your appchain's desired smart contracts as boot code is advantageous for a few reasons:

  • Shipping code as boot code ensures that it's always available for the duration of the chain's existence. No one can reorg it away.
  • There's no block limit on the boot code, so you can run any expensive pre-calculations you want in a boot code smart contract. This includes things like importing data from another chain, or pre-populating a set of accounts.

To deploy your own boot code, you'll need to do the following three things:

  • Add the contract names to the list of boot code contracts in the mining contract, within the appchain-config data var's boot-code list. In the linked example above, this includes "hello-world".

  • Generate the genesis hash with the boot code. You can do this by passing in the --boot-code ClI argument. You can pass it multiple times for multiple contracts. They must have the same filenames as the contract names

$ stacks-node appchain-genesis --config /path/to/your/config.toml --boot-code ./hello-world.clar
  • Launch the boot node with the boot code, so other appchain nodes can go and download it from you. You do this by also passing the source files in with --boot-code:
$ stacks-node appchain --config /path/to/your/config.toml --boot-code ./hello-world.clar

Other users do not need to pass --boot-code; only the appchain creator and people who cold-boot their nodes will need to do this (which will basically be just you, the developer). However, other users will of course need both the contract address and the genesis hash, which you can provide them out-of-band.

One day, there will be a registry for appchains so developers can both reserve chain IDs and publish their mining contract addresses and genesis hashes under a human-readable identifier. Maybe there could be a BNS namespace for it. I haven't thought that far ahead.

Running a Node

I have deployed a sample appchain: on the Stacks testnet that you can play around with. Here is a sample config you can use to connect to it:

[node]
rpc_bind = "0.0.0.0:15301"
p2p_bind = "0.0.0.0:15300"
seed = "<FIXME: fill in with your private key>"
miner = false
# you may want to change this
working_dir = "/var/stacks/appchain-follower/chainstate"
mine_microblocks = true
microblock_frequency = 1000
wait_time_for_microblocks = 5000
deny_nodes = ""

[burnchain]
mining_contract = "ST17ABWV76GQCGWKFQR4G5N23HDXGZ3D3869A8Z5N.appchains-mvp-v2"
genesis_hash = "a32a3ff4b42a3d87d771396ca5ffab850afbc1376927ad2789726a43c881fe72"
chain = "stacks"
mode = "xenon"
peer_host = "44.199.104.134"
rpc_port = 30443
peer_port = 30444
poll_time_secs = 10

[connection_options]
connect_timeout = 5
handshake_timeout = 5

# if you want to mine...
[miner]
min_tx_fee = 600
first_attempt_time_ms = 1000
subsequent_attempt_time_ms = 1000

To run the node, simply execute:

$ stacks-node appchain --config /path/to/the/above/config.toml

Your node should boot up and remain in sync with the appchain described in this contract. Do note that you must use a stacks-node built from the feat/app-chain-mvp feature branch.

Sending Appchain Transactions

If you have some appchain tokens, then sending transactions is basically the same as sending them to the Stacks network. The only difference is that you must supply the appchain's chain ID to the transaction. In this example, this is 0x80000002, or 2147483650. You can get the chain ID from the mining contract, in the appchain-config data var.

To interat with the example appchain above, you would pass --chain_id 2147483650 to blockstack-cli to generate transactions. For example:

$ blockstack-cli --testnet --chain_id 2147483650 publish b8d99fd45da58038d630d9855d3ca2466e8e0f89d3894c4724f0efc9ff4b51f001 515 0 kv-store ./kv-store.clar

Note that in addition to supplying a chain ID, you must also supply --testnet. This is because the appchain itself is the testnet variant of chain ID 2147483650 (there could also be a mainnet chain 2147483650).

Future Work

I consider appchains to be in an alpha state. They work -- you could even ship products with them! -- but there are known limitations in the ecosystem that I'd like to get done before we can declare them a fully-supported feature.

  • There needs to be a SIP that standardizes the mining contract's structure. This includes providing a trait definition for mining, as well as standardizing the data types for the appchain-version, appchain-config, and appchain state.

  • There needs to be an authoritative registry contract for appchains on the Stacks corechains, whereby developers can claim their appchains' chain IDs and specify their mining contract addresses and genesis hashes. There also needs to be a way to expire and re-use chain IDs for defunct appchains. This could be BNS, but I'm not married to the idea if that proves difficult for whatever reason. This registry should be standardized via the SIP process.

  • The appchain codebase needs to use the appchain registry contract to resolve appchain names to their mining contracts and genesis hashes, so the act of spinning up an appchain only requires the user to know the human-readable identifier of the appchain, instead of its more inscrutible (and spoofable) metadata.

  • Appchains will likely want different token emission schedules, and even different rules for when to consider transactions and blocks valid. I think this could be addressed by a special "consensus" smart contract in the boot code that hooks into the block and transaction validation code paths. Basically, it would adhere to a trait that contained public functions for the following:

    • At a given block height, how many tokens need to be emitted?
    • Given a transaction and read-only access to the chain state, is the transaction valid?
    • Given a block and read-only access to the chain state on top of which it is built, is the block valid?

    The reason appchain developers would want the ability to decide these things is to constrain what they want the appchain to be used for. Appchains don't need to have the exact same consensus rules as Stacks -- they can be more restrictive. For example, an appchain that is only meant to process transactions for a particular smart contract could declare all SmartContract transactions invalid, so no one else could create a smart contract on that appchain. As another example, an appchain might require miners to hold host chain tokens as collateral in the mining contract, and execute a "peg-out" to transfer wrapped host chain tokens from the appchain back to the host chain each time they mine a block. This consensus smart contract could verify that this happens as part of deciding whether or not a block is valid.

  • Appchain tokens are not STX. The Clarity VM should allow STX-specific Clarity functions to be given different names by the appchain to reflect the true name of the appchain token.

  • The stacks.js library needs to permit the application to specify the chain ID.

  • There needs to be wallet support for appchains, including Ledger at some point. The web wallet needs to learn the chain ID(s) of the per-application appchain(s) in use. I have not looked into how difficult this will be.

  • There needs to be a fully fleshed-out light client for Stacks, which would permit the user to boot up an appchain without having to boot up instances of all of its ancestor chains. This is blocked on feat/1805, which will ship in Stacks 2.1.

@Deliver88
Copy link

Deliver88 commented Dec 18, 2021

Hello, it is really fantastic to see the progress of the concept of appchains - thanks a lot for that!

I am trying to follow the steps here to get an appchain up and running. I am new to this field and therefore forgive for the many questions. As I understood, the following steps are necessary (where I added my open points to them):

  1. Run a testnet stacks-node from the app-chain-mvp branch - the follower or miner one? To which bootstrap_node? to that one "047435c194e9b01b3d7f7a2802d6684a3af68d05bbf4ec8f17021980d777691f1d51651f7f1d566532c804da506c117bbf79ad62eea81213ba58f8808b4d9504ad@testnet.stacks.co:20444" ?

  2. Deploying the appchain smart contract as you mentioned. Where to deploy that, is the following call debug$: ./blockstack-cli publish pub_key 0 0 appchain ./appchain.clar --testnet | xxd -r -p > tx1.bin right?. To mempoll too?, as follows debug$: curl -X POST -H "Content-Type: application/octet-stream" --data-binary @./tx1.bin http://127.0.0.1:20443/v2/transactions (with error Failed to load Stacks chain tip)

  3. We have to run now an appchain-stacks-node, right? Is this node a second one besides that one from step 1) (differ in burnchain
    Stacks instead of bitcoint)? If so, do we have to do follows: release$: ./stacks-node start --config /path/to/your/config.toml ? To bootstrap_node from stacks would be fine, right?

Thanks a lot

@Deliver88
Copy link

Hi @jcnelson , I have now a stacks-node with local bitcoind up and running , with the following config:

[node]
working_dir = "../../chain-state/testnet/sqlite"
rpc_bind = "0.0.0.0:20443"
p2p_bind = "0.0.0.0:20444"
seed = "3519899623ffaa438d18e5cfa4a905cc2a6226126450362554beebb2d86e708801"
local_peer_seed = "3519899623ffaa438d18e5cfa4a905cc2a6226126450362554beebb2d86e708801"
miner = true
bootstrap_node = "047435c194e9b01b3d7f7a2802d6684a3af68d05bbf4ec8f17021980d777691f1d51651f7f1d566532c804da506c117bbf79ad62eea81213ba58f8808b4d9504ad@testnet.stacks.co:20444"
wait_time_for_microblocks = 10000

[burnchain]
chain = "bitcoin"
peer_host = "192.168.178.27"
username = "bitcoin"
password = "bitcoin"
rpc_port = 18332
peer_port = 18332

[[ustx_balance]]
address = "STB44HYPYAT2BB2QE513NSP81HTMYWBJP02HPGK6"
amount = 10000000000000000

[[ustx_balance]]
address = "ST11NJTTKGVT6D1HY4NJRVQWMQM7TVAR091EJ8P2Y"
amount = 10000000000000000

[[ustx_balance]]
address = "ST1HB1T8WRNBYB0Y3T7WXZS38NKKPTBR3EG9EPJKR"
amount = 10000000000000000

[[ustx_balance]]
address = "STRYYQQ9M8KAF4NS7WNZQYY59X93XEKR31JP64CP"
amount = 10000000000000000

After that, I started a second stacks-node in parallel to be sync. with the app-chain, the config is the follows:

[node]
rpc_bind = "0.0.0.0:15301"
p2p_bind = "0.0.0.0:15300"
seed = "340006e013f186e2c7d4a9cde8e89efd2ddb44d6a72630fe413dd4193c1df6b401"
miner = true
# you may want to change this
working_dir = "../../chain-state/appchain-demo/data"
mine_microblocks = true
microblock_frequency = 1000
wait_time_for_microblocks = 5000
deny_nodes = ""

[burnchain]
mining_contract = "ST17ABWV76GQCGWKFQR4G5N23HDXGZ3D3869A8Z5N.appchains-mvp-v2"
genesis_hash = "a32a3ff4b42a3d87d771396ca5ffab850afbc1376927ad2789726a43c881fe72"
chain = "stacks"
# remove it, otherwise we are getting: INFO [1640096354.080841] [testnet/stacks-node/src/main.rs:145] [main] Loading config at path ../../testnet/stacks-node/conf/testnet-appchain-demo-follower-conf.toml Process abort due to thread panic: panicked at 'Bitcoin network unsupported', testnet/stacks-node/src/burnchains/bitcoin_regtest_controller.rs:223:14
#mode = "xenon"
peer_host = "44.199.104.134"
rpc_port = 30443
peer_port = 30444
poll_time_secs = 10

[connection_options]
connect_timeout = 5
handshake_timeout = 5

# if you want to mine...
[miner]
min_tx_fee = 600
first_attempt_time_ms = 1000
subsequent_attempt_time_ms = 1000

After that, I create the following kv-store.clar contract located at the folder debug:


(define-map store { key: (string-ascii 32) } { value: (string-ascii 32) })

(define-public (get-value (key (string-ascii 32)))
    (match (map-get? store { key: key })
        entry (ok (get value entry))
        (err 0)))

(define-public (set-value (key (string-ascii 32)) (value (string-ascii 32)))
    (begin
        (map-set store { key: key } { value: value })
        (ok true)))

which have been published as follows:

debug$: ./blockstack-cli --testnet --chain_id 2147483650 publish 340006e013f186e2c7d4a9cde8e89efd2ddb44d6a72630fe413dd4193c1df6b401 515 0 kv-store ./kv-store.clar | xxd -r -p > tx1-appchain.bin

This setup provide us a stacks-node connected and synchronized with the burnchain via bitcoind, a stacks-node connected and synchronized with the appchain deployed here right?

Now I am trying to make a transaction ends with an error:

debug$: curl -X POST -H "Content-Type: application/octet-stream" --data-binary @./tx1-appchain.bin http://localhost:15301/v2/transactions
{"error":"transaction rejected","reason":"SignatureValidation","reason_data":{"message":"Invalid tx 93484b97a1a779e1a3d64f6b4555e483bd50bdd24a1bef7de04a77e3c3da929b: invalid chain ID 2147483650 (expected 2147483648)"},"txid":"93484b97a1a779e1a3d64f6b4555e483bd50bdd24a1b

What is wrong here with the chain-id?

Thanks a lot.

@Deliver88
Copy link

FYI, /v2/info shows:

{
   "burn_block_height" : 8175,
   "exit_at_block_height" : null,
   "genesis_chainstate_hash" : "74237aa39aa50a83de11a4f53e9d3bb7d43461d1de9873f402e5453ae60bc59b",
   "network_id" : 2147483648,
   "parent_network_id" : 3669344250,
   "peer_version" : 4207599105,
   "pox_consensus" : "0f9e4a4ea3d7d8da375a369cf8c1e0fa2b240cbe",
   "server_version" : "stacks-node 0.0.1 (feat/app-chain-mvp:f3a8e4676, release build, macos [x86_64])",
   "stable_burn_block_height" : 8174,
   "stable_pox_consensus" : "f7f30a9b8fda9b9bddd1d22aad6a86ab9c7f4cee",
   "stacks_tip" : "92c2a9417443a140bb5f37cc02917fcb861f5a0d5117755cd31abba3b786058f",
   "stacks_tip_consensus_hash" : "5c3581cecbf5a54a517fa7cd37fa374d9ba3534b",
   "stacks_tip_height" : 5972,
   "unanchored_seq" : 0,
   "unanchored_tip" : "0000000000000000000000000000000000000000000000000000000000000000"
}

The network ID here is 2147483648, i.e. it is not running on the appchain deployed here with the config above, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment