The last few weeks I've been prototyping an alternative to polkadot-js/api. My goal is to create a more modular, user-friendly, strongly-typed and light-weight version of polkadot-js/api.
I'm still not sure whether I will be able to accomplish those goals. However, I certanly hope that I will be able to share the knowledge and ideas that I gather during this attempt. That's why I'm creating this gist.
I'm not a technical writer, and I don't have a lot of time to spend on writing this. So, this is meant to be a "brain dump". although, hopefully someone who is good at writting documentation can use this for improving the substrate docs site.
When I started prototyping, I was able to make a lot of progress just by digging into the substrate docs: I was able to created a SCALE codec library, a client that's able to read storage values and make basic RPC calls, etc.
Eventually, I reached a point where I wanted to be able to execute transactions and then I realized that the substrate docs don't have any "advanced" material for creating Exstrinsics. Therefore, I started to dig into the code-base of polkadot-js and to debug an application to figure out how to create a transaction... And let me warn you that it is a lot more complicated than I initially anticipated. That's why I've decided to document my findings, in case tha these can be useful to others.
Warning: This is not intended to be a "canonical" guide on how to create Extrinsics, this is just sharing my chaotic thoughts, and findings, which are centanly incomplete.
One of the first things that I did was to create a client with the WsProvider, so that I was able to easily analyze what's happening over the wire when a transaction get's created. I then realized that there are many messages going back and forth before the client finally sends the message with the submission of the transaction.
Having a quick look at calls that preced the submission message, some of them really seem redundant, and to this day I still think that there must be some redundant calls... And also other calls that I would expect to happen earlier or in parallel, but we will dig into that later.
Let's discuss what happens when we use polkadot-js for using Alice's test account to submit a create_comment call to Adz, while being connected via wss://adz-rpc.parity.io.
These are all the messages that went back and forth (in chronological order) from the moment I submitted the transaction, until the submission was sent:
- Sent:
{"id":147,"jsonrpc":"2.0","method":"chain_getHeader","params":[]}{"id":148,"jsonrpc":"2.0","method":"chain_getFinalizedHead","params":[]}- Received:
{"jsonrpc":"2.0","result":{"digest":{"logs":["0x06617572612067dc200800000000","0x0561757261010160931b90a965057320c19792c82e86c4a92dee1fa82241cea29f04ce40fb8f24c5cb6bf1d348ee8740a58bedc98992e2f3d9537dd429b77cb558c9ee583b6986"]},"extrinsicsRoot":"0xe6a13a7b27868f47d6f00699fba30f963617f63ddba49ae027c56f7be8641cca","number":"0x33ed7","parentHash":"0x0600f4884346a3d895e4456c266d759b75a43722f391a11d53b055ec50f02191","stateRoot":"0x1774fb8227a2867561791c26b4d4227db9e39e80f7ac9adadb9726a1c6e6036c"},"id":147}{"jsonrpc":"2.0","result":"0x0600f4884346a3d895e4456c266d759b75a43722f391a11d53b055ec50f02191","id":148}- Sent
{"id":149,"jsonrpc":"2.0","method":"chain_getHeader","params":["0x0600f4884346a3d895e4456c266d759b75a43722f391a11d53b055ec50f02191"]}- Received:
{"jsonrpc":"2.0","result":{"digest":{"logs":["0x06617572612065dc200800000000","0x05617572610101603622bb52fce8a48651d7411984bedc9d96af40228588d5ecc851a4265c824796e7b499623aa26cf62f32f960ec9a02d6675e27e1aa4969571e8da4afa67485"]},"extrinsicsRoot":"0xe6ffb6c8f318d09b41ccece5f676165c2cc8cd327669d37814744c2558ded379","number":"0x33ed6","parentHash":"0x9d325df53f5a4e3ec241a7897c8cadda48e86e50d1f4d9bb7b50ea9e32c4fead","stateRoot":"0xa76d5eb142cda175292d7440b45d96df83584f2c8f305b2bbbfe13590ae7aa47"},"id":149}- Sent
{"id":150,"jsonrpc":"2.0","method":"state_getRuntimeVersion","params":["0x9d325df53f5a4e3ec241a7897c8cadda48e86e50d1f4d9bb7b50ea9e32c4fead"]}- Received:
{"jsonrpc":"2.0","result":{"apis":[["0xdf6acb689907609b",3],["0x37e397fc7c91f5e4",1],["0x40fe3ad401f8959a",5],["0xd2bc9897eed08f15",3],["0xf78b278be53f454c",2],["0xab3c0572291feb8b",1],["0xdd718d5cc53262d4",1],["0xea93e3f16f3d6962",1],["0xbc9d89904f5b923f",1],["0x37c8bb1350a9a2a8",1]],"authoringVersion":1,"implName":"template-parachain","implVersion":0,"specName":"template-parachain","specVersion":1,"transactionVersion":1},"id":150}- Sent
{"id":151,"jsonrpc":"2.0","method":"chain_getHeader","params":["0x0600f4884346a3d895e4456c266d759b75a43722f391a11d53b055ec50f02191"]}- Received
{"jsonrpc":"2.0","result":{"digest":{"logs":["0x06617572612065dc200800000000","0x05617572610101603622bb52fce8a48651d7411984bedc9d96af40228588d5ecc851a4265c824796e7b499623aa26cf62f32f960ec9a02d6675e27e1aa4969571e8da4afa67485"]},"extrinsicsRoot":"0xe6ffb6c8f318d09b41ccece5f676165c2cc8cd327669d37814744c2558ded379","number":"0x33ed6","parentHash":"0x9d325df53f5a4e3ec241a7897c8cadda48e86e50d1f4d9bb7b50ea9e32c4fead","stateRoot":"0xa76d5eb142cda175292d7440b45d96df83584f2c8f305b2bbbfe13590ae7aa47"},"id":151}- Sent (SUBMISSION MESSAGE)
{"id":152,"jsonrpc":"2.0","method":"author_submitExtrinsic","params":["0x11028400d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d01aea7dbdc0e90544b84b2c5ea08d914db239bb3d2a87ad5d6eab4a732293f1676d3ae4879785486947bf4b79c76e982e97d0ec4515c33aa19195cdae8e3b44d826401b9010036045454657374696e672045787472696e736963732e2e2e22000000"]}- Received
{"jsonrpc":"2.0","result":"0x3ab881762c5149726e8ca8e24a6c34b9556a429e0ab315e821e2d6299cf80cb8","id":152}We will talk more the messages that prelude the submission later. For now, though, let's focus on the "submission" message.
As we can see, the submission message is encoded, and I would expect it to contain some signed fields and some unsigned fields... It would be very useful to be able to decode that message to see what's in it. So, after a lot of debugging, and some trial and error, I was able to decode the message using @unstoppablejs/scale-codec, like this:
import {
Struct,
U8,
U32,
Tuple,
Compat,
Enum,
Str,
Bytes,
U16,
} from "@unstoppablejs/scale-codec";
import { AccountId } from "./AccountId";
const NoIdea = Bytes(32);
const Signature = Struct({
signer: Enum({ sr25519: AccountId, foo: NoIdea, bar: NoIdea }),
signed: Enum({
ed25519: Bytes(64),
sr25519: Bytes(64),
secp256k1: Bytes(64)
}),
era: U16,
nonce: Compat,
tip: Compat,
});
const Method = Struct({
callIndex: Tuple(U8, U8),
args: Tuple(Str, U32),
});
const Extrinsic = Struct({
len: Compat,
version: U8,
signature: Signature,
method: Method,
});
const [, decodeExtrinsic] = Extrinsic
console.log(
decodeExtrinsic(
"0x11028400d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d01aea7dbdc0e90544b84b2c5ea08d914db239bb3d2a87ad5d6eab4a732293f1676d3ae4879785486947bf4b79c76e982e97d0ec4515c33aa19195cdae8e3b44d826401b9010036045454657374696e672045787472696e736963732e2e2e22000000"
).value
)Which displays this in the console:
{
len: 132,
version: 132,
signature: {
signer: {
tag: 'sr25519',
value: '5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY'
},
signed: { tag: 'sr25519', value: [ArrayBuffer] },
era: 356,
nonce: 110,
tip: 0
},
method: { callIndex: [ 54, 4 ], args: [ 'Testing Extrinsics...', 34 ] }
}The first thing that you may have noticed is that the Method codec looks like this:
const Method = Struct({
callIndex: Tuple(U8, U8),
args: Tuple(Str, U32),
}); The callIndex is always a Tuple(U8, U8) (we will later explain where to get those numbers from). However, the args tuple will be different depending on the transaction. In our case, we need a string (the "comment") and a number (the id of the Ad) to call the create_comment call of Adz.
Ok, now that we know what's inside the message, let's discuss how we can generate a message like that.
This field is very straight forward: is a Compat encoded number that represents the length of the remaining bytes of the message (after reading it).
Which in this example its value is also 132. Once you know how to get it, this field is super straight forward... figuring out how the value gets there wasn't fun, though. Anyways, it turns out that's pretty much a hardcoded value. Basically the way it got there is as follows:
- The latest extrinsic version is hardcoded in this line of code.
- Then the
versiongetter of theExtrinsicclass does this. - Which in combination with these constants.
Produces
132for signed extrinsics and4for unsigned extrinsics (which I won't cover in this gist).
This field is an Enum and I still don't know what all its options are. The only thing that I know for sure is that
when the public key (Address) is a sr25519 key (like Alice's test account is), then it goes in the first position of the Enum.
I suspect that the other Enum options must be reserved for other kinds of public keys, but I still haven't figured that out, yet.
This is also an Enum which contains the signed payload (that we will cover later). I know a little bit more about this one, this is what I've found: It seems that the first position of the Enum indicates that the payload is signed using ed25519 (aka naclSign), the second position is for when the payload is signed using sr25519 (aka schnorrkelSign) and the 3rd position is for both ecdsa and ethereum signatures, which are both based on secp256k1Sign (while the former uses blake2 and the latter uses keccak).
I find a bit confusing the fact that we have different 2 enums: one for the public key and the other one for the signed payload... That's proabably because I must be missing something about the Signer Enum... However, if the Signer Enum is what I think that it is, then IMO it would make more sense to have one enum that indicates the signing method and then inside that Enum we could have both the public key and the signed payload... But again, that's probably because I must be missing some context on what the Signer Enum is about.
Later we will see what's inside the signed payload and we will discuss how we can get that information and how to sign it.
I had a very hard time getting to understand what this field was about. As it turns out this is a U16 field that's computed from 2 different values:
- Another number that we will refer to as
mortalLen - A number that we will refer to as
signingHeaderNumber
Once we have those 2 values we can calcualte the Era with the following function:
const calcEra = (
signingHeaderNumber: number,
mortalLen: number
) => {
let calPeriod = Math.pow(2, Math.ceil(Math.log2(mortalLen)));
calPeriod = Math.min(Math.max(calPeriod, 4), 1 << 16);
const phase = signingHeaderNumber % calPeriod;
const quantizeFactor = Math.max(calPeriod >> 12, 1);
const quantizedPhase = (phase / quantizeFactor) * quantizeFactor;
const trailingZeros = calPeriod.toString(2).split("").reverse().indexOf("1");
return (
Math.min(15, Math.max(1, trailingZeros - 1)) +
((quantizedPhase / quantizeFactor) << 4)
);
}; In order to calculate the mortalLen we will first have to retrieve the metadata by making an RPC call to state_getMetadata. Later we will explain how to decode its payload, for now what's important is that we will need to retrieve the following constants from the metadata payload: System.BlockHashCount, Babe.ExpectedBlockTime and Timestamp.MinimumPeriod, which may or may not be present. Once we have those constants we are ready to calculate the mortalLen:
const FALLBACK_MAX_HASH_COUNT = 250;
const FALLBACK_PERIOD = 6_000;
const MAX_FINALITY_LAG = 5;
const MORTAL_PERIOD = 5 * 60 * 1000;
function getMortalLen(blockHashCount, expectedBlockTime, minimumPeriod) {
return Math.min(
blockHashCount ?? FALLBACK_MAX_HASH_COUNT,
MORTAL_PERIOD /
(expectedBlockTime ?? minimumPeriod === undefined
? FALLBACK_PERIOD
: minimumPeriod * 2) +
MAX_FINALITY_LAG
);
}); This is the number property of the signingHeader (which will be used for another field that's in the signed payload). Bellow you can find a way to obtain the signingHeader using the current (and precarious) version of @unstoppablejs/client:
interface HeaderResponse {
digest: {
logs: Array<string>;
};
extrinsicsRoot: string;
number: string;
parentHash: string;
stateRoot: string;
}
type ParsedHeaderResponse = Omit<HeaderResponse, "number"> & {
number: bigint;
};
const MAX_FINALITY_LAG = 5;
const StateRpc = Rpc("state", client);
const ChainRpc = Rpc("chain", client);
const getHeader = ChainRpc<
[hash?: string],
HeaderResponse,
ParsedHeaderResponse
>("getHeader", (input: HeaderResponse) => ({
...input,
number: BigInt(input.number),
}));
const getFinalized = ChainRpc<string>("getFinalizedHead");
export const getSigningHeader = async () => {
const aborter = new AbortController();
const finalizedPromise = getFinalized(aborter.signal).then((h) =>
getHeader(h, aborter.signal)
);
const current = await getHeader().then((currentHeader) => {
if (!currentHeader.parentHash) {
aborter.abort();
return currentHeader;
}
return getHeader(currentHeader.parentHash);
});
if (aborter.signal.aborted) return current
const finalized = await finalizedPromise;
return current.number - finalized.number > MAX_FINALITY_LAG
? current
: finalized
}The nonce of the account sending the transaction... basically, this:
import { BlakeTwo128Concat, EncodedArgs, Storage } from "@unstoppablejs/client";
import { Compat, Struct, U32 } from "@unstoppablejs/scale-codec";
import { AccountId } from "./AccountId";
import { client } from "./client";
const SystemStorage = Storage("System", client);
const AccountInfo = Struct({
nonce: U32,
consumers: U32,
providers: U32,
data: Struct({
free: Compat,
reserved: Compat,
miscFrozen: Compat,
feeFrozen: Compat,
}),
});
const args: EncodedArgs<[accountId: string]> = [BlakeTwo128Concat(AccountId)];
const Account = SystemStorage("Account", AccountInfo, ...args);
export const getNonce = (address: string) =>
Account.get(address).then(
(x) => x?.nonce ?? 0
); Depends on how generous you feel that day :-), usually 0.
The corresponding index that's found on the payload of the metadata call. I will expand on this later when I discuss the metadata.
A tuple with the expected arguments, encoded according to what's described on the metadata.
Now lets discuss what's inside that signed payload. After some live debugging, I was able to console.log the payload before it gets signed, and again after some digging around into the code and debugging I realized that the taxonomy of the payload is encoded like this:
const Signature = Struct({
method: Method,
era: U16,
nonce: Compat,
tip: Compat,
specVersion: U32,
genesisHash: Bytes(32),
blockHash: Bytes(32),
}); We've already explained the first 4 fields, so let's focus on the new ones:
It comes from an RPC call to state_getRunTimeVersion, in the response to that call there is a field named specVersion, so that's it.
It also comes from another RPC call, this time to: chain_getBlockHash while passing 0 as the parameter.
Remember that before when we were discussing how to calculate the Era we talked about the signingHeader? Well, this is the hash of that header... Which we will have to compute ourselves.
TODO: explain how to has the header.
So, once we have all those fields we should SCALE encode them, and then we will have to sign those bytes using the corresponding signing method (usually sr25519Sign), but ofc we should allow the holder of the key to pick the correct signing method. Once signed, we then check whether the byte length is greater than 256, if that's the case we will hash those bytes using blake2b with a dkLen of 32, otherwise we won't do anything else to them.
Now that we have all the necessary fields we are ready to encode the whole Extrinsic, except for the len, ofc. So, after we have encoded the Extrinsic, then we check its byte length and then we Compat encode that value and we prepend those bytes to the previous ones.
TODO: explain it
Explain why I think that there are bogus calls before submitting the Extrinsic, and why some calls that look sequential should probably run in parallel instead.