Backward Compatibility

backcomp.md

I'm designing the non-interactive transaction feature plus stealth address feature on node. First of all, we must agree that a Hard Fork will be there to make these features take effect from a block height. And then, an important "backward compatibility" topic comes here!

Gernally speaking, we can take advantage of the HF(Hard Fork) to break most of backward compatibility, simplify a lot of version handling issues, such as nodes compatibility (old version nodes will be splitted from the new version nodes network), wallets compatibility (old version wallets can not work with new version node), etc. Any backward compatibility need special handling on the version conversion which need extra efforts and complexity. Let's have a detailed list here:

node backward compatibility

1. Suppose the HF node version will be v6.0, and the old versions are v4.x and v5.x.
2. Suppose the HF will start from block height 768,000 or about 1 May 2021. (quite draft number here, to be changed.)
3. Nodes with v6.0 will build a new network, and all old versions (v4.x and v5.x) without upgrading will be kicked out of this new network. (This is also the Hard Fork meaning.)
4. The fresh new nodes with v6.0 launched from scratch will join the new network only.
5. The nodes upgraded from old version (v4.x or v5.x) are able to migrate local database into new version automatically.
6. The block header since height 768,000 will take a new header version (version 3) for this HF. Block header data structure will not be changed in this HF.
7. p2p protocol version 4_000 will be used for this HF. All connection requests from lower p2p protocol versions (1_000, etc.) will be declined.

node API backward compatibility

1. v1 APIs (REST) deprecated and removed from version 5.x?.
2. v2 APIs (Json-RPC) deprecated and removed from version 6.0.
3. Only v3 APIs (Json-RPC) available in nodes with version 6.0.

wallet backward compatibility

1. Suppose the HF wallet version will be v6.0 for mwc-wallet,mwc713 and mwc-qt-wallet.
2. The old version wallets are v3.x for mwc-wallet and mwc713, v1.x for mwc-qt-wallet.
3. The HF wallets v6.0 will not work with old version nodes (with v4.x and v5.x). Warning and exit if connected to old version nodes.
4. The old version wallets (defined above at item 2) will not work with a new version node (v6.0 and above). Warning and exit if connected to a node with version v6.0 and above.

• Note: This behavior need a minor PR to integrate into current wallets release as earlier as possible, to have most of wallets upgraded before that HF date.
• The old version wallets without said PR will not warn and exit when working with new version node, but it will not work at all, both the scaning and sending/receiving doesn't work.

send/receive between old version wallets and v6.0 HF wallets

1. Sending from old version wallets to v6.0 HF wallets will not be received. More explain: Because the old version wallets only works with old version nodes, even this sending works (normally it does not), the final transaction will be posted only into the old version nodes, and the transaction will never reach the new HF network, the receiver will never receive the coins. The coins 'sent' will not be lost, because they are still there as unspent coins in the new HF network.
2. In any case, for receiving, it must stick to wallet confirmation before considering it as received.
3. Sending from v6.0 HF wallets to old version wallets maybe works, or maybe not, but the receiver has to upgrade his/her wallet to see these received coins.
4. A successful sending in v6.0 HF wallets will not lose the coins, in any case the receiver is 100% able to collect the said received coins with his/her wallet seed secret.

wallet API backward compatibility

mwc-wallet APIs

1. ? (TBC)

mwc713 APIs

1. ? (TBC)

Does before HF block 768,000 Nodes v6 will be able to talk with nodes V5.X and older?
Am I understand correctly that node p2p version will be bump to 4000 at the HF block. Before HF block v6 nodes will support current p2p protocols versions 1,2 & 3. Switch to 4000 will be done at the block 768,000?

I think node API can be deprecated earlier, we can release a new wallet with a new node. That will not be a problem. In case of QT wallet people will not see that if they are using the standard setup. Miners will need to upgrade both node and a mwc-wallet.
mwc-wallet will need to be able to read data prom prev version in any case.

In order for new wallets and old wallet to send/receive funds, we need to keep the same slate format and v6 & v5 nodes need to be able to talk with each other. node API compatibility is not needed.

About mwc713 API. We are good with that because mwc713 doesn't have it's own API, it is using mwc-wallet code. You can think about mwc713 as a layer that provide some special functionality for qt wallet. mwc713 retranslate calls to mwc-wallet core.

mwc-wallet API we better to keep as compatible as possible. People did automation and we don't want to break it much.

Thanks @bayk for your comments.

Does before HF block 768,000 Nodes v6 will be able to talk with nodes V5.X and older?

Yes, must be. Otherwise, the HF release can't be deployed before that date.

Am I understand correctly that node p2p version will be bump to 4000 at the HF block. Before HF block v6 nodes will support current p2p protocols versions 1,2 & 3. Switch to 4000 will be done at the block 768,000?

I read the code and find the current p2p protocol version is 1000 in mwc. I will confirm that by a network capture.
There should be no version 1/2/3 in mwc nodes any more, since we had a Hard Fork before, at height 202,500.

It will not be a switching for p2p protocol version. It could work like this:

• v6.0 nodes use same p2p protocol version as v5.x/v4.x to communicate with all old nodes.
• v6.0 nodes use new p2p protocol version to communicate with all other v6.0 nodes.
• At HF height 768,000, v6.0 nodes will kick out all connected old nodes, and refuse connection from old p2p protocol.

One open-point here to be checked: whether the current nodes accept connections from higher protocol version.

In order for new wallets and old wallet to send/receive funds, we need to keep the same slate format and v6 & v5 nodes need to be able to talk with each other. node API compatibility is not needed.

I don't understand here. Here I would like to define the behavior that sending/receiving between new/old wallets will NOT success. After a HF, all wallets must be upgraded, otherwise, they can not send/receive coins.

mwc-wallet API we better to keep as compatible as possible. People did automation and we don't want to break it much.

Let me think this. Perhaps the best way is not touch these old APIs, instead, new APIs for for new features. I will think this later, after the nodes.

current p2p protocol version is 1000 in mwc

That comes from the last rebase. We can put 3 there and that will work as well. Here is a doc about that https://github.com/mimblewimble/docs/wiki/P2P-Protocol#phasing-out-old-peers
Testing shows that old nodes able to sync with a new ones, so nodes really supporting old versions.

I don't understand here. Here I would like to define the behavior that sending/receiving between new/old wallets will NOT success. After a HF, all wallets must be upgraded, otherwise, they can not send/receive coins.

This comment is about the case before HF block. If wallets use slates, why we can't keep them compatible until the HF block? After FH block it will stop working because the that will be two different networks, so receiver will never see confirmation and it is expected. If v6 wallet will not be able to interact with the old ones even before HF, users will never start migrating. My point that first user who does migration to a new wallet first, still should to be able to use exchanges or send wallets to other users.

okay, about mwc protocol version, a little bit investigation here:

• 2 for v3.1.1: https://github.com/mwcproject/mwc-node/blob/v3.1.1/core/src/global.rs#L44
• 2 for v4.0.0: https://github.com/mwcproject/mwc-node/blob/v4.0.0/core/src/global.rs#L45
• 2 for v4.0.2: https://github.com/mwcproject/mwc-node/blob/v4.0.2/core/src/global.rs#L45
• 1000 for v4.1.0: https://github.com/mwcproject/mwc-node/blob/v4.1.0/core/src/global.rs#L48
  So, 1000 has been released on 15th Sep.
  Check a running node and it shows that protocol version 2 and 1000 works well, no problem.

http://127.0.0.1:3413/v1/version

{
  "node_version": "4.1.0",
  "block_header_version": 2
}

http://127.0.0.1:3413/v1/peers/connected

"user_agent": "MW/MWC 3.0.0",
"user_agent": "MW/MWC 3.1.1",
 "user_agent": "MW/MWC 3.1.2",
 "user_agent": "MW/MWC 3.1.4",
"user_agent": "MW/MWC 3.2.3",
"user_agent": "MW/MWC 4.0.0-beta.3",
"user_agent": "MW/MWC 4.1.0",
 "user_agent": "MW/MWC 5.0.0",  (note: a strange number here, not official!)

This comment is about the case before HF block. If wallets use slates, why we can't keep them compatible until the HF block? ...

👍 understand now and thanks @bayk for the explain :-)
Yes. Before HF block height, we have to ensure v6 wallet is compatible with old wallet, to get a smooth migration.
Keep same slates format is a good idea for that.
And I would suggest to give a limited backward compatibility here, to avoid too much version handling workload and complexity. To be detailed when coming to wallet design part.

Sounds good. Thank you.
Also, currently NRD kernels are not enabled, to enable them the hard fork is required. So with NIT they will be activated as well.

MW/MWC 5.0.0

It is rebased version. We will release it pretty soon. Sorry, version doesn't have beta.

People did automation and we don't want to break it much.

I assume this Comment regarding Automation is mostly directed at exchanges and external Integrators.
But let me elaborate on the API Documentation automation anyways;

API Docs should be created fine and reflect changes regardless of what is changed "in the API".

For a new API Version, we would need a new Trait similar to OwnerRpcS compared to OwnerRpc, then this will be picked up as well.
If there would be a new API Version, an according "this is deprecated" note should be added in the code so it is picked up by Cargo, similar to the current V2 API Note.
An example of such a "Note" for documentation can be found here https://github.com/mwcproject/mwc-wallet/blob/40f8986cb50fdc48ed8640c187bd8658659ca510/api/src/owner_rpc.rs#L75

If the API-docs website (https://mwcproject.github.io/) should ever point to a newer API Version we can simply redirect to the newer Version by changing the following line to point to the new Trait Name in https://github.com/mwcproject/mwc-wallet/blob/master/.ci/pushdocs.yml
echo '<meta http-equiv=refresh content=0;url=grin_wallet_api/trait.OwnerRpc.html>' > /home/vsts/work/1/gitpages/index.html && \
Currently, we still point to V2 so this is not relevant for creating the Docs, it's more of a "beauty thing".

TL;DR:
API doc automation will be fine.
If a new API Version is created (new trait) everything should be picked up, but in that case we need to make sure the old API is marked as deprecated. (if it is fully deprecated)

Ofc impacts to other automation/integration should be kept at a minimum either way ☕

Thanks @MyCryptoT for your comments 👍 We will also take care of these.