Skip to content

Instantly share code, notes, and snippets.

@CrazySerGo
Last active August 10, 2022 20:55
Show Gist options
  • Save CrazySerGo/874e15f025d46f5edec0bc28f7199e12 to your computer and use it in GitHub Desktop.
Save CrazySerGo/874e15f025d46f5edec0bc28f7199e12 to your computer and use it in GitHub Desktop.
Solana Testnet Restart - 10 August 2022

IMPORTANT: CLUSTER HAS RESTARTED

THESE INSTRUCTIONS ARE NOW OBSOLETE

Please use your common service file if network already started.

INSTALL THE LATEST SOLANA RELEASE AND ENSURE YOU HAVE THE UPDATED EXPECTED SHRED VERSION:

--expected-shred-version 24371 \

IMPORTANT: Instruction is valid only during restart.

Testnet 1.11.5 Restart

(If you are running Solana v1.11.4 or earlier, please upgrade to current recommended version 1.11.5 https://github.com/solana-labs/solana/releases/tag/v1.11.5)

The restart will start from the highest optimistic slot.

$ cat solana.log  | grep 'optimistic_slot slot=' | tail -n1000 | cut -d "=" -f2 | tr -d i | sort -n | tail -n1
144871251

Note: If your last optimistic slot is lower than that the one listed below, this is likely your node crashed before it was able to observe the latest supermajority. Follow the instructions in the appendix

The known validator arguments provided below are validators who have the last optimistic slot and are providing snapshots for it, don’t keep them in your list as known validators after the restart as these are community validators not Solana Foundation nodes.

Important: DO NOT delete your ledger directory

Step 0: Stop the validator service

IMPORTANT: ⚠️MAKE SURE TO STOP YOUR VALIDATOR BEFORE DOING ANYTHING ELSE.⚠️

IMPORTANT: ⚠️DO NOT delete your ledger directory.⚠️

NOTE: Upgrade to v1.11.5 if you are running older Solana version

Step 1: Create a snapshot at slot 144871251

This document assumes your ledger directory is called ledger/. If not then adjust the following commands accordingly.

Use the ledger tool to create a new snapshot at slot 144871251, replacing the two instances of to your actual ledger path:

$ solana-ledger-tool --ledger <ledger path> create-snapshot 144871251 <ledger path> --hard-fork 144871251

Or else, if you use separate directory for snapshots:

$ solana-ledger-tool --ledger <ledger path> create-snapshot 144871251 <snapshots path> --hard-fork 144871251 --snapshot-archive-path <snapshots path>

Add --snapshot-archive-path before “create-snapshot” if you have a separate snapshot dir, and --accounts after “create-snapshot” if you have a separate accounts dir

The final line of output should be “Shred version: 24371”, and this snapshot file should now exist: ledger/snapshot-144871251-AfonVppYHoYwVXNLRSLZjjw5fV9xNxyAazazQZkFzuDX.tar.zst

Check your ledger/ directory to ensure that you have no snapshot newer than ledger/snapshot-144871251-..... This is very unlikely, but if found should be removed. Snapshots older than ledger/snapshot-144871251-.. should not be removed.

NOTE: If you receive “Error: Slot 144871251 is not available”, please see appendix

Step 2: Adjust your validator command-line arguments, temporarily for this restart to include:

--wait-for-supermajority 144871251 \
--no-snapshot-fetch \
--no-genesis-fetch \
--expected-bank-hash 4NstanApNPjCAd2HwBhHokqCQbJfCAYgp92VvJibSM5M \
--expected-shred-version 24371 \
--entrypoint 5.9.35.85:8001 \
--entrypoint entrypoint.testnet.solana.com:8001 \
--entrypoint entrypoint2.testnet.solana.com:8001 \
--entrypoint entrypoint3.testnet.solana.com:8001 \
--known-validator eoKpUABi59aT4rR9HGS3LcMecfut9x7zJyodWWP43YQ \
--known-validator 4ZtE2XX6oQThPpdjwKXVMphTTZctbWwYxmcCV6xR11RT \
--known-validator 8SRKNfvMerfA1BdU79CAwU4wNfjnDvFrBo3o5f5TS4uv \
--known-validator 9v7E6oEm1V86hjTubtBon7cRYPvQriWZKHZEX6j92Po4 \
--known-validator 3viEMMqkPRBiAKXB3Y7yH5GbzqtRn3NmnLPi8JsZmLQw \
--known-validator 3iPu9xQ3mCFmqME9ZajuZbFHjwagAxhgfTxnc4pWbEBC \

(Remove the previous value of “--expected-shred-version“ if present)

Once the cluster restarts and normal operation resumes, remember to remove these arguments. They are only required for the restart.

Step 3: Start your validator

As it boots, it will load the snapshot for slot 144871251 and wait for 80% of the stake to come online before producing/validating new blocks.

To confirm your restarted validator is correctly waiting for 80% stake, look for this periodic log message to confirm it is waiting: INFO solana_core::validator] Waiting for 80% of activated stake at slot 144871251to be in gossip...

And if you have RPC enabled, ask it repeated for the current slot:

$ solana --url http://127.0.0.1:8899 slot

Any number other than 144871251 means you did not complete the steps correctly.

Appendix: Resolution if you did not preserve your ledger or your last optimistic slot is below 144871251

NOT RECOMMENDED - this resolution should only be attempted if your ledger/ directory is unavailable.

If your ledger history is corrupt or otherwise unavailable, clean up your ledger/ directory instead of Steps 1 & 2, then complete Steps 3.

OBS: remove --no-snapshot-fetch if set

Add these arguments to restart:

--wait-for-supermajority 144871251 \
--expected-shred-version 24371 \
--entrypoint 5.9.35.85:8001 \
--entrypoint entrypoint.testnet.solana.com:8001 \
--entrypoint entrypoint2.testnet.solana.com:8001 \
--entrypoint entrypoint3.testnet.solana.com:8001 \
--expected-bank-hash 4NstanApNPjCAd2HwBhHokqCQbJfCAYgp92VvJibSM5M \
--known-validator eoKpUABi59aT4rR9HGS3LcMecfut9x7zJyodWWP43YQ \
--known-validator 4ZtE2XX6oQThPpdjwKXVMphTTZctbWwYxmcCV6xR11RT \
--known-validator 8SRKNfvMerfA1BdU79CAwU4wNfjnDvFrBo3o5f5TS4uv \
--known-validator 9v7E6oEm1V86hjTubtBon7cRYPvQriWZKHZEX6j92Po4 \
--known-validator 3viEMMqkPRBiAKXB3Y7yH5GbzqtRn3NmnLPi8JsZmLQw \
--known-validator 3iPu9xQ3mCFmqME9ZajuZbFHjwagAxhgfTxnc4pWbEBC \
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment