2024-04-19 Devnet restart instructions

README.md

Summary

For a very long time the devnet snapshot was slightly corrupted in a way that wouldn't cause an issue until crossing the epoch boundary between 678 and 679. In order to get the cluster running again, devnet nodes will need to patch the corrupted snapshot themselves or download a new one and then restart with updated validator args detailed below.

Step 1: Create a snapshot at slot 293327999

(Optional) Download new snapshot instead of creating it

Since it's devnet, it's not as critical to ensure ledger integrity (but it is good practice!). If you so wish, you can download the ledger from one of the following nodes

• singapore (dv1): 145.40.71.85
• europe (dv2): 136.144.49.15
• dallas (dv3): 147.75.55.147
• nyc (dv4): 147.75.105.51

wget --backups=1 --trust-server-names http://$host/snapshot.tar.bz2

If you do this, the only validator argument you need to update is --expected-shred-version:

--expected-shred-version 45127

Step 1.1: Patch ledger tool

In order to create a valid snapshot, you will need to temporarily apply the following patch to ledger tool and build from source. Ensure you're starting from the tagged v1.18.11 release.

diff --git a/runtime/src/serde_snapshot/newer.rs b/runtime/src/serde_snapshot/newer.rs
index 004e6e61d..d40fd5543 100644
--- a/runtime/src/serde_snapshot/newer.rs
+++ b/runtime/src/serde_snapshot/newer.rs
@@ -331,6 +331,12 @@ impl<'a> TypeContext<'a> for Context {
     {
         let mut bank_fields: BankFieldsToDeserialize =
             deserialize_from::<_, DeserializableVersionedBank>(&mut stream)?.into();
+
+        // remove bad entries
+        bank_fields
+            .epoch_stakes
+            .retain(|_, epoch_stakes| epoch_stakes.total_stake() > 150_000_000_000_000_000);
+
         let accounts_db_fields = Self::deserialize_accounts_db_fields(stream)?;
         // Process extra fields
         let lamports_per_signature = ignore_eof_error(deserialize_from(&mut stream))?;

Step 1.2: Create snapshot with patched ledger tool

Note that if you don't need to share your created snapshot with anyone, it's sufficient to create an incremental snapshot here with the --incremental flag.

$ [solana|agave]-ledger-tool --ledger <ledger path> create-snapshot 293327999 <ledger path> --hard-fork 293327999

You should see the following output:

Successfully created snapshot for slot 293327999, hash ActNnRRqNroHnVj8dMqSApp16EoEsBV3ZSDTsDCq4ws9: <snapshot dir>/snapshot-293327999-3XKbyNW3HDNBw2G7fa7u5xCYxPmXxvVtwHAfwcyyTVoL.tar.zst
Shred version: 45127

Check your ledger directory to ensure that you have no snapshot newer than snapshot-293327999-[...].tar.zst Newer snapshots should be removed. Snapshots older than 293327999-[...].tar.zst don't need to be removed.

Step 2: Restart your validator with updated arguments

--wait-for-supermajority 293327999
--expected-shred-version 45127
--expected-bank-hash ActNnRRqNroHnVj8dMqSApp16EoEsBV3ZSDTsDCq4ws9

NOTE: Once the cluster restarts and normal operation resumes, remember to remove --wait-for-supermajority and --expected-bank-hash before the next update or restart. They are only required for the restart.

The --expected-shred-version contains an "en dash" which will break arg-parsing

sorry about that @helius-kurt, thanks for reporting!

Any other IP addresses that can be tried for snapshot download? All 4 mentioned above do not help

You can try https://api.devnet.solana.com/snapshot.tar.bz2 as well

Thank you

Hi @jstarry

If we download the latest snapshot following the instructions at step 1, what values should be provided for the below flags/parameters in step 2?

--wait-for-supermajority ?
--expected-shred-version ?
--expected-bank-hash ?

FYI, I used the below command to download the snapshot:

wget --backups=1 --trust-server-names https://api.devnet.solana.com/snapshot.tar.bz2

@saravadeanil you only need --expected-shred-version 45127 now

@saravadeanil you only need --expected-shred-version 45127 now

Thanks for your reply! @jstarry

So, I should download the snapshot file under the ledger/snapshot/ directory and set the --expected-shred-version to 45127 in my solana-validator command?

Yup that sounds right

Yup that sounds right

Thanks a lot, my node is syncing with latest block now 🎉