Skip to content

Instantly share code, notes, and snippets.

@alexwykoff
Last active July 23, 2022 05:16
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save alexwykoff/11f52421eb664df7a086f957ed602db2 to your computer and use it in GitHub Desktop.
Save alexwykoff/11f52421eb664df7a086f957ed602db2 to your computer and use it in GitHub Desktop.
A listing of Solana network events and a correction on a misinformed Twitter post

List of Solana outages, degraded performances, and their causes

Preamble

The intent on this document is not to cast Solana in a negative light. The author had a recollection corrected by Anatoly and lacking any singular listing via a simple search, decided to organize this document in good faith.

While there have been outages due to software issues and degraded performances due to considerable traffic, as of the time of this document's creation, Solana has never had an outage due to an AWS zone being down.

Corrections, critiques, and amendments are welcome.

2021

September - [Outage] - Hard fork due to attack on Grape Protocol IDO https://solana.com/news/9-14-network-outage-initial-overview

December - [Degraded Performance] - Network under intense load due to an NFT project's success. https://twitter.com/solchicksnft/status/1470557844336361472

2022

January - [Degraded Performance] - Transactions were mis metered. There were transactions which used a lot of compute cycles. Later in the same month, many transactions were created to attempt to capture liquidations which also degraded performance. https://twitter.com/SolanaStatus/status/1479126136953053187 https://twitter.com/laine_sa_/status/1484793907221311488

March - [Degraded Performance] - Several RPC nodes were down. The nodes were a subversion behind and when a new feature set was activated, they forked. https://status.solana.com/incidents/6qvg6z1k43zb

April - [Outage] - Stalled consensus. This was due to a flood of bot transactions attempting to mint an NFT from Candy Machine. There were not enough votes to finalize blocks and this caused many forks which could not be managed. https://solana.com/news/04-30-22-solana-mainnet-beta-outage-report-mitigation https://status.solana.com/incidents/ggym7s4qyzsq https://cointelegraph.com/news/solana-suffers-7th-outage-in-2022-as-bots-invade-the-network ** It should be noted that the URL slug is incorrect and the article was since corrected. **

June - [Outage] - Validators failed to reach consensus. The durable nonce feature caused part of the network to consider the block invalid. https://status.solana.com/incidents/m6qzbgc7np9b https://twitter.com/aeyakovenko/status/1532076517739532288

Source of confusion which led to the original incorrect reply in thread

During the September 2021 outage, there was concern over the concentration of validators in a few cloud providers. An example of the concerns of the time is reflected in this article. https://coinrivet.com/it/amazon-hosts-37-of-actively-staked-sol-could-this-be-a-solana-kill-switch/

FTA-

"Therefore if the Amazon-02 server goes offline – for maintenance or as the result of a cyber attack – the Solana network will be unable to process transactions until it’s able to migrate validators, and the network will fail." ... "Together, the two data centres account for more than 55% of the actively staked SOL on the network.

With a majority stake handled by just two ASNs – decentralisation is seriously under threat, which could likely drive future intermittences and disruption if the network is unable to increase the number of validators operating."

It should be noted that the network has continued to decentralize and multiple sources including Anatoly say that the AWS zone with the most nodes is actually Dublin. https://twitter.com/aeyakovenko/status/1550639352610312192

Other Considerations

Solana is on the receiving end of a lot of FUD as well as some occasionally well deserved criticism. It can be difficult to discern whether or not someone is trolling, especially on mediums like Twitter. Accurate reporting by an objective third party can be difficult to source.

On nuanced and inconsistent terminology. There is a distinct difference between and outage and degraded performance. From the user's perspective, if their transaction can not complete in a timely fashion despite consistent behavior they may likely not regard the nuance as meaningful. Media reports also have used terms like 'crash' which is inaccurate and may lead to confusion when users are trying to make sense of suboptimal network events.

For example, when Anatoly says there were only 4 outages as of 2022-07-22, that may be technically correct. https://twitter.com/aeyakovenko/status/1550640253077114880

However, when the media reports on Solana they have significantly different numbers but use the same terms. In this specific case, Fortune states that there were 6 'outages' in January. https://fortune.com/2022/01/25/solana-founder-anatoly-yakovenko-crypto-crash-blockchain-instability/

On deletion versus correction. In the author's view, it is better to have contextual correction instead of deletion. In the case that others are similarly misinformed, they have the opportunity to see the correction and learn immediately.

Positive Outcomes

Due to the post on Anatoly's personal feed, a thoughtful reply came from etherfuse which looked at decentralization of infrastructure for Proof of Stake networks https://etherfuse.substack.com/p/measuring-proof-of-stake-networks

Sources

Solana Status - https://status.solana.com/history Solana Status Twitter - https://twitter.com/SolanaStatus Original tweet which started the discussion- https://twitter.com/theSamPadilla/status/1550627583049584641 Author's incorrect reply - https://twitter.com/alex_wykoff/status/1550630164295520258 Anatoly asks for deletion in personal feed - https://twitter.com/aeyakovenko/status/1550638014065307648

@alexwykoff
Copy link
Author

Correction: Updated and consolidated April & May from original draft as they were the same event which happened at the end of April.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment