Today, the Solana network came close to halting transaction processing due to a routing error that simultaneously disabled multiple validators.
According to data provided by Marinade Finance, at the peak of the event, 28.83% of staked $SOL entered a 'non-transactional' state. The critical threshold for finalizing transactions on the Solana network is 33.34%. Thus, the network reached a point where it could have incurred losses related to the loss of 'finality' if approximately 19.9 million more $SOL had been deactivated.
According to Marinade Finance's post-incident analysis, the network came within about 86% of the threshold at which block finalization would have ceased. Block production continued during the event, which affected around 90 validators, and Solana's operation did not come to a complete halt. However, if the active staking level had fallen below 66.67%, the finalization of new blocks, i.e., their irreversibility, would have stopped.
Therefore, this event is significant not so much as a classic full network outage of Solana, but rather as the network approaching a critical state known as a 'finality halt.' Marinade estimated that the total staking rewards missed by the 90 affected validators amounted to approximately 333 $SOL. For some operators, the outage lasted about 33 minutes.
In Solana, the Problem Was Caused by a Routing Error in a Teraswitch Switch.
The incident was caused by a routing issue within Teraswitch infrastructure. According to the technical analysis conducted, a default route originating from the company's data center in Miami was announced without the necessary specifications. A route reflector in Amsterdam then propagated this route to other data centers in Europe and the Asia-Pacific region.
As a result, some edge routers preferred the erroneous route as a local default route, while the core network considered this route invalid. Loss of valid traffic routes was reported at a total of 12 locations, including London, Amsterdam, Dublin, Frankfurt, Singapore, and Tokyo. North American infrastructure was largely unaffected.
The problem was identified after about 10 minutes, but it took some validators around 33 minutes to return to normal operation.
Marinade's analysis also raised concerns about the risk of congestion in Solana's validation infrastructure.
According to the data provided, autonomous system AS20326 hosts approximately 118.89 million $SOL. This amount constitutes over a quarter of Solana's total staked amount, with about 94 percent of these stakes reportedly offline simultaneously during the incident.
The repercussions of the issue did not end there. Marinade noted that staked amounts totaling around 14.1 million $SOL on other platforms such as Latitude.sh, Limestone, Butterfly Research, and Allnodes were also unavailable at the same time.
The company stated that based on the available data, it is impossible to determine whether the simultaneous impact on these validators was due to reliance on shared infrastructure or if it was a coincidence. However, the incident demonstrated that concentration measurements based solely on hosting provider data may underestimate the true correlation risk.
*This is not investment advice.
