Pavel Marmalyuk

Data analyst @ p2p.org (previously dentsu inc.), graduated from Moscow State University of Psychology and Education, PhD, father, blockchain enthusiast & investor.

2 posts
Solana Jito’s MEV-boosted client adoption & impact on Solana validators performance

<p>Maximal extractable value (MEV) is the additional value that DeFi ecosystem participants (MEV searchers) can extract by influencing transaction inclusion and ordering in blocks produced by validators. Activities such as arbitrage, front-running, NFT sniping, sandwich trading and collateralized positions liquidation present in any DeFi ecosystem contribute to the MEV. Searchers are willing to pay extra fees for priority access to MEV opportunities. These fees ("MEV rewards") can generate significant amounts of additional revenue for validators and their delegators.</p><p>The Jito client, which was launched on Solana mainnet-beta in October, 2022, is the first third-party validator client for Solana which represents a significant improvement to Solana's validator software. Jito software enables more efficient transaction and bundle processing helping both validators and searchers effectively identify and exploit MEV opportunities while eliminating unproductive network spam. It allows validators running the Jito client and their delegators to earn additional revenue from MEV through the Tip Distribution on-chain program which collects and distributes the fees (or “tips”) in proportion equal to a commission set by a validator. The client adoption is good for the Solana ecosystem because it can increase the network's stability, incentivize more validator operators and stakers to join, and help Solana to become more attractive to DeFi ecosystem participants.</p><p>This article will explore statistics on the Jito client adoption within the Solana mainnet-beta cluster, such as the growth of the number of validators running the Jito client, their total active stake and market share. We will also explore the dynamics of MEV rewards generated and compare the performance of Jito validators with that of the rest of the cluster. Additionally, we will investigate whether the adoption of the Jito client has a significant impact on the performance of validators who started to use it. Through this analysis, we aim to shed light on the potential benefits and drawbacks of using the Jito client for validators operating within the Solana network. The data used in this report is publicly available through the P2P Validator public dashboard at: <a href="https://reports.p2p.org/superset/dashboard/jito_client_adoption/?ref=p2p.org">https://reports.p2p.org/superset/dashboard/jito_client_adoption/</a>.</p><h3 id="the-jito-client-adoption">The Jito client adoption</h3><p>The Jito client has been gaining traction among Solana validator operators, as reflected by the growing number of Jito validators (see the left chart in the figure below).</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2023/03/100.png" class="kg-image" alt loading="lazy" width="1450" height="1133" srcset="https://p2p.org/economy/content/images/size/w600/2023/03/100.png 600w, https://p2p.org/economy/content/images/size/w1000/2023/03/100.png 1000w, https://p2p.org/economy/content/images/2023/03/100.png 1450w" sizes="(min-width: 720px) 720px"></figure><p>The significant increase in the number of Solana validators using the Jito client indicates a growing recognition of the software's advantages among operators.</p><p>The number of stakers receiving MEV rewards from Jito-enabled validators (the right chart in the figure above) is showing a positive trend, with two anomalies observed in epochs 385 (+84,075 stakers) and 404 (-82,420 stakers). These anomalies can be explained by the fact that during the epoch 385, the Everstake validator started using the Jito client, and then stopped doing so during the epoch 404, resulting in a sharp change in the number of stakers receiving MEV rewards.</p><p>The table below shows that a few validators have discontinued using the Jito client, with Everstake validator being the most notable among them. The reasons for these validators stopped using the client are unclear as there were no significant changes in the validators performance during the usage of the Jito client.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2023/03/101-1.png" class="kg-image" alt loading="lazy" width="1892" height="1271" srcset="https://p2p.org/economy/content/images/size/w600/2023/03/101-1.png 600w, https://p2p.org/economy/content/images/size/w1000/2023/03/101-1.png 1000w, https://p2p.org/economy/content/images/size/w1600/2023/03/101-1.png 1600w, https://p2p.org/economy/content/images/2023/03/101-1.png 1892w" sizes="(min-width: 720px) 720px"></figure><p>Such important metrics measuring the Jito client adoption as total active stake and market share of validators running Jito client have significantly increased over the last ~50 epochs (as seen in the left and right charts in the figure below). The more active stake the validators running the Jito client have, the more slots are processed with the Jito client, and the more MEV opportunities become available for efficient utilization and redistribution<br></p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2023/03/102.png" class="kg-image" alt loading="lazy" width="1900" height="1073" srcset="https://p2p.org/economy/content/images/size/w600/2023/03/102.png 600w, https://p2p.org/economy/content/images/size/w1000/2023/03/102.png 1000w, https://p2p.org/economy/content/images/size/w1600/2023/03/102.png 1600w, https://p2p.org/economy/content/images/2023/03/102.png 1900w" sizes="(min-width: 720px) 720px"></figure><p>The trend of decreasing average active stake per validator using the Jito client (see the middle chart in the figure above) indicates that more validators with smaller stakes are adopting the software. The increasing trend of Jito client adoption among smaller validators is a positive sign, indicating that even smaller validators can successfully run the software. The Jito client democratizes access to MEV with equal treatment for all validators. The growth in adoption by lower-stake validators demonstrates a strong interest in MEV opportunities from a community that was previously unable to access these benefits.</p><p>The Jito-related MEV rewards currently are very low (as seen in the figure below), which can be attributed to the current limited adoption of the client and lack of participation from MEV searchers. However, a sudden MEV rewards level change after epoch #403 cannot be solely attributed to the increase in the number of validators using the client or the growth of Jito-related active stake. This indicates that a relatively large DeFi ecosystem participant might have started leveraging the MEV extraction tools offered by Jito. As the Jito client gains validator market share, MEV searchers may see more benefits from integration and MEV rewards could rise.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2023/03/103.png" class="kg-image" alt loading="lazy" width="1450" height="1080" srcset="https://p2p.org/economy/content/images/size/w600/2023/03/103.png 600w, https://p2p.org/economy/content/images/size/w1000/2023/03/103.png 1000w, https://p2p.org/economy/content/images/2023/03/103.png 1450w" sizes="(min-width: 720px) 720px"></figure><p>The share of MEV rewards taken by validators running the Jito client has recently increased from about 8% to 21.5% (see figure below). This is mainly due to new validators setting their MEV rewards commission rate to 100%, with many of them being unnamed validators taking 100% stake rewards commission, such as private or white label validators. <br></p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2023/03/109.png" class="kg-image" alt loading="lazy" width="1550" height="1114" srcset="https://p2p.org/economy/content/images/size/w600/2023/03/109.png 600w, https://p2p.org/economy/content/images/size/w1000/2023/03/109.png 1000w, https://p2p.org/economy/content/images/2023/03/109.png 1550w" sizes="(min-width: 720px) 720px"></figure><h3 id="performance-of-the-validators-running-the-jito-client-vs-others">Performance of the validators running the Jito client vs. others</h3><p>The comparison of performance metrics between validators using the Jito client and those who are not is important to gain insights into the differences between the two groups and understand the potential impact of the Jito client on the Solana network.</p><p>Based on the chart below (see figure below), it appears that, on average, Jito validators have better uptime than other validators. This outcome is likely due to the fact that more experienced validators are more likely to experiment with the new Jito client software, while Solana also has a significant number of inexperienced validators with small stake who may not have yet developed the skills or infrastructure necessary to maintain high uptime.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2023/03/110.png" class="kg-image" alt loading="lazy" width="1450" height="1080" srcset="https://p2p.org/economy/content/images/size/w600/2023/03/110.png 600w, https://p2p.org/economy/content/images/size/w1000/2023/03/110.png 1000w, https://p2p.org/economy/content/images/2023/03/110.png 1450w" sizes="(min-width: 720px) 720px"></figure><p>There are outliers for some epochs where the average uptime for Jito validators dropped significantly which is due to a specific validator named “DO NOT DELEGATE” with the vote account pubkey Dn2cRSWAfQpb3NyUJ2q33t1scBLxzo8TZBAyKsWhX7zh, which experienced downtime for 2700 minutes during that epoch and also experienced several very long periods of downtime in other epochs.</p><p>The average vote success rate chart displayed below (see figure below) indicates that Jito validators also generally earn more vote credits for their participation in Solana consensus compared to all other validators. However, due to the metric's volatility, the difference seems insignificant.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2023/03/111.png" class="kg-image" alt loading="lazy" width="1450" height="1080" srcset="https://p2p.org/economy/content/images/size/w600/2023/03/111.png 600w, https://p2p.org/economy/content/images/size/w1000/2023/03/111.png 1000w, https://p2p.org/economy/content/images/2023/03/111.png 1450w" sizes="(min-width: 720px) 720px"></figure><p>The chart below (see figure below) displays the dynamics of the stake-weighted average block production rate for Jito validators and others, indicating a significant difference in favor of Jito validators. This suggests that the Jito client may indeed optimize transaction block processing.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2023/03/112.png" class="kg-image" alt loading="lazy" width="1450" height="1080" srcset="https://p2p.org/economy/content/images/size/w600/2023/03/112.png 600w, https://p2p.org/economy/content/images/size/w1000/2023/03/112.png 1000w, https://p2p.org/economy/content/images/2023/03/112.png 1450w" sizes="(min-width: 720px) 720px"></figure><h3 id="performance-changes-after-adopting-the-jito-client">Performance changes after adopting the Jito client</h3><p>In the previous section, we visually compared the performance metrics of validators running the Jito client and those who are not and observed that there could be statistically significant differences between the two groups. However, the observed differences cannot be solely attributed to the client switch and suggest that other factors may be at play. For instance, early adopters of the Jito client may have more experience in operating validators, and there may be differences in hardware configurations, network connection, or operating conditions that affect performance. Additionally, there are over 2000 Solana validators not running the Jito client, many of which may be operated by inexperienced operators using cheaper hardware, which could further contribute to the observed performance differences.</p><p>In this section we estimate the impact of adopting the Jito client on the performance of Solana validators by comparing their performance metrics before and after adoption, while considering the unique characteristics of each validator. Due to the considerable variation in metrics epoch over epoch caused by external factors, the data was normalized by dividing their values in each epoch by the corresponding epoch average. This normalization method enables a better comparison of validators' relative performance over time and accounts for external factors that greatly impact the metrics for each validator in the cluster. The normalized metrics for each validator before and after the Jito client adoption were averaged to form two samples, which can be compared using the Wilcoxon signed-ranks test. By using the test on the averaged normalized data, we determined whether the adoption of the Jito client had a statistically significant impact on the performance metrics of Solana validators. To ensure sufficient statistical data for both periods, we only compared the performance metrics of 52 validators who ran the Jito client during 25% to 75% of the observed epochs (from 345 to 415). Comparing the relative uptime of validators before and after adopting the Jito client one can observe (see figure below) that the distribution of relative uptime before adoption is wider and has a heavier right tail.</p><figure class="kg-card kg-image-card"><img src="https://lh6.googleusercontent.com/KTSC4o2GgrZQ5C9E0unNZ7yc7qyvikUpwzMhY8_bL0n86jES3VN-gbeqTf30y19e_cOlIHCsvvAX4qjhYwx-8Bs0G0yjZHLKG0QM_hWeyWLdGLkyJylmA8uXk3Mn0dsG4DvEsyQxc36a7Tf5Ay24hxc" class="kg-image" alt loading="lazy" width="602" height="448"></figure><p>Using the Wilcoxon signed-ranks test, we found strong evidence (N = 52, V = 371, p &lt; 0.01) that adopting the Jito client had a small negative impact on the relative uptime reducing the median by ~1.9 p.p. (from 105.8% to 103.9%).</p><p>It's possible that the negative impact on relative uptime is due to the fact that the software is relatively new and still undergoing updates and improvements. Testing of new features requires validator restarts that contribute to some of the downtime. Further research and analysis is needed to better understand the specific factors contributing to the observed differences.</p><p>Comparing the relative vote success rate of validators before and after adopting the Jito client, one can observe (see figure below) that the two distributions are almost identical.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2023/03/107.png" class="kg-image" alt loading="lazy" width="1450" height="1080" srcset="https://p2p.org/economy/content/images/size/w600/2023/03/107.png 600w, https://p2p.org/economy/content/images/size/w1000/2023/03/107.png 1000w, https://p2p.org/economy/content/images/2023/03/107.png 1450w" sizes="(min-width: 720px) 720px"></figure><p>The Wilcoxon signed-ranks test showed no significant positive impact of adopting the Jito client on the relative vote success rate (N = 52, V = 648, p &gt; 0.05).</p><p>Comparing the relative block production rate of validators before and after the adoption of the Jito client, one can observe (see figure below) that there are significant differences in the two distributions: the distribution of relative block production rate after adopting the Jito client is centered around 120%, while the distribution before adoption is centered around 105%.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2023/03/108.png" class="kg-image" alt loading="lazy" width="1450" height="1080" srcset="https://p2p.org/economy/content/images/size/w600/2023/03/108.png 600w, https://p2p.org/economy/content/images/size/w1000/2023/03/108.png 1000w, https://p2p.org/economy/content/images/2023/03/108.png 1450w" sizes="(min-width: 720px) 720px"></figure><p>The Wilcoxon signed-ranks test showed strong evidence (N = 52, V = 946, p &lt; 0.01) that the adoption of the Jito client had a significant positive impact on the relative block production rate of Solana validators increasing the median by ~9.4 p.p. (from 111.5% to 120.9%). The increased block production rate is likely due to the more efficient transaction processing enabled by the Jito client's optimized block engine.</p><h3 id="conclusion">Conclusion</h3><p>The Jito client represents a valuable addition to the Solana ecosystem, providing validators and their delegators with a new revenue stream from MEV opportunities, while helping the Solana network to be more stable.</p><p>The Jito client has yet to gain widespread adoption, however, the number of validators utilizing the software is steadily growing, along with the total active stake and staking market share attributed to the Jito client. Some validators have stopped using the client, but they constitute a small fraction and the reasons for this are unclear.</p><p>Additionally, Jito validators and their stakers have not yet earned significant MEV rewards, which may be due to the lack of usage of the client by MEV searchers and users. This situation should improve with broader adoption of the client and as searchers become more accustomed to the new tools.</p><p>On average, validators running the Jito client have better performance than others, although statistical analysis shows that uptime of the validators currently running Jito client has slightly decreased after the switch, while vote success rate has remained largely unchanged and block production rate has increased significantly.</p><p>For those interested in exploring the data further, P2P Validator's public dashboard provides access to all the data used in the report preparation: <a href="https://reports.p2p.org/superset/dashboard/jito_client_adoption/?ref=p2p.org">https://reports.p2p.org/superset/dashboard/jito_client_adoption/</a>.</p><h3 id="acknowledgments">Acknowledgments</h3><p>We would like to express our gratitude and appreciation to the P2P Validator team members, including <a href="https://twitter.com/pavpvlv?ref=p2p.org">Pavel Pavlov</a>, <a href="https://twitter.com/Sybertuk?ref=p2p.org">Anton Yakovlev</a>, <a href="https://twitter.com/stevencquinn?ref=p2p.org">Steven Quinn</a>, and <a href="https://twitter.com/abondar92?ref=p2p.org">Alexey Bondar</a> , for their invaluable guidance, support, and encouragement throughout this research. Furthermore, we would like to express gratitude to the <a href="https://twitter.com/jito_labs?ref=p2p.org">Jito Labs</a> team, especially <a href="https://twitter.com/brian_smith_0?ref=p2p.org">Brian Smith</a> and <a href="https://twitter.com/buffalu__?ref=p2p.org">Lucas Bruder</a>, for their support and openness during the research. We would also like to thank <a href="https://twitter.com/brianlong?ref=p2p.org">Brian Long</a> and his team for creating the <a href="https://twitter.com/ValidatorsApp?ref=p2p.org">Validators.app API</a>.</p><h3 id="sources">Sources</h3><p>Overall information on Jito &amp; MEV:</p><ol><li><a href="https://jito-foundation.gitbook.io/mev/?ref=p2p.org">https://jito-foundation.gitbook.io/mev/</a></li><li><a href="https://medium.com/@Jito-Foundation/solving-the-mev-problem-on-solana-a-guide-for-stakers-7768308e93bc?ref=p2p.org">https://medium.com/@Jito-Foundation/solving-the-mev-problem-on-solana-a-guide-for-stakers-7768308e93bc</a></li></ol><p>Dashboards:</p><ol><li><a href="https://reports.p2p.org/superset/dashboard/jito_client_adoption/?ref=p2p.org">https://reports.p2p.org/superset/dashboard/jito_client_adoption/</a></li><li><a href="https://dune.com/pavelm/jitovalidatorsmevrewards?ref=p2p.org">https://dune.com/pavelm/jitovalidatorsmevrewards</a></li><li><a href="https://jito.retool.com/embedded/public/7e37389a-c991-4fb3-a3cd-b387859c7da1?ref=p2p.org">https://jito.retool.com/embedded/public/7e37389a-c991-4fb3-a3cd-b387859c7da1</a></li><li><a href="https://jito.retool.com/embedded/public/e9932354-a5bb-44ef-bce3-6fbb7b187a89?ref=p2p.org">https://jito.retool.com/embedded/public/e9932354-a5bb-44ef-bce3-6fbb7b187a89</a></li></ol><p>Data sources / APIs:</p><ol><li><a href="https://docs.solana.com/api/http?ref=p2p.org">https://docs.solana.com/api/http</a></li><li><a href="https://www.validators.app/?ref=p2p.org">https://www.validators.app/</a></li><li><a href="https://console.cloud.google.com/storage/browser/jito-mainnet?ref=p2p.org">https://console.cloud.google.com/storage/browser/jito-mainnet</a></li><li><a href="https://jito-foundation.gitbook.io/mev/jito-solana/tracking-jito-solana-validators?ref=p2p.org">https://jito-foundation.gitbook.io/mev/jito-solana/tracking-jito-solana-validators</a></li></ol><p><br></p><p><br></p><p><br></p><p><br></p>

Pavel Marmalyuk

from p2p validator

Solana Solana Validators Performance Research, part 1: Downtime Analysis

<p>Welcome to the first article in the series of publications on the Solana validators performance research by P2P Validator. In our opinion, performance of the Solana network validators is one of the most important aspects which determine the network growth and sustainability. Our team has done a deep dive into this topic and we want to share insights gained to benefit the Solana community. </p><p>The research is devoted to the analysis of the two most important metrics reflecting Solana network validators’ performance: downtime duration (node delinquency/unavailability duration) and skip rate (measuring how frequently a node fails to produce a transaction block which is subsequently confirmed by consensus on the network). </p><p>In this article we reveal the first part of the research findings regarding analysis of downtime. You can find <a href="https://www.stakingrewards.com/journal/solana-validators-performance-research-report-part-2-skip-rate-analysis/?ref=p2p.org">Part 2 here</a>, covering block skip rate analysis results. </p><h2 id="table-of-contents">Table of contents</h2><!--kg-card-begin: markdown--><ul> <li><a href="#preface">Preface</a></li> <li><a href="#introduction">Introduction</a></li> <li><a href="#solana-validators-downtime">Solana validators downtime</a></li> <li><a href="#factors-influencing-downtime-duration">Factors influencing downtime duration</a></li> <li><a href="#downtime-data-analysis">Downtime data analysis</a></li> <li><a href="#nodes-downtime-duration-by-epochs">Nodes downtime duration by epochs</a></li> <li><a href="#supermajority-and-superminority-validators-comparison">Supermajority and superminority validators comparison</a></li> <li><a href="#downtime-duration-distribution-for-updates-and-other-causes">Downtime duration distribution for updates and other causes</a></li> <li><a href="#average-update-time-by-solana-software-versions">Average update time by Solana software versions</a></li> <li><a href="#summary">Summary</a></li> <li><a href="#acknowledgements">Acknowledgements</a></li> <li><a href="#disclaimer">Disclaimer</a></li> <li><a href="#sources-of-data">Sources of data</a></li> </ul> <!--kg-card-end: markdown--><h2 id="preface">Preface</h2><p>All data used for analysis in the research were obtained from publicly available sources such as Solana JSON RPC API, Solanabeach API, Validators.app API and are relevant for Mainnet beta epochs №194-236 unless another epoch or time period is explicitly specified.</p><h2 id="introduction">Introduction</h2><p>Solana is a relatively new (went live in March 2020) public high-performance distributed blockchain platform curated by Solana Foundation (non-profit organization headquartered in Geneva, Switzerland) along with professional blockchain developers, organizations and individuals running validator and RPC nodes and DevOps specialists from all over the world who are dedicated to the decentralization, growth, and security of the Solana network.</p><p>Solana is one of blockchains that aims to be fast and scalable, without compromising its security or decentralization. Its theoretical throughput limit of 50k transactions per second (TPS) which is twice more than VISA’s limit, which means it can be used for many real-time applications in various business areas. Solana mainnet has already handled more than 35 billion transactions with current throughput exceeding 2000 TPS (see Figure 1) due to high demand for its capabilities and various use cases including: ultra-fast on-chain payments, token creation and distribution, staking through delegation to network validators, smart-contracts, NFTs issuance and trading. Solana ecosystem also provides many different DeFi services such as decentralized exchange, token swaps, liquidity farming and bridging ensuring cross-chain interoperability with other blockchains.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh5.googleusercontent.com/VYf4yl-UeoEH2-bfH8v7ACSsiTxl2n6d4TM46f_sAKeOUwW_rRCkSelwlOGzzqb9uTXHyj5YQKAK-Wk3XSZDJZLg5EK94MuogDL1J-DWXv5oW8awbwR-tptoGxwmCvY2LYTYlg3L" class="kg-image" alt loading="lazy"><figcaption><em>Figure 1. Solana’s TPS on 27.10.2021 (see <a href="https://explorer.solana.com/?ref=p2p.org">explorer.solana.com</a> for live data).</em></figcaption></figure><p></p><p>There are currently more than 1000 independent validators and 800 RPC nodes (see Figure 2) which comprise a physical layer for above mentioned functionality while making the network highly secure and decentralized. Each validator supports the network's operation by providing <a href="https://docs.solana.com/running-validator/validator-reqs?ref=p2p.org">high-end hardware resources</a> and properly configuring their systems to keep the network running as fast and smooth as possible.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh4.googleusercontent.com/78LCSkFGLBansse0T67-5AmbeRFdQkV4tnzyPE06EXNouo_Kr28DMAXKOv7DiFUo-7m05j0ntYPc0RbblbubK4aLwMSUfr3jgFb17XdETt55bJar-v6s19xi2HrZsfJCpkTQK0jq" class="kg-image" alt loading="lazy"><figcaption><em>Figure 2. Solana validator nodes map (see <a href="https://solanabeach.io/?ref=p2p.org">https://solanabeach.io/</a> for live data).</em></figcaption></figure><p></p><p>Validators receive SOL tokens from stakers, participate in the consensus-based process of transactions validation, get rewards proportional to delegated stake amount and distribute these rewards to stakers (proportionally to staked shares) charging a variable commission. The more stake is delegated to a validator, the more this validator (and its delegators) earns and is more frequently chosen to process new transactions on the ledger and, thus, exposed to greater hardware and network load. Thus, on the one hand, validators are economically motivated to keep their hardware and software running without interruptions, and, on the other, to timely update Solana software and to improve their nodes and network connection as their stake and the Solana network load increase.</p><h1 id="solana-validators-downtime">Solana validators downtime</h1><p>It is normal for a node to be temporarily unavailable/offline sometimes as every technical system needs periodic maintenance and reconfiguration. Typical reasons for server unavailability are usually quite simple such as planned reboot to update host configuration or software, an emergency (power outage), network problems in the data center or at the provider.</p><p>The longer a node is unavailable, the fewer staking rewards and transaction fees it receives. Staking rewards are paid proportionally to node’s vote transactions count which it cannot post if it is offline or functioning incorrectly. Validator  downtime negatively affects its delegates’ rewards, which is why one should consider checking validator recent downtime duration history before delegating to it.</p><p>During periods of downtime an unavailable validator is assigned the “delinquent” status which can be checked using Solana CLI <em>solana validators</em> command or by parsing corresponding json response (<em>solana --output json validators</em>). By constantly fetching statuses of all validators on the network it is possible to measure delinquency periods durations which is a good approximation for downtime duration for further quantitative analysis. The downtime data analyzed is available through the <a href="https://redash.p2p.org/public/dashboards/ZEW9RvuBXPdYHUU5aC7DfVjY8DJaXQrGenG2HUmo?org_slug=default&ref=p2p.org">public Redash dashboard</a>.</p><h2 id="factors-influencing-downtime-duration">Factors influencing downtime duration</h2><p>There are many factors influencing downtime duration and these are typical ones:</p><ul><li>node operator reaction time (node operators may or may not use specialized monitoring and alerting systems);</li><li>node operator skill (imagine the difference between inexperienced enthusiasts and mature professionals who are working with such high-load systems for years);</li><li>time to repair breakdowns in the power grid or communication network (which does not depend on a node operator);</li><li>time needed to debug and fix specific configuration errors or replace hardware parts;</li><li>complexity and duration of software update (i.e., different Solana versions take different time to install), node startup duration, etc.</li></ul><p>Despite most of these factors can not be measured directly, we have managed to collect and analyse some important on-chain data related to the topic, which allowed to quantitatively describe several aspects regarding Solana network nodes unavailability such as downtime duration statistics over time, its variability across nodes as well as duration of node software updates.</p><h2 id="downtime-data-analysis">Downtime data analysis</h2><p>Here we illustrate retrospective downtime statistics of Solana nodes that were active in the period from epoch №209 (5th of August, 2021) to epoch №236 (17th of October, 2021). Historical data allow to reveal trends in the dynamics of downtime making it easier to understand the normal behavior of the metric as well as to identify abnormal fluctuations.</p><h3 id="nodes-downtime-duration-by-epochs">Nodes downtime duration by epochs</h3><p>The descriptive statistics for downtime duration by epochs are presented in the Figure 3 below. Quantile values of 5%- and 95%-level reflect the maximum downtime among the top 5 and top 95 percent of validators, respectively, for each epoch. Average downtime is the simple arithmetic mean and median defines a downtime duration which divides the top 50 and worst 50 percent of validators.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh3.googleusercontent.com/aQCsTug3snvF4n1VUW3AXUBYvC6FytRw5Q2ICuSuMsZMr8RGjpyI-ylB6CZK8qU8WR5EKHnBNMPfMBsV6CXTN7UM_sMO3k_QPjq5JhoVDPJW0dEU6FTJUr0skrGCbjcKdpVQAB5m" class="kg-image" alt loading="lazy"><figcaption><em>Figure 3. Downtime mean (cyan line), median (lilac), 5%-quantile (red), 95%-quantile (green) and its actual values for each node (black transparent dots) over epochs.</em></figcaption></figure><p><br>As can be seen from the chart above, <strong>typical average downtime duration for a node is around 1.5 hours</strong>, which is quite low, while <strong>median downtime duration is almost always zero</strong> (which means that most nodes usually don’t experience shutdowns). Also there were several epochs (№214, 223 and 234) with high downtime duration upticks mainly due to simultaneous upgrades of Solana software version. Epoch 223 is especially interesting as it is known that on 14 of September, 2021, the Solana network experienced a <a href="https://jumpcrypto.com/reflections-on-the-sept-14-solana-outage/?ref=p2p.org">severe overload which led to the network halt</a>, and after a successful network restart almost all the nodes had to update to a new Solana version with the necessary fixes.</p><h3 id="dispersion-of-downtime-duration">Dispersion of downtime duration</h3><p>As many factors affect downtime duration, it varies greatly across validators within the same epoch. The dispersion measures indicate the metric’s spread magnitude which is slightly changing over time as shown in Figure 4.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh5.googleusercontent.com/pYyFq6WJf7ZUvigy7YMAFPcT0e9OISiG4_mNYWZLs-N_c7vUrs09jUb9wtgCxgH4OqNIAppwlrjif3vupB0ZWDAtSc2mv8gI8ixFAkvcVVYs859GnOtTiXvGN5SQSDjPA2ql9Hjv" class="kg-image" alt loading="lazy"><figcaption><em>Figure 4. Measures of dispersion of downtime duration over time: interquartile range (red line) representing difference between 25%- and 75%-quantiles of the metric and standard deviation (cyan) representing average deviation from mean.</em></figcaption></figure><p><br>It can be seen from the chart above that downtime duration dispersion across validator nodes is dropping slightly over time which indicates that validators, on average, have both lower downtime durations and lower deviations of the metric from the mean.</p><h3 id="supermajority-and-superminority-validators-comparison">Supermajority and superminority validators comparison</h3><p>Since the leading validators with a large stake amount take much more financial risks compared to the smaller ones, their nodes' technical characteristics are far better than of the majority. Therefore, it makes sense to compare performance of the superminority set of validators (the minimal set of validators that together control more than 33.33% of the total stake) with the rest falling into the supermajority set with 66.66% of total stake (see Figure 5).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh5.googleusercontent.com/pXXTHF4CC5szdub_hF8cYrIu_BpisJxblMbSODq5p4IWU7gdhGI5XXOyDGOLbATya7vLvaH0rDLWXJsLZY_D1OiAD9MpKSRR3ZkrB_oXBSagqZKNeAxlE1dik8nVo9moZvIN3aC6" class="kg-image" alt loading="lazy"><figcaption><em>Figure 5. Average downtime duration by epochs for superminority validators set (blue line), supermajority (green) and P2P Validator (red).</em></figcaption></figure><p><br>As the charts above represent, the supermajority is usually much worse in terms of average downtime duration, especially after epoch №220, especially during hard times like epoch №223, when the Solana network halted and most validators had to perform major software updates.</p><p>In contrast, superminority validators (especially P2P Validator) have an average downtime duration and an average number of downtimes (see Figure 6 below) that are much lower than for the supermajority, and there is a much smaller probability that a validator from the superminority set is offline for more than 5% of total epoch duration (see Figure 7 below).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh5.googleusercontent.com/vUWC-dbCA4C7q5xb2U29_ch4guExlxtY1-5QMdK8vTS9Ek_3lpPFnPFqV12qM2u8ErrpDUtgMfD4333AjhqiP1cPMqHfxGfZpaI-AmZGitsKPgLKJz5_dvesr5OMehcU1Mljfhvi" class="kg-image" alt loading="lazy"><figcaption><em>Figure 6. Average number of downtimes over epochs №209-236 for the superminority (cyan line) and supermajority set of validators (red).</em></figcaption></figure><p></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh4.googleusercontent.com/ZQeA4Y0DFh4ontzVLbMRC2MNw_6Dh1_7oe8is52gxvxSWfROVrXbE2PjzlrbIOoT-hVp8pu9CkfuyhAH9ovBd2dUq1-yLrexjo2xwOKjXbJdsylkLYtfzUxD1Zh6J9r_hvkCsNLU" class="kg-image" alt loading="lazy"><figcaption><em>Figure 7. Share of validators with downtime duration greater than 5% of total epoch duration over epochs №209-236 for superminority (cyan line) and supermajority set of validators (red).</em></figcaption></figure><h3 id="downtime-duration-distribution-for-updates-and-other-causes"><br>Downtime duration distribution for updates and other causes</h3><p>As it was described previously, downtimes may happen due to Solana node software updates as well as due to hardware upgrades and unexpected halts. The available on-chain data allows to distinguish between downtimes related to software updates and related to other causes and compare downtime duration distributions for the supermajority and superminority groups of validators.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh5.googleusercontent.com/vKMR9AxmKV1-dm-jSA0f_mSFiEZV_oR0ur9g8_R5666r4z1vSn8OJcikdkq9La4l99IWwzLZgv-1OATphpDFbdEh2izdijhIDTmYTJ4GlpPPr7-vKAa-0HoaZXwhoNmzYu1ilpBJ" class="kg-image" alt loading="lazy"><figcaption><em>Figure 8. Downtime duration distribution (in logarithmic scale) unrelated to software updates for supermajority (green line), superminority (blue) set of validators and P2P Validator (red). Dashed lines of corresponding colors show the average downtime duration written nearby</em></figcaption></figure><p><br>According to the distributions of downtime duration not related to software updates (see Figure 8), validators groups are quite similar apart from the fact that supermajority validators are more likely to have very long outages that greatly increase the average value of downtime duration (69 vs. 34 minutes for the superminority group). It should be noted that even if the P2P Validator goes down (or delinquent), on average it happens for an extremely short time of 1.5 minutes.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh6.googleusercontent.com/Ho-gbXZDhA6SyHlPnJxSGrHE2F5x-Cn8HGwX0K30wuoibCH6Ay_Rx1Obq8D4Se3MoUATYUxICIKq-7RuKkZhcct1ARAUruNtCaGi4_1lNBsDEhJYpGS3Fao5osWpA1lV4oeWyYhC" class="kg-image" alt loading="lazy"><figcaption><em>Figure 9. Downtime duration distribution (in logarithmic scale) related to software updates for supermajority (green line), superminority (blue) set of validators and P2P Validator (red). Dashed lines of corresponding colors show the average downtime duration written nearby.</em></figcaption></figure><p><br>Concerning downtimes due to software updates (see Figure 9), the distributions for the groups differ considerably: for the superjmajority group there is much more variability in downtime duration when compared to the superminority and again supermajority validators frequently have much longer update times leading to higher average (195 vs. 76 minutes for superminority group). Superminority validators including the P2P Validator demonstrate high consistency in update duration presumably due to specific administration standards developed by professional engineers who operate these validators.</p><h3 id="average-update-time-by-solana-software-versions">Average update time by Solana software versions</h3><p>Different Solana node software versions vary significantly in the complexity and duration of the installation process, which directly affects the downtime duration associated with updates. Figure 10 below shows the average update time of Solana node software versions by validators from the supermajority and superminority groups.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh3.googleusercontent.com/0ahrMb4jlGfbkVuMN5FNxFwdjqx5563dDEDqWSLqdpz0w9edfuUv7_Y6QWh0FFM704pbpyQHtmjKPTskVDj54VjWDgnPdARp5nSOnUiUueOuJq6BJbVUjpo45J66poJ8ZK26HX6c" class="kg-image" alt loading="lazy"><figcaption><em>Figure 10. Average update time for different Solana node software versions.</em></figcaption></figure><p><br>Of all the most used versions of Solana node software, the update to version 1.6.25 took the longest for both supermajority (4.5 hours on average) and superminority (3.5 hours on average) validators. Long updates to versions 1.7.11 and 1.7.15 were performed only by validators from the supermajority group and took approximately 2-3 hours to complete. Overall, validators from the superminority group usually perform the updates significantly faster ensuring less rewards losses for them and their delegators.</p><h2 id="summary">Summary</h2><p>Downtime duration is a very important metric as it reflects Solana validators operators efficiency and influences rewards received by validators and their delegators as well as overall network’s stability and security. Solana Foundation and the network validators do everything they can to improve performance of nodes and quality of software that control nodes operation, and we can say with confidence that they do it very well, especially validators from superminority group thanks to the experience and professionalism of DevOps engineers.</p><h2 id="acknowledgements">Acknowledgements</h2><p>Authors of the report would like to express gratitude and appreciation for the <a href="https://p2p.org/?ref=p2p.org">P2P Validator</a><strong> </strong>team whose guidance, support and encouragement have been invaluable throughout the research. We would also like to thank <a href="https://twitter.com/stephenakridge?ref=p2p.org">Stephen Akridge</a>, co-founder of Solana, <a href="https://twitter.com/vmulps?ref=p2p.org">Ruud van Asseldonk</a>, software engineer at Chorus One, and <a href="https://twitter.com/RDorzbach?ref=p2p.org">Robert Dörzbach</a>, product manager of the Solana Beach, for helpful advice, comments and corrections.</p><h2 id="disclaimer">Disclaimer</h2><p>Information presented in this report and referenced sources are for educational purposes only. It is not financial/investment advice. Seek a licensed professional for any financial advice. Authors of the report made every reasonable effort to ensure the accuracy and validity of the information provided. However, as price points, conditions, and information are continually changing, authors reserve the right to change at any time without notice, information contained in the report and make no warranties or representations as to its accuracy or up-to-dateness.</p><p>Authors of the report are employees of P2P Validator company which provides professional services and consulting for highly secure non-custodial staking across more than 25 blockchain networks, including the Solana network with mainnet and testnet validator nodes as well as RPC nodes. Therefore, P2P Validator is not a neutral party with its own business interests in the Solana ecosystem. Nevertheless, authors did their best to make the report as objective as possible with the main purpose in mind being to educate and inform the community.</p><h2 id="sources-of-data">Sources of data</h2><ol><li><a href="https://app.swaggerhub.com/apis-docs/V2261/solanabeach-backend_api/0.0.1?ref=p2p.org">https://app.swaggerhub.com/apis-docs/V2261/solanabeach-backend_api/0.0.1</a></li><li><a href="https://docs.solana.com/developing/clients/jsonrpc-api?ref=p2p.org">https://docs.solana.com/developing/clients/jsonrpc-api</a></li><li><a href="https://www.validators.app/api-documentation?ref=p2p.org">https://www.validators.app/api-documentation</a></li><li><a href="https://redash.p2p.org/public/dashboards/ZEW9RvuBXPdYHUU5aC7DfVjY8DJaXQrGenG2HUmo?org_slug=default&ref=p2p.org">https://redash.p2p.org/public/dashboards/ZEW9RvuBXPdYHUU5aC7DfVjY8DJaXQrGenG2HUmo?org_slug=default</a></li></ol><hr><h2 id="about-p2p-validator">About P2P Validator</h2><p>P2P Validator is a world-leading non-custodial staking provider <strong><strong>securing more than </strong>4<strong> billion USD value from over </strong>2<strong>0,000 delegators across 25+ high-class networks</strong></strong>. We are early investors in Solana and have supported the network from the first block taking part in all stages of testing and voting.</p><hr><p><strong><strong>Web</strong></strong>: <a href="https://p2p.org/?ref=p2p.org">p2p.org</a><br><strong><strong>Stake SOL with us</strong></strong>: <a href="https://p2p.org/solana?ref=p2p.org">p2p.org/solana</a><br><strong><strong>Twitter</strong></strong>: @p2pvalidator<br><strong><strong>Telegram</strong></strong>: <a href="https://t.me/P2Pstaking?ref=p2p.org">t.me/P2Pstaking</a></p>

Pavel Marmalyuk

from p2p validator