TheGraph Loss of Epoch Rewards in Graph Network: A Post-mortem

<h2 id="tldr">TLDR</h2><p>Due to an old version of an indexer agent combined with a short-time node desynchronisation, closed allocation resulted in a zero proof of indexing on February 6th 2021. Due to this rewards for the epoch vanished. Lost rewards will be fully compensated by waiving our fee and sharing our own indexing rewards with our delegators. </p><h2 id="what-happened">What happened?</h2><p>An oversight in an upgrade process led to the indexer-agent version being out of date. A minor hiccup in Ethereum node operation coincided with the allocation closing, resulting in the closing of an allocation without a proof of indexing being submitted. Rewards will be fully compensated over the course of 12 days by charging 0% fee and sharing our own indexing rewards. </p><h2 id="what-went-wrong">What went wrong?</h2><p>We provide regular reward stats on a daily/weekly basis for delegators with locked GRT and thus had no monitoring of epoch rewards after closing an allocation. This led to a minor delay in reaction as reports are filled automatically without human intervention. Right after realising that rewards had been lost we set up an internal investigation.</p><p>Additionally was the absence of a well-reviewed upgrade procedure for new indexer releases. The upgrade procedure had no clear standard and checklist of actions to follow. New allocations were opened and no technical mistakes occurred, but existing monitoring that allocation was insufficient. We noticed that monitoring for a subgraph syncing delay had insufficient resolution time-wise so failed to catch a small hiccup in Ethereum node operation.</p><h2 id="what-went-well">What went well?</h2><p>We close and open allocations on a daily basis so we lost only one day of rewards. We collect statistics on stake changes every day and spotted a problem with reward crediting. We also thank our delegators who have noticed the issue and pointed concerns out in our <a href="https://t.me/P2Pstaking">Telegram channel</a>. </p><h2 id="impact-on-clients">Impact on clients</h2><p>All our Graph Network delegators lost one day of indexing rewards. To compensate and mitigate their loss, P2P will waive the fees for 12 days in addition to distributing all the rewards from our own indexer stake for that period. </p><h2 id="lessons-learned">Lessons learned</h2><p>We should have had monitoring for new version releases and checked that software is up to date before taking actions. To avoid such issues in the future we have defined a clear written procedure for upgrades and set monitoring for all necessary releases. This case revealed flaws in our existing monitoring so we have now initiated a deep analysis of all metrics improving it from ground up and adding necessary alerts to support this. </p><p>We have decided to make reviews for all standard operating procedures more rigorous by allocating more engineering time to these activities. Testing and improvement will be continued on the new testnet where we are already set up. All the crucial operations, upgrades and innovations that can potentially lead to a financial loss will be implemented on testnet first.</p><p>P2P takes full responsibility for this and we are sorry for the inconvenience. Please be assured that P2P is taking actions to eliminate even a small probability of such an event occurring in future.</p>

Alex Bond

from p2p validator

Staking, TheGraph Graph Network - Brief Overview

<p>The Graph Network is an indexing protocol built for the decentralised web. Users of the Graph Network can build and share open APIs across Ethereum and IPFS to allow for fast and reliable blockchain data access. Data is managed and processed via open networks, providing users with verifiable integrity.</p><p></p><h2 id="current-network-state">Current network state</h2><p>The Graph Network was <a href="https://medium.com/graphprotocol/introducing-the-graph-4a281b28203e">introduced</a> in July 2018 and launched their <a href="https://thegraph.com/blog/the-graph-network-launches-mainnet">mainnet</a> this year, on December 17th, as the first decentralized blockchain data network.</p><p>The Graph Network works to improve access to decentralized applications through public and open APIs named subgraphs. Subgraphs dictate which data the network will index from Ethereum, as well as how this data is stored. Upon deployment, a subgraph forms part of a global blockchain data network.<br>During their testnet phase, Graph saw over 200 active indexers, together with more than 1600 individuals, complete their <a href="https://thegraph.com/blog/curator-program-launch">Curator Program</a> for decentralising data across DeFi.</p><p>Between June and November 2020 they have seen a 10x increase - from 1 to 10+ billion monthly queries - with usage across Curve, Uniswap, Decentraland and more.</p><p><br></p><h2 id="grt-token-allocation">GRT Token allocation</h2><p>In October 2020, Graph Foundation <a href="https://thegraph.com/blog/public-sale-announcement">announced a public sale</a> of GRT and successfully raised $12M. Earlier this year the Graph Foundation raised $5M in a <a href="https://thegraph.com/blog/funding-announcement">private token sale</a>.</p><p>The details of the GRT token can be found below:<br></p><ul><li><strong>Symbol</strong>: GRT</li><li><strong>Type</strong>: ERC-20</li><li><strong>Initial Circulating Token Supply</strong>: ~1,245,666,867 GRT</li><li><strong>Initial Total Token Supply</strong>: 10,000,000,000 GRT</li></ul><figure class="kg-card kg-image-card"><img src="https://economy.p2p.org/content/images/2020/12/GRT_Piechart_v2-2.png" class="kg-image" alt srcset="https://economy.p2p.org/content/images/size/w600/2020/12/GRT_Piechart_v2-2.png 600w, https://economy.p2p.org/content/images/size/w1000/2020/12/GRT_Piechart_v2-2.png 1000w, https://economy.p2p.org/content/images/size/w1600/2020/12/GRT_Piechart_v2-2.png 1600w, https://economy.p2p.org/content/images/size/w2400/2020/12/GRT_Piechart_v2-2.png 2400w" sizes="(min-width: 720px) 720px"></figure><p>More information on the <a href="https://thegraph.com/blog/announcing-the-graphs-grt-sale">GRT token sale</a> is available here: thegraph.com/blog/announcing-the-graphs-grt-sale </p><p>4% of the total GRT supply - 400M tokens - was allocated to 4,500+ persons across more than 90 countries. These delegators can contribute to the Graph Network through delegating their stake to network Indexers - like P2P - to earn indexing rewards.</p><p></p><h2 id="graph-network-staking-economics">Graph Network staking economics</h2><p>Staking in the Graph Network is a method of securing their data network and rewarding network contributions.</p><p>Node operators across the network can lock their GRT tokens to provide indexing and query processing services for improved data access. These node operators - also known as ‘Indexers’ can earn fees and indexer rewards for this.</p><p>Delegators across the network are those who hold the GRT token can stake it to contribute to the overall network security and receive a share of indexer rewards.</p><p>Delegation lets passive, less technical participants secure the Graph Network without the need to run a node.Indexers - like P2P - set a Fee and Reward Percentage which they keep, with the rest of the rewards transferring to Delegators. The annual emission in The Graph network is set to 3%.<br><br>There is no slashing risk for delegators but the current Indexer slashing percentage is at 2.5%.</p><p><a href="https://p2p.org">P2P Validator</a> currently has the largest Graph Network Indexer globally, with a 33M GRT balance and 530M GRT capacity. Our Indexer balances are available on network.thegraph.com.</p><p></p><p><strong>Address 1: </strong>0x5A8904be09625965d9AEc4BFfD30D853438a053e (0x5a89—8a053e)</p><p><strong>Address 2: </strong>0xf4a097ce3a4efbd1748b2ef2076813961e4e6fa7 (0xf4a0—4e6fa7)</p><p></p><p>Users that want to contribute to the stability and security of the Graph Network without the complexities associated with node management can delegate staking to Indexers and earn a portion of on-chain indexing reward. </p><p>To delegate GRT with us, please refer to the resources below:<br></p><ul><li><strong><a href="https://economy.p2p.org/the-graph-delegation-guide-for-locked-grt/">Graph (GRT) Delegation Guide - Locked tokens</a></strong></li><li><strong><a href="https://economy.p2p.org/the-graph-delegation-guide-for-unlocked-grt/">Graph (GRT) Delegation Guide - Unlocked tokens</a></strong></li></ul><p></p><figure class="kg-card kg-embed-card"><blockquote class="twitter-tweet" data-width="550"><p lang="en" dir="ltr">We are proud to have launched the largest <a href="https://twitter.com/graphprotocol?ref_src=twsrc%5Etfw">@graphprotocol</a> Indexer and self-bond worldwide with an owned stake of 33.4M GRT and 500M GRT available capacity.<br><br>0x5a8904be09625965d9aec4bffd30d853438a053e<br><br>See our GRT delegation guide 👉 <a href="https://t.co/MJMCtrztfY">https://t.co/MJMCtrztfY</a></p>&mdash; P2P Validator (@P2Pvalidator) <a href="https://twitter.com/P2Pvalidator/status/1341371588139229185?ref_src=twsrc%5Etfw">December 22, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> </figure><p></p><h2 id="graph-network-resources">Graph Network resources</h2><ul><li><strong>Website</strong>: <a href="https://thegraph.com/">thegraph.com</a></li><li><strong>Explorer</strong>: <a href="https://thegraph.com/explorer/">thegraph.com/explorer</a></li><li><strong>Docs</strong>: <a href="https://thegraph.com/docs/">thegraph.com/docs</a></li><li><strong>Network</strong>: <a href="https://network.thegraph.com/">network.thegraph.com</a></li><li><strong>Github</strong>: <a href="https://github.com/graphprotocol">github.com/graphprotocol</a></li><li><strong>Blog</strong>: <a href="https://thegraph.com/blog/">thegraph.com/blog</a></li></ul><p></p><h2 id="about-p2p-validator">About P2P Validator</h2><p><a href="https://p2p.org/">P2P Validator</a> is a world-leading non-custodial staking provider securing more than $350 million by over 4,000 delegators across 25+ top-notch networks. We have participated in the incentivized testnet since the beginning. P2P Validator invested its own funds in The Graph project in 2018 and intends to run indexers in the long term.</p>

Kasper Rasmussen

from p2p validator

TheGraph The Graph Delegation Guide For Locked GRT

<p><em>If you have participated in private sale of GRT and received email from The Graph team with a vesting contract address this guide will help you to delegate locked tokens.</em></p><ol><li>Go to<a href="https://network.thegraph.com/"> </a><a href="https://remix.ethereum.org/">https://remix.ethereum.org/</a></li><li>Press on a “<em>browser +</em>” symbol</li></ol><figure class="kg-card kg-image-card"><img src="https://lh4.googleusercontent.com/R0dIgqHs-32c5FRTcmhy3Bqhm-0x_aiuz2TMuGww8eHf5IkfH2K-rzG8IT9SfFN64TZeiFr6oGuxkJbkrBLjIwc9QaNcK8ODUWbayleXAWT-ms-PfgCJDsCW5Cf8UqMXtQmQEVBn" class="kg-image" alt></figure><p>3.  Add a name <em>TokenLock.abi</em></p><figure class="kg-card kg-image-card"><img src="https://lh4.googleusercontent.com/al_4QAWsYDWJPt82NVQ14sDP5nF_HkQwtZTLqmeMkOZPgKOoU9du05ZFGEdj2TgCyyolWDlhLtJ9Ol-Oyatb6pSL3DWArNYzXtpQFWMevYHio9ltQ9z4Y1GJXqelF_b9ijtpXGe2" class="kg-image" alt></figure><p>4.  In a separate browser tab open: <a href="https://gist.githubusercontent.com/abarmat/20ac76441a7f4832ddc7e9a8b6fc2301/raw/3cc959f8233d35852afd81f42eba1fa8dac9f5a2/TokenLockWalletABIRemix.json">https://gist.githubusercontent.com/abarmat/20ac76441a7f4832ddc7e9a8b6fc2301/raw/3cc959f8233d35852afd81f42eba1fa8dac9f5a2/TokenLockWalletABIRemix.json</a><br>Copy the whole code and paste it in the <em>TokenLock.abi</em> that you have created in the previous step</p><figure class="kg-card kg-image-card"><img src="https://lh6.googleusercontent.com/JVNktjogZZ-Rez2_5t3qbaLtUduetWNY1JmaAFAdjfd44YnRrIgZ6GKxsjz870NLasKtCVjObj9NJqjYPRXcIc1eIhmHkxPwVZBGZvmqO9eEWenkBGF2Yis6FSTYGijq1jD6bYza" class="kg-image" alt></figure><p>5.  Press on a <em>Deploy &amp; Run a Transaction</em> button</p><figure class="kg-card kg-image-card"><img src="https://lh3.googleusercontent.com/cNdKvmMPeZ_iAAJqsRQlDGiEjXnPuXMoSiYpJNgForBl0kNrwfJsoydwTgjjkaOc6cK-Gsd63BlXL5sPgIPssuoeU4vTdnYnp_7Mjrm91ZWDHAGVxDsyPgstcNfqwikGPSqermFp" class="kg-image" alt></figure><p>6.  In the <em>Environment</em> drop-down field select <em>Injected Web3. </em>It should cause automatic connection to MetaMask extension</p><figure class="kg-card kg-image-card"><img src="https://lh5.googleusercontent.com/jXoXqtUWd2Wc20yKZToqCBumxLJmoUGmNufRPn4W_Isu2F9fosV4LYTaOyqzAj1m_E7DidgtJD1lPItlbi8m5SkeQUGkjRMsJHpeNXaL4Am9tRppwS7-fwEEuekJ5KaqJrQHed-3" class="kg-image" alt></figure><p>7.  Paste vesting contract address from email that was sent by The Graph team. Press <em>At Address</em></p><figure class="kg-card kg-image-card"><img src="https://lh6.googleusercontent.com/TVCJOSfer3NfjH9txEIg5jGDrJvaAMHkBQesuEcloRDLPL2W1uVG1IWDqGpObRXZUGKbfvZ3DNpF62aft4k_pcLXKqQA1mVWGp6a5I5ifonqcDKj2dT2PRmMm4EnWj-NXZrJDtex" class="kg-image" alt></figure><p>8.  Click on a <em>Deploy Contract arrow</em> to expand the drop-down menu</p><figure class="kg-card kg-image-card"><img src="https://lh3.googleusercontent.com/BxV18U5PUuikOFP-U3N_Pkpx6dEr552c2dqKcHf9ebP4uA5DcuMNRRSjCwnTHqOm2aafkKJnnjHEw-ndiUZl4SwPH4QpgSLm0Pu0H0gFfw8exDsYZyCaomDuvOuVt4MXEB410SKF" class="kg-image" alt></figure><p>9.  Click on <em>currentBalance</em> to check if it matches with the one from the email. If balance is correct press <em>approveProtocol. </em><strong><em>C</em>onfirm the transaction</strong></p><figure class="kg-card kg-image-card"><img src="https://lh5.googleusercontent.com/coyF3BSuofW8efgvr8lkB9L_5xAjhJeusvit-FjyLZxbZvqis-f73f0aoRjO3syB62G9koW2JPSJARxPeJOHr5wB51ZJA5lXSGT6e73RforKeIp0BK4pxph0DL6ancPyk-fQTRmi" class="kg-image" alt></figure><p>10.  Copy the balance to proceed with delegation</p><figure class="kg-card kg-image-card"><img src="https://lh4.googleusercontent.com/9t8Lm9LHLThfgDdKd16g4CNsyXx5OKShCKYThvZZB63m1y6MEMd9qjU8gzMuRsT4UbpL_l2s9EPiGwU5TqR3aySvr9Oma0zV-TLy5QWIrnr9KthDMYok3VS7Pl8UtmYHvWqOzT9F" class="kg-image" alt></figure><p>11.  Press on the <em>Delegate</em> <em>arrow</em> to expand the section</p><figure class="kg-card kg-image-card"><img src="https://lh4.googleusercontent.com/NKXvNbArlEBrPPlfaN0070G1KT0zEGNFCLnNAVTHUmd833CSOnjEQs18zoJvnneAxXpu-K0q02NwDlRJYDVhU6RhbcqMJyk_7aC0GapwBdyJ1Safe7Wk0wfmfZwkY5soQQTbltbj" class="kg-image" alt></figure><p>12.  Paste the balance you copied to the <em>tokens </em>field. <strong>P2P has two indexers</strong> available for delegation:<br><br><strong>a) </strong>Indexer address: <strong>0x5a8904be09625965d9aec4bffd30d853438a053e</strong><br>Max capacity:<strong> 533M GRT</strong><br>Fee:<strong> 8% </strong><br><br><strong>b)</strong> Indexer address: <strong>0xf4a097ce3a4efbd1748b2ef2076813961e4e6fa7</strong><br>Max capacity:<strong> 17M GRT</strong><br>Fee:<strong> 8%</strong></p><p>Paste the chosen indexer address in the <em>indexer </em>field and press <em>transact </em>button. Confirm the transaction. You will be required to pay 0,5% GRT deposit tax</p><figure class="kg-card kg-image-card"><img src="https://lh5.googleusercontent.com/5YkXxRZbCoovWL51VKKX8nfO4TNImtGpPzKbT32VedjWYi7qvAz7HFtye3yZnUycJhhMlhcgENCt-I2KOHxN50XGp0bVEyF-NHnq-TAbK5Ubxypf4tRwPjDES_KKXFKB0SGJlmxZ" class="kg-image" alt></figure><p>Rewards are automatically added to staked deposit and paid when indexer closes allocation. There is no slashing for delegators. Unbonding period lasts 28 days.</p><hr><p><em>If you have any questions, contact @Aleksei_W in Telegram or via email [email protected]</em></p><hr><h2 id="about-p2p-validator">About P2P Validator</h2><p>P2P Validator is a world-leading staking provider with the best industry security practices and proven expertise. We provide comprehensive due-diligence of digital assets and offer only top-notch staking opportunities. At the time of publishing, more than 350 million of USD value is staked with P2P Validator by over 4000 delegators across 15+ networks. We are early The Graph investors who participated in testnet from the day one with a goal to provide long term support for the ecosystem. We operate the biggest indexer globally with highest own stake of 33M GRT and available capacity of &gt; 500M GRT making it ideal choice for large GRT holders.</p><hr><ul><li><strong>Web:</strong><a href="https://p2p.org/?utm_source=blog&amp;utm_campaign=flow_annoncement"> https://p2p.org</a></li><li><strong>Twitter:</strong><a href="https://twitter.com/p2pvalidator"> @p2pvalidator</a></li><li><strong>Telegram:</strong><a href="https://t.me/P2Pstaking"><strong> </strong>https://t.me/P2Pstaking</a></li></ul>

Alex Bond

from p2p validator

TheGraph The Graph Delegation Guide For Locked & Unlocked GRT

<p><em>Use this simple step-by-step guide to delegate either locked and unlocked GRT to receive a share of staking rewards. If your funds are managed by multi-sig you may contact directly @Aleksei_W in telegram or via email [email protected] for personal assistance.</em></p><ol><li>Go to<a href="https://network.thegraph.com/"> https://network.thegraph.com</a>.</li><li>Connect your wallet using MetaMask.</li></ol><figure class="kg-card kg-image-card"><img src="https://economy.p2p.org/content/images/2021/02/connect-wallet.PNG" class="kg-image" alt srcset="https://economy.p2p.org/content/images/size/w600/2021/02/connect-wallet.PNG 600w, https://economy.p2p.org/content/images/size/w1000/2021/02/connect-wallet.PNG 1000w, https://economy.p2p.org/content/images/size/w1600/2021/02/connect-wallet.PNG 1600w, https://economy.p2p.org/content/images/2021/02/connect-wallet.PNG 1902w" sizes="(min-width: 720px) 720px"></figure><p>If you have tokens locked with a vesting contract press an arrow and navigate to in in the UI. If your GRT are not vested, skip this step.</p><figure class="kg-card kg-image-card"><img src="https://economy.p2p.org/content/images/2021/02/--------------2021-02-03---20.07.05.png" class="kg-image" alt srcset="https://economy.p2p.org/content/images/size/w600/2021/02/--------------2021-02-03---20.07.05.png 600w, https://economy.p2p.org/content/images/size/w1000/2021/02/--------------2021-02-03---20.07.05.png 1000w, https://economy.p2p.org/content/images/size/w1600/2021/02/--------------2021-02-03---20.07.05.png 1600w, https://economy.p2p.org/content/images/size/w2400/2021/02/--------------2021-02-03---20.07.05.png 2400w" sizes="(min-width: 720px) 720px"></figure><p>3.  Choose an indexer. P2P has launched two indexers. Select our indexer <strong>0x5a89</strong> from the list in the beginning or paste <strong>0xf4a0 </strong>in the search field to select the second one. <br><br><strong>a) </strong>Indexer address: <strong>0x5a8904be09625965d9aec4bffd30d853438a053e</strong><br>Max capacity:<strong> 600M GRT</strong><br>Fee:<strong> 8% </strong><br><br><strong>b)</strong> Indexer address: <strong>0xf4a097ce3a4efbd1748b2ef2076813961e4e6fa7</strong><br>Max capacity:<strong> 17M GRT</strong><br>Fee:<strong> 8%</strong><br><br>Press on the blue button to proceed and select <strong>Delegate.</strong></p><figure class="kg-card kg-image-card"><img src="https://economy.p2p.org/content/images/2020/12/scroll-down.PNG" class="kg-image" alt srcset="https://economy.p2p.org/content/images/size/w600/2020/12/scroll-down.PNG 600w, https://economy.p2p.org/content/images/size/w1000/2020/12/scroll-down.PNG 1000w, https://economy.p2p.org/content/images/size/w1600/2020/12/scroll-down.PNG 1600w, https://economy.p2p.org/content/images/2020/12/scroll-down.PNG 1903w" sizes="(min-width: 720px) 720px"></figure><p>4. In the opened window specify the amount you wish to delegate. Press <strong>Submit Transaction</strong>. You will be required to pay 0.5% GRT deposit tax.</p><figure class="kg-card kg-image-card"><img src="https://economy.p2p.org/content/images/2020/12/input-amount-1.PNG" class="kg-image" alt srcset="https://economy.p2p.org/content/images/size/w600/2020/12/input-amount-1.PNG 600w, https://economy.p2p.org/content/images/size/w1000/2020/12/input-amount-1.PNG 1000w, https://economy.p2p.org/content/images/size/w1600/2020/12/input-amount-1.PNG 1600w, https://economy.p2p.org/content/images/2020/12/input-amount-1.PNG 1893w" sizes="(min-width: 720px) 720px"></figure><p>5. Approve the transaction using MetaMask. First, allow interaction with your wallet and second, confirm the delegation. You will need to have some ETH to pay the transaction fee.</p><p>6. After a successful confirmation you can observe information about your delegation by pressing on your account in the right top corner</p><figure class="kg-card kg-image-card"><img src="https://economy.p2p.org/content/images/2020/12/my-wallet.PNG" class="kg-image" alt srcset="https://economy.p2p.org/content/images/size/w600/2020/12/my-wallet.PNG 600w, https://economy.p2p.org/content/images/size/w1000/2020/12/my-wallet.PNG 1000w, https://economy.p2p.org/content/images/size/w1600/2020/12/my-wallet.PNG 1600w, https://economy.p2p.org/content/images/2020/12/my-wallet.PNG 1908w" sizes="(min-width: 720px) 720px"></figure><p>7.  Rewards are automatically added to staked deposit and paid when the indexer closes allocation. There is no slashing for delegators on The Graph. </p><p>The Graph GRT unbonding period lasts 28 days.</p><figure class="kg-card kg-image-card"><img src="https://lh5.googleusercontent.com/LnKQ69DE9wFPNdY53-Liog91UcZZyU0MmTgNkmwFEAlYrvoHxw1EdqYVtyMGsC_RAj2KQVlth7S14HD5_erh0DE9fUwLNZeXduXgIeTqMVbYM3H5y0TLT3-IShUtPMytallp0mIY" class="kg-image" alt></figure><hr><p><em>If you have any questions, feel free to ask in our <a href="https://t.me/P2Pstaking">Telegram chat</a> on contact Alex via email [email protected]</em></p><hr><h2 id="about-p2p-validator">About P2P Validator</h2><p>P2P Validator is a world-leading staking provider with the best industry security practices and proven expertise. We provide comprehensive due-diligence of digital assets and offer only top-notch staking opportunities. At the time of publishing, more than 1 billion of USD value is staked with P2P Validator by over 4000 delegators across 15+ networks. We are early The Graph investors who participated in testnet from the day one with a goal to provide long term support for the ecosystem. We operate the biggest indexer globally with highest own stake of 33M GRT and available capacity of &gt; 500M GRT making it ideal choice for large GRT holders.</p><hr><ul><li><strong>Web:</strong><a href="https://p2p.org/?utm_source=blog&amp;utm_campaign=flow_annoncement"> https://p2p.org</a></li><li><strong>Twitter:</strong><a href="https://twitter.com/p2pvalidator"> @p2pvalidator</a></li><li><strong>Telegram:</strong><a href="https://t.me/P2Pstaking"><strong> </strong>https://t.me/P2Pstaking</a></li></ul>

Alex Bond

from p2p validator