# Ropsten TTD Postmortem [toc] ## TL;DR: * Ropsten hash rate surged overnight, causing it to hit TTD before the Ropsten Beacon Chain Genesis * Because most users had not upgraded their nodes yet, it was decided to increase the TTD on Ropsten to an incredibly large number for now: `100000000000000000000000` (~2x mainnet total difficulty) * **Client teams will put out a release in the next few days with the new value and tell users which have previously updated to the Ropsten TTD release to manually override TTD to `100000000000000000000000`.** * Document explaining how to do so for each client: https://hackmd.io/ngKLqVvvTTGZLj1bGPuCoA * As part of the Ropsten fork announcement, we will tell users to expect to override the high TTD to a value which will be chosen after the Bellatrix fork has happened on Ropsten. * This issue was caused by very low hash rate on PoW testnets and the Ropsten Beacon Chain not being live at the time TTD was chosen, two things which are not the case on mainnet. * One potential concern for mainnet is hitting TTD before the Bellatrix slot height on the Beacon Chain. While this is unlikely given that mainnet total difficulty is easier to predict, the same strategy of first setting a very large TTD and then lowering it after Bellatrix would work. ## Action Items - [x] Run a TTD override on Genesis Ropsten Validators to adjust the TTD to `100000000000000000000000` - [ ] [IN PROGRESS] New client released using a TTD of `100000000000000000000000` for Ropsten - [ ] [[IN PROGRESS]](https://hackmd.io/ngKLqVvvTTGZLj1bGPuCoA) Provide documentation for users about TTD override - [ ] Add tests cases for TTD happening before Bellatrix is activated - [ ] Launch Sepolia Beacon Chain "ASAP" ## Ropsten TTD Issue Given the low hash rate on Ropsten (and Sepolia), it is cheap for someone to increase it. On May 26th, someone did that and 20x'd the hash rate, see this chart from https://bordel.wtf:  We estimate doing so costs ~X00$/day, so costs to affect hash rate even another 10-100x are relatively low. This means that choosing a new TTD value based on this current hash rate was similarly unsafe. Coversely, if we did that, Ropsten miners could simply stop, meaning our TTD would now be too far out. The Ropsten miner had also reduced the gas limit on the network to [~21000 gas](https://ropsten.etherscan.io/block/12308106), so it was reasonable to expect they were purposefully trying to disrupt things, or, in their words, ["STRESS TESTING ROPSTEN NETWORK TO SEE WHAT WOULD HAPPEN IF THE MERGE HAPPENED OUT OF SCHEDULE"](https://ropsten.etherscan.io/tx/0xc3649f78b11b21fac66a2cc6aed59fa3171798cad351d5009321368ebdd80751). Instead of hitting TTD around June 8th, it was hit on May 26 at block [12308106](https://ropsten.etherscan.io/block/12308106). ## Transition Issues The TTD was hit before (1) the Ropsten Beacon Chain was live (ETA May 30) and (2) the Bellatrix fork had been activated on it (ETA June 3). We could never hit **(1)** on mainnet given the Beacon Chain is already live, and **(2)** is easy to prevent given the high hash rate on the network and PoW security assumptions (if someone can >2x the hash rate, they can 51% attack the network). In practice, clients "should" be able to handle this early TTD scenario the following way: 1. Upgraded EL clients stall when TTD is hit 2. The Beacon Chain launches 3. Bellatrix is hit 4. At the same epoch as (3), The Merge happens That said, that would have caused downtime on Ropsten between (1) and (4), and with miners staying on the old chain + the majority of nodes not having updated prior to TTD, the PoW chain would continue and been seen as a "default" by most of the community. This may also reveal weird bugs/edge cases in client combinations that likely aren't applicable to mainnet. ## TTD Override Solution To fix this issue, a group of client developers, researchers and test engineers agreed to update the TTD for the Ropsten fork to an incredibly large number and put out releases with this new TTD, instructing users to be ready to perform a manual TTD override ~1 week after. | TTD | Value | ETA | | -------- | -------- | -------- | | Previous | `43531756765713534` | May 26, 2022, was planned for June 8, 2022 | | New (large) | `100000000000000000000000` | ~250 years | TBD | TBD | ~June 8, 2022, specific value to be chosen after Bellatrix activation on June 3, 2022 | A new TTD value will be chosen and communicated once Bellatrix has been hit on the Ropsten Beacon Chain. Users will then need to perform a manual TTD override to set this value. Users who have already upgraded to client versions with the current TTD (`43531756765713534`) must therefore do two overrides: - ASAP, to the new, large, value (`100000000000000000000000`); * After Bellatrix, to a value we estimate will be hit around June 8 Instructions for how to perform these for various clients can be found here: https://hackmd.io/ngKLqVvvTTGZLj1bGPuCoA ## Can This Happen on Mainnet? In short, not really. For The Merge to happen, the following sequence of events needs to take place: 1. The Beacon Chain is Launched 2. The Bellatrix upgrade is activated on the Beacon Chain 3. A Terminal Total Difficulty (TTD) is hit on the proof-of-work chain **(1)** has happened on mainnet a long time ago, but hasn't on Ropsten or Sepolia yet. We need the TTD to be chosen for **(2)**, as "Bellatrix" is when CL clients start listenning for the TTD to be hit on the EL. Given that the hash rate is relatively stable on mainnet, it is easy to estimate a TTD value we will not hit before Bellatrix is activated on the CL. If a miner wanted to hit this TTD earlier, they would need to contribute a significant extra share of hash rate to the network, which is costly. For it to happen 2x quicker than expected, for example, this would require doubling the hash rate. At that point, a miner could simply 51% attack Ethereum. It is more realistic that, due to The Merge approaching, hash rate on the network goes down. In that case, it means the TTD would be hit later than expected. This is because, as hash rate drops, the additional difficulty contributed by each block goes down, and the number of blocks required to reach a specific TTD increases. That said, if we want to prevent this from even being a possibility, a similar process as will be used for Ropsten could be used on mainnet. First, we would release a version of the EL and CL clients with an exceedingly high TTD which would take years to hit. Then, once the Bellatrix upgrade is activated on the Beacon Chain, a new TTD value can be chosen and users can use that. This would reduce the risk that the TTD is hit too early, at the cost of greater coordination overhead. ## Timeline All times in PDT ### May 25 * 20:15: [first mention](https://discord.com/channels/595666850260713488/692062809701482577/979221237568004136) on the R&D discord of hash rate increasing on Ropsten ### May 26 * 1:00-3:00: confirmation on R&D discord that hash rate has increased >30x on Ropsten * 7:00: zoom call started with client/testing/research teams to discuss potential mitigations * 7:36: tentative decision to increase TTD and have users manually override to a lower value later shared in [#allcoredevs](https://discord.com/channels/595666850260713488/745077610685661265/979392525481681006)  * 8:38: [rough consensus](https://discord.com/channels/595666850260713488/745077610685661265/979408221800382474) is reached amongst client teams to follow this strategy  * 9:12: Genesis validators controlled by EF + client teams updated to use new TTD value * 9:19: [Tweet by Marius](https://twitter.com/vdWijden/status/1529859778129842176) letting the community know this will happen  * 11:29: [TTD Hit](https://ropsten.etherscan.io/block/12308106) on Ropsten