# Spadina rehearsal launch retrospective notes ## Kudos - Spadina launch call was running well 👍 - Beaconcha.in's genesis countdown feature is awesome - Prysm responded quickly on the release process issue below. Spadina is getting recovery. (~63.50% participant rate at epoch 40) ## Issues ### [Client] Release process failure - Prysm/Terence: (for [peer connection issue](https://github.com/prysmaticlabs/prysm/issues/7372)) This is more on our release process. All the well known issues were discovered in alpha-v26 and fixed in latest master. We missed releasing alpha-v27 last night which would have contained all the Spadina updates. We now are helping the stakers in our discord to get alpha-v26 working with Spadina, It involves using 2 additional CLI flags. (See Nishant’s announcement) - The detailed report: [No finality at Spadina testnet launch (Incident #11) ](https://docs.google.com/document/d/1pn9giIODBUCpgImZRPR34RbF-jAp1W4xTT_8gdcKFCw/edit) ### [Client] New release announcement communication - Dankrad/Carl: Many teku users didn't know there was [a critical bugfix within 24 hours before the launch](https://github.com/PegaSysEng/teku/pull/2863). ### [Client] Issues of running nodes for `>=2` testnets on the same machine - Justin: to get Lighthouse working I needed to manually specify a datadir (separate from the Medalla one) - Prysm/Terence: Some of them are having issues with docker where running both Medalla and Spadina via docker compose ### [Infrastucture] Beaconcha.in - spadina.beaconcha.in was down for a while (at ~1PM UTC). But they fixed it soon. ### [Infrastucture] Launchpad - [Prysm's instruction link is pointing to Medalla instruction](https://github.com/ethereum/eth2.0-deposit/issues/179#issuecomment-700679181) so the users wrongly used Medalla config when they wanted to run on Spadina ### The users forgot/ignored the launch date? - No incentives on testnets ## Improvement suggestions - Danny: - Let's do another dress rehearsal - To notify users in time, clients can maintain opt in email lists for critical updates - Set up eth2stats next time so that we can have better visibility of peer connections. - Hsiao-Wei: - Improve communication. Use EF blog to announce the client release versions 3 days before the launch. - Concur [Pico on ethstaker discord](https://discordapp.com/channels/694822223575384095/738442543477555280/760488951030218832), we can have a separate staker checklist on the launchpad so that the user can come back to review it after the client software is ready. - Ask client teams to provide mainnet landing page URL/the release announcement center URL for launchpad to link to. - Carl: Add a static "preparing for genesis" page on launchpad - Terence: I think `GENESIS_DELAY` at 2 days is too short. Say we discover at bug at genesis, there's 2 days to fix, test and release. Ideally we will like more time to test and verify it doesnt break existing testnet (e.g. medella) - Afri: Let's talk about release management/release strategies! - Client teams should maintain stable release branches that: - only recieve bug fixes - don't recieve new features - don't get massively refactored - For new features/refactoring/etc. you should use beta/alpha/nightly releases. - End users in most cases want _stable_ releases, especially on mainnet or pre-launch testnets. - We had similar issues prior to Medalla launch, the time to introduce stable releases is now. - I'm happy to consult on different release strategies.