# Rayonism - Meta-Spec ☀️ To implement a testnet with Merge and Sharding, with multiple clients we need a spec. Although the eth2.0-specs repo details many parts of the Merge and Sharding features today, it is not an appropriate place for hackathon specific changes, nor will the Merge specification necessarily stabilize during the hackathon implementation work. Instead, we maintain a meta-spec: - Pin Merge features for implementation - Quick iteration on new features or hackathon-specific modifications - Merge into upstream spec as it stabilizes - Notes + coordination point for implementers The meta-spec will live in this HackMD until the hackathon starts and then move into a GitHub repository. # **Latest merge spec here :arrow_right: : [`github.com/ethereum/rayonism`](https://github.com/ethereum/rayonism/blob/master/specs/merge.md)** ## Table of Contents [TOC] ---- ## Updates - Wed 7 April: Rayonism early-bird call ([notes](https://notes.ethereum.org/W-abMNqMTtexfiLwY1Y7rQ)). Mikhail publishes [Rayonism merge spec](https://notes.ethereum.org/@n0ble/rayonism-the-merge-spec) - Fri 9 April: Office hours nr 0: `09:00 UTC Fri April 9` ([quick notes](https://gist.github.com/protolambda/56bae574795e6c29dae49d3bfeac9a7e)) - Sun 11 April: [New Devnet setup guide](https://notes.ethereum.org/@protolambda/merge-devnet-setup-guide) - Mon 12 April: Office hours nr 1: `17:00 UTC Mon April 12` - Thu 15 April: Office hours nr 2: `01:00 UTC Thu April 15`. Merge implementers call ([Notes](https://notes.ethereum.org/@protolambda/merge_impl_call_2)) - Fri 16 April: Office hours nr 3: `09:00 UTC Fri April 16` ([Notes](https://notes.ethereum.org/@protolambda/BkrtiC8Iu)) - Sun 18 April: [Mergenet tutorial](https://github.com/protolambda/mergenet-tutorial), setup your own local testnet! - Mon 19 April: Office hours nr 4: `17:00 UTC Fri April 19` ([Notes](https://gist.github.com/protolambda/e61192c63facfffd0a54d515d5f978b4)). Rayonism Merge Spec v0.1.2 - Fri 23 April: Office hours nr 5: `09:00 Fri April 23` ([Notes](https://gist.github.com/protolambda/ee21b07918aa16bbc0e02f7dafa2d102)) - Mon 26 April: Office hour nr 6: `17:00 Mon April 26` ([Notes](https://gist.github.com/protolambda/d99dbe49dc3b89a254053cc1d6abb8dd)) - Thu 29 April: Planned 1-day devnet setup day (delayed because mainnet incident response work) - Fri 30 April: Planned 1-day devnet launch day (configuration to be published around Wed.) Office hours nr 7: `09:00 Fri April 30` ## Plan General picture, up for discussion (**disclaimer**: nothing confirmed, this is unstable): - **29 Mar - 16 Apr**: Hackathon preparation - Continue developing Sharding prototype spec - Proto, Danny, others - Continue Merge work, update teku/catalyst reference implementations to match latest specification around Hackathon start - Mikhail, Guillaume, others - Coordinate with Hackathon contributors - Get involved! We'll update you with all the details so you can prepare for the hackathon. For now: read up on specs, ask questions, and prepare for work in your domain - **Week 1: 19 Apr - 24 Apr**: Hackathon starting phase - *Merge* - Run short-lived public Merge testnet (No transition, EVM at genesis) - Update: Delayed to 30 Apr after eth2 mainnet incident response. - Teku-Catalyst to start, possibly others - Update: Nethermind joined initial devnet work - *Sharding* - Discuss designs and plans with with Eth2 Merge core contributors - Schedule hackathon calls - **Week 2: 26 Apr - 30 Apr**: Hackathon BUIDL phase - *Merge* - Iterate on and expand Merge implementations (minimal RPC and forkchoice changes) in Eth1 clients - *Sharding* - Work on sharding requirements and general implementation approach - Prototype between Merge and Sharding code in Eth2 clients - Update: Teku is making progress here - **Week 3: 3 May - 7 May**: Hackathon iteration phase - *Merge* - Another new Merge devnet - One-week devnet target, launch TBD (unstable) - Plug in more eth1 and eth2 clients as they become available - Update: Nimbus and Open Ethereum might join - Monitor load and other interesting metrics - Experiment with live PoW -> PoS transition and/or large states - *Sharding* - Small prototype sharding devnets - Sharding-application API prototypes - **Week 4: 10 May - 14 May**: Hackathon final phase - *Merge+Sharding* - Short-lived Sharding-Merge devnet(s), experimental, hackathon product! - L2 discussions/engineering around shard data - Stretch goal: hack Optimism data-transport-layer to utilize shard data API - **Post-Hackathon**: - Public Short-lived Merge-Sharding devnet - Further L2 work, more L1 devnet deployments - Merge and Sharding results feed back into specs and roadmap! --- ## Eth1 Clients ### Catalyst Catalyst is a fork of Go-Ethereum by Guillaume that is the first Eth1 client to add a set of RPC methods for the Merge: - Expose the fork-choice to an Eth2 beacon node. - Produce blocks on demand for beacon proposers - Add finality info to the RPC for application users The Eth1 block history, state DB, transaction pool, application RPC are all unchanged. Catalyst GitHub: https://github.com/gballet/go-ethereum/tree/catalyst-for-executable-beacon-chain Catalyst Setup scripts by Paul, to run with Lighthouse: https://github.com/paulhauner/merge-testnet Also see docker compose setup with Teku. ### Additional clients More Eth1 clients are joining the hackathon. We'll add links and info as they come online. Reach out when you have some info to share! --- ## Eth2 Clients ### Teku fork Teku TXRX GitHub: https://github.com/txrx-research/teku/tree/executable-beacon-chain Docker compose demo, using Catalyst for Eth1: https://github.com/txrx-research/executable-beacon-chain-localnet ### Lighthouse fork Lighthouse GitHub: https://github.com/paulhauner/lighthouse/tree/merge-v1 ### Prysm fork Prysm quick-merge-demo: https://hackmd.io/@prysmaticlabs/prysm-catalyst-interop ### Stand-alone Sharding node (new) In addition to modifying existing Eth2 clients, a new type of node will be prototyped: to serve L2 applications (incl KZG shard blob proof data) and handle bulk shard data. @protolambda will be building this. Stack: Golang, go-kzg, go-libp2p. GitHub: Soon^:TM: ### Additional clients More Eth2 clients are joining the hackathon. We'll add links and info as they come online. Reach out when you have some info to share! --- ## Merge meta-specs (Eth1) The Eth1 **node** is transformed into an Eth1 **engine**: the transaction pool and sync stays for the time being, but consensus is maintained by an Eth2 node that informs the engine of new blocks and chain reorganization over a local RPC. ### Forkchoice / Chain management Primary new functionality: insert a block, without PoW seal verification. [Catalyst Example](https://github.com/gballet/go-ethereum/blob/3ded09cd6ba2ac286f42ce05e7ffcc19c7f1fb8b/core/blockchain.go#L1628) The Eth2 client is expected to insert blocks in order, and explicitly change the head when re-orging. ### Networking changes - Disable block propagation ([Geth `ProtocolManager.BroadcastBlock`](https://github.com/gballet/go-ethereum/blob/3ded09cd6ba2ac286f42ce05e7ffcc19c7f1fb8b/eth/handler.go#L827)) - Disable block processing from dev-p2p ([Geth `ProtocolManager.handleMsg`](https://github.com/gballet/go-ethereum/blob/3ded09cd6ba2ac286f42ce05e7ffcc19c7f1fb8b/eth/handler.go#L712)) ### RPC The current proposed local RPC for Eth2-Eth1 communication is spec'd as JSON-RPC. This RPC has changed several times and is still being minimized to keep the Eth1-Eth2 merge changes as simple as possible. There is some discussion to instead run the Eth2-Eth1 communication on a separate simplified REST HTTP endpoint, to enable different security (any HTTP authentication proxy) and explicitly never have things like query sessions. NEW Rayonism spec: https://notes.ethereum.org/@n0ble/rayonism-the-merge-spec _{OLD DRAFT RPC spec: https://hackmd.io/@n0ble/eth1-eth2-communication-protocol-draft} The general extended idea is that the Eth2 client inserts blocks into the Eth1 client, along with `BeaconChainContext` data (beacon block roots, beacon state roots, randao mix, whatever the chosen new opcodes need). **The current spec minimizes this data, no opcode support yet** The Eth1 client skips PoW verification, and extends the chain with the new block. And the head may be changed by the Eth2 node on a re-org. - [Teku Eth2 RPC client bindings](https://github.com/txrx-research/teku/tree/executable-beacon-chain/ethereum/exec/src/main/java/tech/pegasys/teku/exec/eth1engine) - [Lighthouse Eth2 RPC client bindings](https://github.com/paulhauner/lighthouse/blob/merge-v1/beacon_node/eth1/src/http.rs) - [Catalyst RPC server implementation](https://github.com/gballet/go-ethereum/blob/catalyst-for-executable-beacon-chain/eth/eth2.go) Additionally, clients can disable/remove the `miner` RPC namespace when running in PoS mode. ### `BEACONBLOCKROOT` opcode Not implemented in Catalyst; still under consideration for a more minimal merge. This is enough to prove a state root that is embedded in a `BeaconBlockHeader` (no signature), of which the SSZ-hash-tree-root matches the `BEACONBLOCKROOT`. It works like `BEACONSTATEROOT` (see below), fetching recent history, up to the current block slot (excl.). Related work: [Eth2 withdrawal simulation](https://github.com/zilm13/withdrawal-simulation) ### Alternative/extra opcodes *These are experimental opcodes, and not part of the most minimal version of the merge.* Catalyst: - [Opcode registration](https://github.com/gballet/go-ethereum/blob/3ded09cd6ba2ac286f42ce05e7ffcc19c7f1fb8b/core/vm/jump_table.go#L1022) - [Implementation](https://github.com/gballet/go-ethereum/blob/3ded09cd6ba2ac286f42ce05e7ffcc19c7f1fb8b/core/vm/instructions.go#L483) Opcodes: - `RANDAOMIX`: get randomness from the beacon chain. Pushes to current randao mix (of same block) onto the stack memory. - `BEACONSTATEROOT`: instead of a merkle proof to the state root in the block header that matches `BEACON_BLOCK_ROOT`, a state root could be exposed directly. Reads `stack 0` as big-endian int argument (like any `uint256` in EVM), must be `< 256`, and fetches the post state-root at slot `current - i -1`. Extra: `SLOT` opcode. Can be derived from timestamp already, but may avoid user errors when interacting with recent state/block roots. --- ## Merge meta-specs (Eth2) The beacon-node specification is updated to include an `ApplicationPayload` in the block data along with a call to validate this payload in the state transition function. Current Eth2 Merge specs: - [Beacon Chain changes](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/merge/beacon-chain.md) - [Fork Choice changes](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/merge/fork-choice.md) (for transition process, can ignore if Merge starts at chain genesis) - [Validator changes](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/merge/validator.md) ### BeaconBlock ApplicationPayload The `ApplicationPayload` is an SSZ type, embedded in the `BeaconBlockBody`, and was introduced here: https://github.com/ethereum/eth2.0-specs/pull/2257 Note: the Teku-Merge implementation implemented this differently before. The spec'd `ApplicationPayload` has changed since. This should align again before the hackathon. ### Block validation After initial beacon block processing, the `ApplicationPayload` contents are inserted into the Eth1 node via RPC. If not valid, the RPC returns an error, and the validation rejects the input. This process is defined in the Eth2 merge specs. ### Block proposals Validators make requests to the Eth1 node for an `ApplicationPayload`. This is similar mechanics to producing a payload for miners today. The [validator guide](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/merge/validator.md) in the eth2.0-specs repo describes a minimal version of this. ---- ## Sharding meta-specs (Eth1) Sharding is mostly an addition on the Eth2 side, with the exception of a new KZG precompile. ### KZG pre-compile The `BEACONBLOCKROOT` and `BEACONSTATEROOT` only enable merkle proofs over the SSZ beacon state (static binary tree composition). To verify shard data in the EVM, KZG is required. KZG fits shard-data especially well because of the fixed format: a single KZG proof (only 48 bytes) is enough to proof any subset of points (just under 32 bytes per point) in the whole shard blob (~1/4 MB). The verification can be a pure function that is exposed in the EVM, similar to a hash function, or things like the `ecrecover` (0x01) pre-compile. The idea is to implement it as pre-compile (or maybe opcode) and enable dapps to verify shard data, after establising a trusted shard blob commitment via a regular merkle proof. More details: [Kate Polynomial commitments](https://dankradfeist.de/ethereum/2020/06/16/kate-polynomial-commitments.html), post by Dankrad KZG implementations: - Go (Optimized, Kilic-BLS, Herumi BLS): https://github.com/protolambda/go-kzg - C (Optimized, BLST): https://github.com/benjaminion/c-kzg - Rust (Unoptimized, BLST): https://github.com/ralexstokes/kzg/ - Nim (Optimized, Constantine): https://github.com/mratsim/constantine/tree/master/research/kzg_poly_commit --- ## Sharding meta-specs (Eth2) Very experimental. See [eth2.0-specs PR with modularized phase1, including updated sharding spec](https://github.com/ethereum/eth2.0-specs/pull/2254) ---- ## Hacking TODO list: Merge: - Decide on any opcodes in EVM: Catalyst is out of sync with latest proposals. `BEACONBLOCKROOT`? Do we make things easier for users with a `BEACONSTATEROOT`? - Maybe time to update Catalyst to match Berlin fork - Clarify latest RPC state: one method was unused (still necessary?), RPC spec draft updated yet? - Team updates, Eth1/Eth2 teams are at varying stages of implementation - Testnet planning Sharding: - Shard node prototype: - Purely interfacing with beacon API, or also active on LibP2P to serve data? - Spec validator API changes: - AttestationData will also vote for ShardHeader root (not the commitment) - BeaconBlocks to sign over will contain a list of pending shard headers - Network changes: - Join global headers topic - Join shard blob subnet as attester - ReqResp: altruistic shard data sync for hackathon purposes? - Block proposals: - Hackathon solution to replace fee market for shard data. - Easy: central server to bridge dapp users who want to suggest data to proposers - Difficult: additional gossipsub topic for txpool of shard blobs with fee payments - Shard blob storage: - Beacon node should store blobs, so other nodes can sync from it - Extra: Validator Client stores blobs, to prepare for future custody-game - KZG team updates: - Every Eth1 client will need a minimal version of it: `KZG Verify` for the precompile - Every Eth2 beacon node will need a medium version of it: extend shard input data, commit with KZG, validate gossip input (field elements within modulus, compute commitment) - Every User-facing Eth2 beacon API could use a proof-generation endpoint. Maybe shard-node specific?