# Shadow fork tracker NOTE!: Ideally keep this doc internal :) ## mainnet-shadow-fork-8 #### Proposal tracker: | EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? | |--------| -------- | --------------- | ---------- |----------- | ------------------------- | |Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Besu |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ❌ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Erigon |Lighthouse| ? | ? | ? | ? | | |Teku | ✅ | ✅ | ✅ | ? | | |Nimbus | ✅ | ✅ | ✅ | ? | | |Lodestar | ? | ? | ? | ? | | |Prysm | ? | ? | ? | ? | Note: Erigon had sync issues late last week, its been fixed now but the nodes are still syncing to head. They still need to sync ~900k blocks at the time of creating this chart, it'll be updated once its at head. #### Issues: - nimbus-besu-2 seems to be out of sync. nimbus-besu-1 is fine, so it doesn't seem to be an issue with the pair. - Besu nodes are proposing blocks with 0 transactions, likely a regression relating to timeout ## mainnet-shadow-fork-7 #### Proposal tracker: | EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? | |--------| -------- | --------------- | ---------- |----------- | ------------------------- | |Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Besu |Lighthouse| ❌ | ❌ | ❌ | ❌ | | |Teku | ❌ | ❌ | ❌ | ❌ | | |Nimbus | ✅ | ✅ | ✅ | ❌ | | |Lodestar | ✅ | ✅ | ✅ | ❌ | | |Prysm | ✅ | ✅ | ✅ | ❌ | |Erigon |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ❌ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | #### Issues: - Erigon had issues with peers not having the state and dropping out of sync - Besu had a world state issue ## ropsten #### Proposal tracker: | EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? | |--------| -------- | --------------- | ---------- |----------- | ------------------------- | |Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Besu |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ❌ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Erigon |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ? | ? | ? | ? | | |Prysm | ✅ | ✅ | ✅ | ✅ | #### Issues: - Besu-Nimbus incompatibility (probably limited to ws being used by Besu, EF nodes are fine) - Nethermind concurrency bug ## mainnet-shadow-fork-6 #### Verisons used: - Same as those recommended in the EF Ropsten blogpost #### Proposal tracker: | EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? | |--------| -------- | --------------- | ---------- |----------- | ------------------------- | |Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ❌ | | |Lodestar | ✅ | ✅ | ✅ | ✅ (some are still 0) | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Besu |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ❌ | | |Nimbus | ✅ | ✅ | ✅ | ❌ | | |Lodestar | ✅ | ✅ | ✅ | ❌ | | |Prysm | ✅ | ✅ | ✅ | ❌ | |Erigon |Lighthouse| ❌ | ❌ | ❌ | ❌ | | |Teku | ❌ | ❌ | ❌ | ❌ | | |Nimbus | ✅ | ✅ | ✅ | ❌ | | |Lodestar | ✅ | ✅ | ✅ | ❌ | | |Prysm | ✅ | ✅ | ✅ | ❌ | #### Issues: - Prysm-erigon-2 fell out of sync (but prysm-erigon-1 is doing fine, issue isolated to this node) - Nimbus-erigon-1 fell out of sync (but nimbus-erigon-2 is doing fine, issue isolated to this node) - LH-Erigon + Teku-Erigon seem to not work at all - Besu and Erigon still propose blocks with 0 transactions - Nimbus-nethermind proposes with 0 transactions ## mainnet-shadow-fork-5 #### Verisons used: Consensus layer | client name | git branch | docker image if possible | -------- |-------- | -------- |-------- | | Lighthouse |unstable|sigp/lighthouse:latest-unstable| | | | Teku | master | consensys/teku:develop | | | Lodestar | master | g11tech/lodestar:elfcus | | | Nimbus | kiln-dev-auth | parithoshj/nimbus:merge-e48c291 | | | Prysm | develop | gcr.io/prysmaticlabs/prysm/beacon-chain:latest,gcr.io/prysmaticlabs/prysm/validator:latest| Execution layer | client name |git branch | docker image if possible | | -------- |-------- |-------- | | Geth | master| parithoshj/geth:master-440c9fc | | Nethermind| kiln | nethermindeth/nethermind:kiln_shadowfork | | Erigon | devel | parithoshj/erigon:devel-1e3ad39 | | Besu | develop | hyperledger/besu:develop-openjdk-11-debug | #### Proposal tracker: | EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? | |--------| -------- | --------------- | ---------- |----------- | ------------------------- | |Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ❌ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Besu |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ❌ | ❌ | ❌ | ❌ | | |Nimbus | ✅ | ✅ | ✅ | ❌ | | |Lodestar | ✅ | ✅ | ✅ | ❌ | | |Prysm | ✅ | ✅ | ✅ | ❌ | |Erigon |Lighthouse| ❌ | ❌ | ❌ | ❌ | | |Teku | ❓ | ❌ | ❌ | ❌ | | |Nimbus | ❌ | ❌ | ❌ | ❌ | | |Lodestar | ❌ | ❌ | ❌ | ❌ | | |Prysm | ✅ | ✅ | ✅ | ✅ | #### Issues: - Erigon sync issue - Besu 0 tx proposals with most clients (likely timeout related) - Teku-besu fails to sync to head/stay synced - Nethermind-nimbus 0 proposal (likely timeout related) #### Sync tests: mainnet-shadow-fork-5-sync-test-prysm-geth-1: Pause CL mainnet-shadow-fork-5-sync-test-lighthouse-geth-1: Pause EL mainnet-shadow-fork-5-sync-test-prysm-nethermind-1: Pause EL mainnet-shadow-fork-5-sync-test-lighthouse-nethermind-1: Pause CL mainnet-shadow-fork-5-sync-test-prysm-besu-1: Pause CL mainnet-shadow-fork-5-sync-test-lighthouse-besu-1: Pause EL ##### Sync test results: mainnet-shadow-fork-5-sync-test-prysm-geth-1: ✅ Synced to head mainnet-shadow-fork-5-sync-test-lighthouse-geth-1: ✅ Synced to head mainnet-shadow-fork-5-sync-test-prysm-nethermind-1: ❌ `"No state available for block 0x6a8e30aefca8dd84dbe8654a65df062ff3264a4e8fd3a09c68ca9a5acff84309"` mainnet-shadow-fork-5-sync-test-lighthouse-nethermind-1: ❌ No sync peers mainnet-shadow-fork-5-sync-test-prysm-besu-1: ✅ Synced to head mainnet-shadow-fork-5-sync-test-lighthouse-besu-1: ✅ Synced to head ## mainnet-shadow-fork-4 #### Verisons used: Consensus layer | client name | git branch | docker image if possible | -------- |-------- | -------- |-------- | | Lighthouse |unstable|sigp/lighthouse:latest-unstable| | | | Teku | master | consensys/teku:develop | | | Lodestar | master | chainsafe/lodestar:next | | | Nimbus | kiln-dev-auth | parithoshj/nimbus:merge-9cbe61c | | | Prysm | develop | gcr.io/prysmaticlabs/prysm/beacon-chain:fix-error-is-ordering-d4ddcd,gcr.io/prysmaticlabs/prysm/validator:latest| Execution layer | client name |git branch | docker image if possible | | -------- |-------- |-------- | | Geth | master| parithoshj/geth:merge-59a3672 | | Nethermind| kiln | nethermindeth/nethermind:kiln_shadowfork_test | | Erigon | devel | parithoshj/erigon:devel-0929df8 | | Besu | develop | hyperledger/besu:develop | #### Proposal tracker: | EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? | |--------| -------- | --------------- | ---------- |----------- | ------------------------- | |Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ✅ | ✅ | | |Nimbus | ✅ | ✅ | ✅ | ✅ | | |Lodestar | ✅ | ✅ | ✅ | ✅ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Besu |Lighthouse| ✅ | ✅ | ✅ | ✅ | | |Teku | ✅ | ✅ | ❓ | ❓ | | |Nimbus | ✅ | ❓ | ❓ | ❓ | | |Lodestar | ✅ | ✅ | ✅ | ❌ | | |Prysm | ✅ | ✅ | ✅ | ✅ | |Erigon |Lighthouse| ❌ | ❌ | ❌ | ❌ | | |Teku | ❌ | ❌ | ❌ | ❌ | | |Nimbus | ❌ | ❌ | ❌ | ❌ | | |Lodestar | ❌ | ❌ | ❌ | ❌ | | |Prysm | ❌ | ❌ | ❌ | ❌ | #### Issues: - Geth fails sync test - prysm-nethermind failing (potential fix available but waiting on nethermind team) - Nethermind proposes empty blocks (not their fault afaik), related to async block production - nimbus-besu has issues - Erigon fails to sync to head, going to use full sync but it takes time