Shadow fork tracker

NOTE!: Ideally keep this doc internal :)

mainnet-shadow-fork-8

Proposal tracker:

EL CL Synced to head? Attesting? Proposing? Proposing non-empty slot?
Geth Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Nethermind Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Besu Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ❌
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Erigon Lighthouse ? ? ? ?
Teku ✅ ✅ ✅ ?
Nimbus ✅ ✅ ✅ ?
Lodestar ? ? ? ?
Prysm ? ? ? ?

Note: Erigon had sync issues late last week, its been fixed now but the nodes are still syncing to head. They still need to sync ~900k blocks at the time of creating this chart, it’ll be updated once its at head.

Issues:

  • nimbus-besu-2 seems to be out of sync. nimbus-besu-1 is fine, so it doesn’t seem to be an issue with the pair.
  • Besu nodes are proposing blocks with 0 transactions, likely a regression relating to timeout

mainnet-shadow-fork-7

Proposal tracker:

EL CL Synced to head? Attesting? Proposing? Proposing non-empty slot?
Geth Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Nethermind Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Besu Lighthouse ❌ ❌ ❌ ❌
Teku ❌ ❌ ❌ ❌
Nimbus ✅ ✅ ✅ ❌
Lodestar ✅ ✅ ✅ ❌
Prysm ✅ ✅ ✅ ❌
Erigon Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ❌
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅

Issues:

  • Erigon had issues with peers not having the state and dropping out of sync
  • Besu had a world state issue

ropsten

Proposal tracker:

EL CL Synced to head? Attesting? Proposing? Proposing non-empty slot?
Geth Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Nethermind Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Besu Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ❌
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Erigon Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ? ? ? ?
Prysm ✅ ✅ ✅ ✅

Issues:

  • Besu-Nimbus incompatibility (probably limited to ws being used by Besu, EF nodes are fine)
  • Nethermind concurrency bug

mainnet-shadow-fork-6

Verisons used:

  • Same as those recommended in the EF Ropsten blogpost

Proposal tracker:

EL CL Synced to head? Attesting? Proposing? Proposing non-empty slot?
Geth Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Nethermind Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ❌
Lodestar ✅ ✅ ✅ ✅ (some are still 0)
Prysm ✅ ✅ ✅ ✅
Besu Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ❌
Nimbus ✅ ✅ ✅ ❌
Lodestar ✅ ✅ ✅ ❌
Prysm ✅ ✅ ✅ ❌
Erigon Lighthouse ❌ ❌ ❌ ❌
Teku ❌ ❌ ❌ ❌
Nimbus ✅ ✅ ✅ ❌
Lodestar ✅ ✅ ✅ ❌
Prysm ✅ ✅ ✅ ❌

Issues:

  • Prysm-erigon-2 fell out of sync (but prysm-erigon-1 is doing fine, issue isolated to this node)
  • Nimbus-erigon-1 fell out of sync (but nimbus-erigon-2 is doing fine, issue isolated to this node)
  • LH-Erigon + Teku-Erigon seem to not work at all
  • Besu and Erigon still propose blocks with 0 transactions
  • Nimbus-nethermind proposes with 0 transactions

mainnet-shadow-fork-5

Verisons used:

Consensus layer
| client name | git branch | docker image if possible
| -------- |-------- | -------- |-------- |
| Lighthouse |unstable|sigp/lighthouse:latest-unstable| | |
| Teku | master | consensys/teku:develop | |
| Lodestar | master | g11tech/lodestar:elfcus | |
| Nimbus | kiln-dev-auth | parithoshj/nimbus:merge-e48c291 | |
| Prysm | develop | gcr.io/prysmaticlabs/prysm/beacon-chain:latest,gcr.io/prysmaticlabs/prysm/validator:latest|

Execution layer

client name git branch docker image if possible
Geth master parithoshj/geth:master-440c9fc
Nethermind kiln nethermindeth/nethermind:kiln_shadowfork
Erigon devel parithoshj/erigon:devel-1e3ad39
Besu develop hyperledger/besu:develop-openjdk-11-debug

Proposal tracker:

EL CL Synced to head? Attesting? Proposing? Proposing non-empty slot?
Geth Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Nethermind Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ❌
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Besu Lighthouse ✅ ✅ ✅ ✅
Teku ❌ ❌ ❌ ❌
Nimbus ✅ ✅ ✅ ❌
Lodestar ✅ ✅ ✅ ❌
Prysm ✅ ✅ ✅ ❌
Erigon Lighthouse ❌ ❌ ❌ ❌
Teku ❓ ❌ ❌ ❌
Nimbus ❌ ❌ ❌ ❌
Lodestar ❌ ❌ ❌ ❌
Prysm ✅ ✅ ✅ ✅

Issues:

  • Erigon sync issue
  • Besu 0 tx proposals with most clients (likely timeout related)
  • Teku-besu fails to sync to head/stay synced
  • Nethermind-nimbus 0 proposal (likely timeout related)

Sync tests:

mainnet-shadow-fork-5-sync-test-prysm-geth-1: Pause CL
mainnet-shadow-fork-5-sync-test-lighthouse-geth-1: Pause EL
mainnet-shadow-fork-5-sync-test-prysm-nethermind-1: Pause EL
mainnet-shadow-fork-5-sync-test-lighthouse-nethermind-1: Pause CL
mainnet-shadow-fork-5-sync-test-prysm-besu-1: Pause CL
mainnet-shadow-fork-5-sync-test-lighthouse-besu-1: Pause EL

Sync test results:

mainnet-shadow-fork-5-sync-test-prysm-geth-1: ✅ Synced to head
mainnet-shadow-fork-5-sync-test-lighthouse-geth-1: ✅ Synced to head
mainnet-shadow-fork-5-sync-test-prysm-nethermind-1: ❌ "No state available for block 0x6a8e30aefca8dd84dbe8654a65df062ff3264a4e8fd3a09c68ca9a5acff84309"
mainnet-shadow-fork-5-sync-test-lighthouse-nethermind-1: ❌ No sync peers
mainnet-shadow-fork-5-sync-test-prysm-besu-1: ✅ Synced to head
mainnet-shadow-fork-5-sync-test-lighthouse-besu-1: ✅ Synced to head

mainnet-shadow-fork-4

Verisons used:

Consensus layer
| client name | git branch | docker image if possible
| -------- |-------- | -------- |-------- |
| Lighthouse |unstable|sigp/lighthouse:latest-unstable| | |
| Teku | master | consensys/teku:develop | |
| Lodestar | master | chainsafe/lodestar:next | |
| Nimbus | kiln-dev-auth | parithoshj/nimbus:merge-9cbe61c | |
| Prysm | develop | gcr.io/prysmaticlabs/prysm/beacon-chain:fix-error-is-ordering-d4ddcd,gcr.io/prysmaticlabs/prysm/validator:latest|

Execution layer

client name git branch docker image if possible
Geth master parithoshj/geth:merge-59a3672
Nethermind kiln nethermindeth/nethermind:kiln_shadowfork_test
Erigon devel parithoshj/erigon:devel-0929df8
Besu develop hyperledger/besu:develop

Proposal tracker:

EL CL Synced to head? Attesting? Proposing? Proposing non-empty slot?
Geth Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Nethermind Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ✅ ✅
Nimbus ✅ ✅ ✅ ✅
Lodestar ✅ ✅ ✅ ✅
Prysm ✅ ✅ ✅ ✅
Besu Lighthouse ✅ ✅ ✅ ✅
Teku ✅ ✅ ❓ ❓
Nimbus ✅ ❓ ❓ ❓
Lodestar ✅ ✅ ✅ ❌
Prysm ✅ ✅ ✅ ✅
Erigon Lighthouse ❌ ❌ ❌ ❌
Teku ❌ ❌ ❌ ❌
Nimbus ❌ ❌ ❌ ❌
Lodestar ❌ ❌ ❌ ❌
Prysm ❌ ❌ ❌ ❌

Issues:

  • Geth fails sync test
  • prysm-nethermind failing (potential fix available but waiting on nethermind team)
  • Nethermind proposes empty blocks (not their fault afaik), related to async block production
  • nimbus-besu has issues
  • Erigon fails to sync to head, going to use full sync but it takes time
Select a repo