owned this note
owned this note
Published
Linked with GitHub
# Shadow fork tracker
## mainnet-shadow-fork-8
#### Proposal tracker:
| EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? |
|--------| -------- | --------------- | ---------- |----------- | ------------------------- |
|Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Besu |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ❌ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Erigon |Lighthouse| ? | ? | ? | ? |
| |Teku | ✅ | ✅ | ✅ | ? |
| |Nimbus | ✅ | ✅ | ✅ | ? |
| |Lodestar | ? | ? | ? | ? |
| |Prysm | ? | ? | ? | ? |
Note: Erigon had sync issues late last week, its been fixed now but the nodes are still syncing to head. They still need to sync ~900k blocks at the time of creating this chart, it'll be updated once its at head.
#### Issues:
- nimbus-besu-2 seems to be out of sync. nimbus-besu-1 is fine, so it doesn't seem to be an issue with the pair.
- Besu nodes are proposing blocks with 0 transactions, likely a regression relating to timeout
## mainnet-shadow-fork-7
#### Proposal tracker:
| EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? |
|--------| -------- | --------------- | ---------- |----------- | ------------------------- |
|Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Besu |Lighthouse| ❌ | ❌ | ❌ | ❌ |
| |Teku | ❌ | ❌ | ❌ | ❌ |
| |Nimbus | ✅ | ✅ | ✅ | ❌ |
| |Lodestar | ✅ | ✅ | ✅ | ❌ |
| |Prysm | ✅ | ✅ | ✅ | ❌ |
|Erigon |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ❌ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
#### Issues:
- Erigon had issues with peers not having the state and dropping out of sync
- Besu had a world state issue
## ropsten
#### Proposal tracker:
| EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? |
|--------| -------- | --------------- | ---------- |----------- | ------------------------- |
|Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Besu |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ❌ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Erigon |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ? | ? | ? | ? |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
#### Issues:
- Besu-Nimbus incompatibility (probably limited to ws being used by Besu, EF nodes are fine)
- Nethermind concurrency bug
## mainnet-shadow-fork-6
#### Verisons used:
- Same as those recommended in the EF Ropsten blogpost
#### Proposal tracker:
| EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? |
|--------| -------- | --------------- | ---------- |----------- | ------------------------- |
|Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ❌ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ (some are still 0) |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Besu |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ❌ |
| |Nimbus | ✅ | ✅ | ✅ | ❌ |
| |Lodestar | ✅ | ✅ | ✅ | ❌ |
| |Prysm | ✅ | ✅ | ✅ | ❌ |
|Erigon |Lighthouse| ❌ | ❌ | ❌ | ❌ |
| |Teku | ❌ | ❌ | ❌ | ❌ |
| |Nimbus | ✅ | ✅ | ✅ | ❌ |
| |Lodestar | ✅ | ✅ | ✅ | ❌ |
| |Prysm | ✅ | ✅ | ✅ | ❌ |
#### Issues:
- Prysm-erigon-2 fell out of sync (but prysm-erigon-1 is doing fine, issue isolated to this node)
- Nimbus-erigon-1 fell out of sync (but nimbus-erigon-2 is doing fine, issue isolated to this node)
- LH-Erigon + Teku-Erigon seem to not work at all
- Besu and Erigon still propose blocks with 0 transactions
- Nimbus-nethermind proposes with 0 transactions
## mainnet-shadow-fork-5
#### Verisons used:
Consensus layer
| client name | git branch | docker image if possible
| -------- |-------- | -------- |-------- |
| Lighthouse |unstable|sigp/lighthouse:latest-unstable| | |
| Teku | master | consensys/teku:develop | |
| Lodestar | master | g11tech/lodestar:elfcus | |
| Nimbus | kiln-dev-auth | parithoshj/nimbus:merge-e48c291 | |
| Prysm | develop | gcr.io/prysmaticlabs/prysm/beacon-chain:latest,gcr.io/prysmaticlabs/prysm/validator:latest|
Execution layer
| client name |git branch | docker image if possible |
| -------- |-------- |-------- |
| Geth | master| parithoshj/geth:master-440c9fc |
| Nethermind| kiln | nethermindeth/nethermind:kiln_shadowfork |
| Erigon | devel | parithoshj/erigon:devel-1e3ad39 |
| Besu | develop | hyperledger/besu:develop-openjdk-11-debug |
#### Proposal tracker:
| EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? |
|--------| -------- | --------------- | ---------- |----------- | ------------------------- |
|Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ❌ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Besu |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ❌ | ❌ | ❌ | ❌ |
| |Nimbus | ✅ | ✅ | ✅ | ❌ |
| |Lodestar | ✅ | ✅ | ✅ | ❌ |
| |Prysm | ✅ | ✅ | ✅ | ❌ |
|Erigon |Lighthouse| ❌ | ❌ | ❌ | ❌ |
| |Teku | ❓ | ❌ | ❌ | ❌ |
| |Nimbus | ❌ | ❌ | ❌ | ❌ |
| |Lodestar | ❌ | ❌ | ❌ | ❌ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
#### Issues:
- Erigon sync issue
- Besu 0 tx proposals with most clients (likely timeout related)
- Teku-besu fails to sync to head/stay synced
- Nethermind-nimbus 0 proposal (likely timeout related)
#### Sync tests:
mainnet-shadow-fork-5-sync-test-prysm-geth-1: Pause CL
mainnet-shadow-fork-5-sync-test-lighthouse-geth-1: Pause EL
mainnet-shadow-fork-5-sync-test-prysm-nethermind-1: Pause EL
mainnet-shadow-fork-5-sync-test-lighthouse-nethermind-1: Pause CL
mainnet-shadow-fork-5-sync-test-prysm-besu-1: Pause CL
mainnet-shadow-fork-5-sync-test-lighthouse-besu-1: Pause EL
##### Sync test results:
mainnet-shadow-fork-5-sync-test-prysm-geth-1: ✅ Synced to head
mainnet-shadow-fork-5-sync-test-lighthouse-geth-1: ✅ Synced to head
mainnet-shadow-fork-5-sync-test-prysm-nethermind-1: ❌ `"No state available for block 0x6a8e30aefca8dd84dbe8654a65df062ff3264a4e8fd3a09c68ca9a5acff84309"`
mainnet-shadow-fork-5-sync-test-lighthouse-nethermind-1: ❌ No sync peers
mainnet-shadow-fork-5-sync-test-prysm-besu-1: ✅ Synced to head
mainnet-shadow-fork-5-sync-test-lighthouse-besu-1: ✅ Synced to head
## mainnet-shadow-fork-4
#### Verisons used:
Consensus layer
| client name | git branch | docker image if possible
| -------- |-------- | -------- |-------- |
| Lighthouse |unstable|sigp/lighthouse:latest-unstable| | |
| Teku | master | consensys/teku:develop | |
| Lodestar | master | chainsafe/lodestar:next | |
| Nimbus | kiln-dev-auth | parithoshj/nimbus:merge-9cbe61c | |
| Prysm | develop | gcr.io/prysmaticlabs/prysm/beacon-chain:fix-error-is-ordering-d4ddcd,gcr.io/prysmaticlabs/prysm/validator:latest|
Execution layer
| client name |git branch | docker image if possible |
| -------- |-------- |-------- |
| Geth | master| parithoshj/geth:merge-59a3672 |
| Nethermind| kiln | nethermindeth/nethermind:kiln_shadowfork_test |
| Erigon | devel | parithoshj/erigon:devel-0929df8 |
| Besu | develop | hyperledger/besu:develop |
#### Proposal tracker:
| EL | CL | Synced to head? | Attesting? | Proposing? | Proposing non-empty slot? |
|--------| -------- | --------------- | ---------- |----------- | ------------------------- |
|Geth |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Nethermind |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ✅ | ✅ |
| |Nimbus | ✅ | ✅ | ✅ | ✅ |
| |Lodestar | ✅ | ✅ | ✅ | ✅ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Besu |Lighthouse| ✅ | ✅ | ✅ | ✅ |
| |Teku | ✅ | ✅ | ❓ | ❓ |
| |Nimbus | ✅ | ❓ | ❓ | ❓ |
| |Lodestar | ✅ | ✅ | ✅ | ❌ |
| |Prysm | ✅ | ✅ | ✅ | ✅ |
|Erigon |Lighthouse| ❌ | ❌ | ❌ | ❌ |
| |Teku | ❌ | ❌ | ❌ | ❌ |
| |Nimbus | ❌ | ❌ | ❌ | ❌ |
| |Lodestar | ❌ | ❌ | ❌ | ❌ |
| |Prysm | ❌ | ❌ | ❌ | ❌ |
#### Issues:
- Geth fails sync test
- prysm-nethermind failing (potential fix available but waiting on nethermind team)
- Nethermind proposes empty blocks (not their fault afaik), related to async block production
- nimbus-besu has issues
- Erigon fails to sync to head, going to use full sync but it takes time