# Goerli debugging Problematic slots: 7413760 and 7413770 Missing keys(all patched): 10K EF (Running Prysm), 40k Nimbus (non-upgraded nodes), 40k Prysm Root cause: - Prysm had an issue where on the upgrade to deneb they zero-ed out the historical root. The devnets all forked at <256 epochs, meaning the network was too new to even hit one cycle of `SLOTS_PER_HISTORICAL_ROOT`. So we never triggered this issue in devnets. - Ideally the spec tests and hivetests get updated to catch the problem ## 7413760 Proposer: Teku Slot ssz: https://drive.google.com/file/d/1ltnPSXxiN-A_6S9vhFMq7IQjNv9tCTZL/view?usp=sharing Prestate for slot 7413759: https://drive.google.com/file/d/1PY9-d4M5m1s3ESNNmUzYHb3smlPbjARh/view?usp=sharing Post state for slot: https://drive.google.com/file/d/16fCCfbCqbzUmugbHs7DGt74dTqck86CW/view?usp=sharing Nimbus-besu: accepted as valid ``` INF 2024-01-17 06:32:00.002+00:00 Slot start topics="beacnde" slot=7413760 epoch=231680 sync=synced peers=157 head=44c7dd0a:7413759 finalized=231677:68b97d60 delay=2ms281us832ns ``` Lighthouse-geth: accepted as valid ``` Jan 17 06:32:06.000 INFO Synced slot: 7413760, block: 0x92d8…5135, epoch: 231680, finalized_epoch: 231678, finalized_root: 0x721e…5320, exec_hash: 0x04e5…f066 (verified), peers: 77, service: slot_notifier ``` Lodestar: accepted ``` Jan-17 06:32:06.001[] info: Synced - slot: 7413760 - head: 0x92d8…5135 - exec-block: valid(10388176 0x04e5…) - finalized: 0x721e…5320:231678 - peers: 48 Jan-17 06:32:06.002[] info: New sync committee period 905 ``` Teku: accepted ``` 2024-01-17 06:32:04.107 INFO - Slot Event *** Slot: 7413760, Block: ... empty, Justified: 231679, Finalized: 231678, Peers: 100 2024-01-17 06:32:04.357 WARN - Late Block Import *** Block: 92d898cb189bae13cb97db2818ec1426dc1ce9047f1d184790eca8d0a4995135 (7413760) proposer 135919 arrival 1914ms, gossip_validation +4ms, pre-state_retrieved +6ms, processed +2136ms, execution_payload_result_received +0ms, begin_importing +45ms, transaction_prepared +175ms, transaction_committed +0ms, completed +77ms ``` Prysm: No log/missing logs We see logs about 7413759, then a p2p error failing to validate the transition. The next slot it syncs is 7413770 and we don't have logs for anything in between - as if prysm hung on the transition. ``` time="2024-01-17 06:31:48" level=info msg="Synced new block" block=0x44c7dd0a... epoch=231679 finalizedEpoch=231677 finalizedRoot=0x68b97d60... prefix=blockchain slot=7413759 time="2024-01-17 06:31:48" level=info msg="Finished applying state transition" attestations=128 payloadHash=0xeb6da0b2d211 prefix=blockchain slot=7413759 syncBitsCount=418 txCount=69 time="2024-01-17 06:32:00" level=warning msg="Attestation subnets with digest 0x628941ef are no longer valid, unsubscribing from all of them." prefix=sync time="2024-01-17 06:32:00" level=warning msg="Sync subnets with digest 0x628941ef are no longer valid, unsubscribing from all of them." prefix=sync time="2024-01-17 06:32:00" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=5000 time="2024-01-17 06:32:02" level=error msg="Could not handle p2p pubsub" error="failed to validate consensus state transition function: could not execute state transition: could not validate state root, wanted: 0x0498b3bee228592e79df7c54fa568adbfc463919bf02c35d7990dfe01f71e392, received: 0xcaedd8c6d9dc0162b57b0ddf63f75c43a38c4bdb137516df2288799ff604bade" prefix=sync topic="/eth2/a75dccf2/beacon_block/ssz_snappy" ``` ## 7413770 Proposer: https://dora.goerli.ethpandaops.io/validator/40887 (Prysm team) Block number: 10388176 Prysm accepted: ``` time="2024-01-17 06:34:01" level=info msg="Synced new block" block=0xb95d233b... epoch=231680 finalizedEpoch=231678 finalizedRoot=0x721ecad9... prefix=blockchain slot=7413770 ``` Everyone else rejected it Lighthouse: ``` Jan 17 06:34:06.000 INFO Synced slot: 7413770, block: … empty, epoch: 231680, finalized_epoch: 231678, finalized_root: 0x721e…5320, exec_hash: 0x9e16…f90b (verified), peers: 73, service: slot_notifier ``` Teku: ``` 2024-01-17 06:34:01.699 WARN - State transition error while importing block 7413770 (0xb95d233b9812fdb3277edbc7c6c791ee158cef387246c8134409f003935387b1) tech.pegasys.teku.spec.logic.common.statetransition.exceptions.BlockProcessingException: Block state root does NOT match the calculated state root! Block state root: 0xd44e450fbc92d9fc4d6f03d4c975b8dff8a5b8eddf4b98619fbe2d9d197c544c New state root: 0xc8996e286687a5560e29da4c23865922aee8d3b4035cc29443ee55f412a7078b block proposer: 40887 ``` Nimbus: ``` INF 2024-01-17 06:34:00.000+00:00 Slot start topics="beacnde" slot=7413770 epoch=231680 sync=synced peers=154 head=c9f39ace:7413769 finalized=231678:721ecad9 delay=568us612ns INF 2024-01-17 06:34:02.189+00:00 State replayed topics="chaindag" blocks=0 slots=10 current=c9f39ace:7413769@7413770 ancestor=44c7dd0a:7413759@7413760 target=44c7dd0a:7413759@7413770 ancestorStateRoot=34da42de targetStateRoot=8a9e17ce found=false assignDur=69ms520us228ns replayDur=370ms513us166ns INF 2024-01-17 06:34:02.329+00:00 Invalid block ``` Lodestar: ``` Jan-17 06:34:01.744[network] error: Error receiving block slot=7413770, peer=16Uiu2HAmGDTrvyguFuDQmDf1cPz7dc1nTuvcFP4GdXagkw7v1VFh, code=BLOCK_ERROR_INVALID_STATE_ROOT, slot=7413770, root=0xc8996e286687a5560e29da4c23865922aee8d3b4035cc29443ee55f412a7078b, expectedRoot=0xd44e450fbc92d9fc4d6f03d4c975b8dff8a5b8eddf4b98619fbe2d9d197c544c Error: BLOCK_ERROR_INVALID_STATE_ROOT at verifyBlocksStateTransitionOnly (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlocksStateTransitionOnly.ts:69:13) at BeaconChain.verifyBlocksInEpoch (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlock.ts:112:7) at BeaconChain.processBlocks (file:///usr/app/packages/beacon-node/src/chain/blocks/index.ts:72:7) at JobItemQueue.runJob (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:101:22) Jan-17 06:34:06.003[] info: Synced - slot: 7413770 - head: (slot -1) 0xc9f3…9c73 - exec-block: valid(10388180 0x9e16…) - finalized: 0x721e…5320:231678 - peers: 51 ```