# Post Fusaka issues summary Preliminary incident reports ## Prysm participation drop Prysm nodes were halted due to stale attestations triggering generation of older states. During the incident, fork choice experienced lot of short stale branches, block delivery times became very long starting block 23937064 and nodes switched from builder API to local building. The solution for affected users is to simply restart the node with a config flag. Client release fix or resync was not needed, the network started recovering quickly after the issue was identified. More details in [Prysm postmortem](https://prysm.offchainlabs.com/docs/misc/mainnet-postmortems/) [thread in R&D](https://discord.com/channels/595666850260713488/1445979272421572649), [interop channel](https://discord.com/channels/595666850260713488/892088344438255616/1445980055615574116) ### Timeline At [epoch 411440](https://light-mainnet.beaconcha.in/epoch/411440 ), attestation rate starts to decline from stable ~97% to around 77%. The difference roughly corresponds to share of Prysm nodes in the network. - Around 3:25 AM, reports on R&D notice participation drop - Around 3:45, circuit breaker triggers local block building - 4:35 AM, the solution is shared by Prysm devs - 4:45 - 9:15 AM, participation rate steadly improves until it reached the original ~96% ## Missing data in Dasmon Dasmon probing shows many reports of missing data. Might be caused by late blob deliveries, still being investigated. https://lab.ethpandaops.io/ethereum/data-availability/probes?isLive=true&page=1&result=missing ## Nethermind bug with Nimbus Nethermind reported a wrong invalid block at slot 13164552 running with Nimbus CL. Not many users affected, seems like hard to reproduce. Nethermind was compiled in a specific way in Nix environment: https://github.com/NethermindEth/nethermind/issues/9878 ## Late delivery reorgs Despite slot [13164887](https://light-mainnet.beaconcha.in/slot/13164887) receiving only few attestations, next slot was built on top of it. Block arrived late, at t+4.65s, with only 2 head votes and normally it would be an orphan but later became canonical. Someone proposed a block on top of it which got enough votes and because fork choice evaluates the whole tree, the parent became valid. ## Reth bug to investigate (potentially related to Fusaka): https://github.com/paradigmxyz/reth/issues/20111 https://github.com/paradigmxyz/reth/issues/20110