# msf3 manual sync tests All nodes were started on fresh machines, they were all synced till the head of the chain. ~2H before TTD the respective EL/CL was paused with `docker pause ...`. Once TTD was hit (but before finality post TTD), the container was un-paused with `docker unpause ...`. ## Nodes: - mainnet-shadow-fork-3-pause-test-prysm-geth-1: Stop EL - mainnet-shadow-fork-3-pause-test-lighthouse-geth-1: Stop CL - mainnet-shadow-fork-3-pause-test-prysm-nethermind-1: Stop CL - mainnet-shadow-fork-3-pause-test-lighthouse-nethermind-1: Stop EL ## Results: ### mainnet-shadow-fork-3-pause-test-prysm-geth-1: Stop EL Access: `ssh [email protected]` Failed to catch up to head post unpause. Prysm logs: ``` time="2022-05-05 13:53:13" level=debug msg="Block is not processed" error="beacon node doesn't have a parent in db with root: (in processBlock, slot=41529) 0xa272b8e15d81fc4105e9a6cd7810d63307b36a2637913ff5a0ba5c33ee0d1d9a" prefix=initial-sync ``` Geth logs: ``` INFO [05-05|13:54:01.417] Legacy sync reached merge threshold number=14,716,821 hash=71ec49..8da10d td=48,144,739,311,386,193,556,332 ttd=48,144,711,970,238,236,721,152 INFO [05-05|13:54:01.417] State heal in progress accounts=247,[email protected] slots=469,[email protected] [email protected] nodes=3,813,[email protected] pending=17 WARN [05-05|13:54:01.417] Local chain is post-merge, waiting for beacon client sync switch-over... WARN [05-05|13:54:01.452] Unexpected trienode heal packet peer=8c987cc3 reqid=3,557,761,924,799,430,402 DEBUG[05-05|13:54:02.872] Served eth_getBlockByNumber conn=5.34.180.243:41512 reqid=1 duration="148.096µs" INFO [05-05|13:54:03.127] Disabling direct-ancient mode origin=14,716,820 ancient=14,626,820 INFO [05-05|13:54:03.502] Legacy sync reached merge threshold number=14,716,821 hash=71ec49..8da10d td=48,144,739,311,386,193,556,332 ttd=48,144,711,970,238,236,721,152 INFO [05-05|13:54:03.502] State heal in progress accounts=247,[email protected] slots=469,[email protected] [email protected] nodes=3,813,[email protected] pending=17 WARN [05-05|13:54:03.542] Unexpected trienode heal packet peer=8c987cc3 reqid=7,389,355,313,485,979,547 WARN [05-05|13:54:03.742] Failed to retrieve stats server message err="websocket: close 1006 (abnormal closure): unexpected EOF" WARN [05-05|13:54:04.245] Transaction stats report failed err="use of closed network connection" WARN [05-05|13:54:05.320] Stats server unreachable err="dial tcp 164.92.169.47:80: connect: connection refused" INFO [05-05|13:54:07.974] Disabling direct-ancient mode origin=14,716,820 ancient=14,626,820 INFO [05-05|13:54:08.422] Legacy sync reached merge threshold number=14,716,821 hash=71ec49..8da10d td=48,144,739,311,386,193,556,332 ttd=48,144,711,970,238,236,721,152 INFO [05-05|13:54:08.423] State heal in progress accounts=247,[email protected] slots=469,[email protected] [email protected] nodes=3,813,[email protected] pending=17 WARN [05-05|13:54:08.555] Unexpected trienode heal packet peer=ca53cbf1 reqid=1,547,625,417,515,909,221 ``` ### mainnet-shadow-fork-3-pause-test-lighthouse-geth-1: Stop CL Access: `ssh [email protected]` CL refuses to advance head state. Lighthouse logs: ``` May 05 13:56:06.001 DEBG Eth1 cache sync status ready: true, voting_target_timestamp: 1651718348, latest_cached_timestamp: 1651726797, latest_cached_block_number: 14715284, eth1_head_block: 14716820, service: slot_notifier May 05 13:56:08.001 DEBG Proposer prepare routine firing service: beacon May 05 13:56:08.077 DEBG Committee cache miss head_block_root: 0x6e74…18d7, shuffling_id: Epoch(1311), service: beacon May 05 13:56:09.001 DEBG Refused to advance head state current_slot: 41955, head_slot: 41685, service: state_advance ``` Geth logs: ``` INFO [05-05|13:57:00.137] Legacy sync reached merge threshold number=14,716,821 hash=71ec49..8da10d td=48,144,739,311,386,193,556,332 ttd=48,144,711,970,238,236,721,152 INFO [05-05|13:57:00.362] State heal in progress accounts=259,[email protected] slots=468,[email protected] [email protected] nodes=3,878,[email protected] pending=4369 WARN [05-05|13:57:00.362] Local chain is post-merge, waiting for beacon client sync switch-over... ``` ### mainnet-shadow-fork-3-pause-test-prysm-nethermind-1: Stop CL Access: `ssh [email protected]` Synced to head ### mainnet-shadow-fork-3-pause-test-lighthouse-nethermind-1 Access: `ssh [email protected]` Synced to head