# bigboi-beaconchain-test-3 ### Details: - 1.386M validators total - 420 nodes - 16c/32Gb each machine - 3300k keys/node - Prysm 43%, lighthouse 33.5%, teku 17.5%, nimbus 5% and lodestar 1% ### Overall health: Epoch 1: Proposals: 28/32 (87.50%) Attestations: 1179660/1386000 (85.11%) Source timely: 1094136/1386000 (78.94%) Target correct: 1149956/1386000 (82.97%) Target timely: 1149956/1386000 (82.97%) Head correct: 1111231/1386000 (80.18%) Head timely: 918835/1386000 (66.29%) Sync committees: 12927/14336 (90.17%) ### Overall Network Report: - ![overview-resource-use](https://storage.googleapis.com/ethereum-hackmd/upload_f0c32807e29183674d9b13b823de6817.png) ^ The resource use on the nodes are reasonable compared to the 2.1M tests. The resource use is ~20% lower than in previous tests. I'm sure the finalizing nature of the testnet also helps to a great degree. - ![per-client-participation](https://storage.googleapis.com/ethereum-hackmd/upload_cc8486ad60570d7d8c781d0ff4aa15c5.png) ![per-client-head-participation](https://storage.googleapis.com/ethereum-hackmd/upload_0bd3d1782e205c57acbdf08acc021f14.png) ^ Above we can see the performance level of various client pairs at this size. We do see source participation rates of ~95% in some clients, which is very healthy. It also showcases which clients might need some optimizations. - ![block-propagation](https://storage.googleapis.com/ethereum-hackmd/upload_6baeef7ff166b678dc1a07de3a553092.png) ^ The blocks are propagated through the network extremely quickly, relatively close to mainnet rates. The p95 is worse than mainnet (mainnet p95 is 5s), but still at an acceptable rate (mostly before the 8s mark). -![block-propagation-heatmap](https://storage.googleapis.com/ethereum-hackmd/upload_56a7ce53d29143cf1b6cf0b6db12c04e.png) ^ The heatmap shows that most proposals are occuring before the 2s mark. This is roughly 0.5s slower than mainnet.The block proposal rate is also a lot more uniform compared to other tests. - ![attestation-propagation](https://storage.googleapis.com/ethereum-hackmd/upload_4d91f53c32ca9730aef16db6d226d85f.png) ![attestation-propagation-heatmap](https://storage.googleapis.com/ethereum-hackmd/upload_0621b12784a8ec477d4f361c147181cb.png) ^ The p95 for the attestation propagation is still quite high. Mainnet has a far tighter band. This could indicate the performance optimizations needed. - ![attestation seen rate](https://storage.googleapis.com/ethereum-hackmd/upload_63999b77ba8789ecba9a98eabd36bd01.png) ^ This graph has an interval of 2mins -> 400k * 3 = ~1.2m. The xatu query is taking a unique count of all attestation signatures seen in that time window. ### Client Specific Reports: - Teku: Recurring `WARN` log ``` 2023-08-11 10:20:18.083 WARN - Failed callback java.io.IOException: Close SendCallback@750df666[PENDING][i=null,cb=null] in state PENDING at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:458) ~[jetty-util-11.0.15.jar:11.0.15] at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:551) ~[jetty-server-11.0.15.jar:11.0.15] at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:330) ~[jetty-io-11.0.15.jar:11.0.15] at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:1113) ~[jetty-io-11.0.15.jar:11.0.15] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) [jetty-util-11.0.15.jar:11.0.15] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) [jetty-util-11.0.15.jar:11.0.15] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) [jetty-util-11.0.15.jar:11.0.15] at java.lang.Thread.run(Unknown Source) [?:?] ```