- 2.1M validators total
- 420 nodes
- 16c/32Gb each machine
- 5k keys/node
- Prysm 43%, lighthouse 33.5%, teku 17.5%, nimbus 5% and lodestar 1%
### Overall health:
Proposals: 25/32 (78.12%)
Attestations: 928136/2100000 (44.20%)
Source timely: 867459/2100000 (41.31%)
Target correct: 928136/2100000 (44.20%)
Target timely: 928136/2100000 (44.20%)
Head correct: 897751/2100000 (42.75%)
Head timely: 714156/2100000 (34.01%)
Sync committees: 10148/12800 (79.28%)
### Overall Network Report:
- The network is clearly not finalizing. The proposal rates are extremely healthy at ~80% range(compared to attestations). The attestations however aren't, indicating an issue in performing attestation duties in time or attestation propagation issues.
- Overall, the mainnet distribution is more consistent and performant compared to the more equal distribution.
- Per client resource use:
^ The above graph indicates that on average every client pair has enough resources. There are no clients that are hitting 100% use, the individual node graphs also do not indicate that any node goes above 85% usage of CPU. RAM use is far more stable and doesn't seem to increase beyond the initial increase due to non-finality periods.
^ The above block propagation chart is far healthier than earlier tests. We rarely see the block propagation values taking too long, they are mostly propagated within the expected slot time - average being ~4s and p95 being 8-14s.
^ The heatmap reinforces the idea, most blocks are infact propagated in an expepcted band (however that band is still slightly slower than mainnet which manages the 750ms-2s range).
^ Attestation propagation however isn't as healthy. The p95 mark is routinely above 20s, indicating that a lot of attestations are just too late. The average value is ~8-10s.
^ Xatu sees more attestations per epoch than ethdo (which is looking exclusively at what was include onchain). This graph has an interval of 2mins -> 500k * 3 = ~1.5m attestations per epoch (ethdo believes there are ~950k per epoch). The xatu query is taking a unique count of all attestation signatures seen in that time window, so bursts of attestations/delays may be slightly throwing things out by including old attestations in this metric.
**May indicate that `MAX_ATTESTATIONS` is insufficient at this scale?**
^ The attestation heatmap is far better than the previous run. We can indeed see that most attestations go out in the 4-5s mark as excpected. However, the variance is still a bit wide, we see a lot of attestations towards the end of the slot as well.
- Client specific performance for a duration (averaged by client pair) can be found here: https://grafana.observability.ethpandaops.io/d/participation_rates_dashboard/validator-participation-rates-dashboards?orgId=1&var-network=goerli&from=1691521652584&to=1691528618251
^ In the above graph (link in the previous point), we can see the averaged performance per client pair. Lodestar and Lighthouse are routinely able to hit the ~70% mark whereas Prysm is able to hit the ~45% mark. Teku and Nimbus are lower at the ~20% mark. Reach out to Dapplion to know more about how these stats are collected.
### Client Specific Reports:
- Pending, help out!