# Big Blocks Test Analysis ## Block ranges ### Test day 1 (28/05/23) #### 128 kB Blocks 17355587 –> 17355596 Slots 6535785 -> 6535791 #### 256 kB Blocks 17355613 –> 17355622 Slots 6535811 -> 6535820 #### 512 kB Blocks 17355721 –> 17355730 Slots 6535920 -> 6535929 #### 768 kB ##### Set 1 Blocks 17355770 –> 17355789 Slots 6535970 -> 6535979 ##### Set 2 Blocks 17355843 –> 17355852 Slots 6536043 -> 6536052 #### 1MB ##### Set 1 Blocks 17355914 –> 17355925 Slots 6536115 -> 6536126 ##### Set 2 Blocks 17355996 –> 17356004 Slots 6536197 -> 6536207 ### Test day 2 (11/06/23) #### 768kb ##### Set 1 Blocks 17455992 –> 17456003 Slots 6637646 -> 6637657 - [Prysm](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=xatu-sentry-syd1-mainnet-prysm-002&var-client_name=xatu-sentry-sfo3-mainnet-prysm-001&var-client_name=xatu-sentry-syd1-mainnet-prysm-001&var-client_name=xatu-sentry-ams3-mainnet-prysm-002&var-consensus_implementation=prysm&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686475460783&to=1686476072340) - [Lighthouse](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686475281626&to=1686476271222&var-client_name=All) - [Teku](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=All&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686475570202&to=1686476059640&var-client_name=xatu-sentry-sfo3-mainnet-teku-002&var-client_name=xatu-sentry-sfo3-mainnet-teku-003&var-client_name=xatu-sentry-syd1-mainnet-teku-001&var-client_name=xatu-sentry-ams3-mainnet-teku-003&var-client_name=xatu-sentry-ams3-mainnet-teku-002&var-client_name=xatu-sentry-ams3-mainnet-teku-001&var-client_name=xatu-sentry-syd1-mainnet-teku-002&var-client_name=xatu-sentry-syd1-mainnet-teku-003) - [Lodestar](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lodestar&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686475524306&to=1686476044518&var-client_name=xatu-sentry-ams3-mainnet-lodestar-002&var-client_name=xatu-sentry-ams3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-002&var-client_name=xatu-sentry-syd1-mainnet-lodestar-003&var-client_name=xatu-sentry-syd1-mainnet-lodestar-002) - [Nimbus](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=nimbus&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686475569495&to=1686476005373&var-client_name=All&var-attestation_prefix=) ##### Set 2 Blocks 17457486 –> 17457496 Slots 6639163 -> 6639173 - [Prysm](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=xatu-sentry-syd1-mainnet-prysm-002&var-client_name=xatu-sentry-sfo3-mainnet-prysm-001&var-client_name=xatu-sentry-syd1-mainnet-prysm-001&var-client_name=xatu-sentry-ams3-mainnet-prysm-002&var-consensus_implementation=prysm&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686493609993&to=1686494615368) - [Lighthouse](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686493708805&to=1686494277655&var-client_name=All) - [Teku](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=All&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686493701892&to=1686494380154&var-client_name=xatu-sentry-sfo3-mainnet-teku-002&var-client_name=xatu-sentry-sfo3-mainnet-teku-003&var-client_name=xatu-sentry-syd1-mainnet-teku-001&var-client_name=xatu-sentry-ams3-mainnet-teku-003&var-client_name=xatu-sentry-ams3-mainnet-teku-002&var-client_name=xatu-sentry-ams3-mainnet-teku-001&var-client_name=xatu-sentry-syd1-mainnet-teku-002&var-client_name=xatu-sentry-syd1-mainnet-teku-003) - [Lodestar](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lodestar&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686493674352&to=1686494385920&var-client_name=xatu-sentry-ams3-mainnet-lodestar-002&var-client_name=xatu-sentry-ams3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-002&var-client_name=xatu-sentry-syd1-mainnet-lodestar-003&var-client_name=xatu-sentry-syd1-mainnet-lodestar-002) - [Nimbus](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=nimbus&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686493674352&to=1686494385920&var-client_name=All&var-attestation_prefix=) #### Others ##### Set 1 Blocks 17456037 –> 17456040 Slots 6637693 -> 6637696 - [Prysm](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=xatu-sentry-syd1-mainnet-prysm-002&var-client_name=xatu-sentry-sfo3-mainnet-prysm-001&var-client_name=xatu-sentry-syd1-mainnet-prysm-001&var-client_name=xatu-sentry-ams3-mainnet-prysm-002&var-consensus_implementation=prysm&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686475877992&to=1686476842698) - [Lighthouse](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=All&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686475877992&to=1686476842698&var-attestation_prefix=) - [Lodestar](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lodestar&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686475867813&to=1686476854886&var-client_name=xatu-sentry-ams3-mainnet-lodestar-002&var-client_name=xatu-sentry-ams3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-002&var-client_name=xatu-sentry-syd1-mainnet-lodestar-003&var-client_name=xatu-sentry-syd1-mainnet-lodestar-002) - [Teku](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=All&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686475882681&to=1686476856617&var-client_name=xatu-sentry-sfo3-mainnet-teku-002&var-client_name=xatu-sentry-sfo3-mainnet-teku-003&var-client_name=xatu-sentry-syd1-mainnet-teku-001&var-client_name=xatu-sentry-ams3-mainnet-teku-003&var-client_name=xatu-sentry-ams3-mainnet-teku-002&var-client_name=xatu-sentry-ams3-mainnet-teku-001&var-client_name=xatu-sentry-syd1-mainnet-teku-002&var-client_name=xatu-sentry-syd1-mainnet-teku-003) - [Nimbus](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=All&var-consensus_implementation=nimbus&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686475877992&to=1686476842698&var-attestation_prefix=) ##### Set 2 Blocks 17456081 –> 17456083 Slots 6637738 -> 6637740 - [Prysm](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=xatu-sentry-syd1-mainnet-prysm-002&var-client_name=xatu-sentry-sfo3-mainnet-prysm-001&var-client_name=xatu-sentry-syd1-mainnet-prysm-001&var-client_name=xatu-sentry-ams3-mainnet-prysm-002&var-consensus_implementation=prysm&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686476396698&to=1686477527686) - [Lighthouse](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686476377190&to=1686477446234&var-client_name=All&var-attestation_prefix=) - [Teku](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=All&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686476400525&to=1686477359425&var-client_name=xatu-sentry-sfo3-mainnet-teku-002&var-client_name=xatu-sentry-sfo3-mainnet-teku-003&var-client_name=xatu-sentry-syd1-mainnet-teku-001&var-client_name=xatu-sentry-ams3-mainnet-teku-003&var-client_name=xatu-sentry-ams3-mainnet-teku-002&var-client_name=xatu-sentry-ams3-mainnet-teku-001&var-client_name=xatu-sentry-syd1-mainnet-teku-002&var-client_name=xatu-sentry-syd1-mainnet-teku-003) - [Lodestar](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lodestar&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686476377190&to=1686477446234&var-client_name=xatu-sentry-ams3-mainnet-lodestar-002&var-client_name=xatu-sentry-ams3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-002&var-client_name=xatu-sentry-syd1-mainnet-lodestar-003&var-client_name=xatu-sentry-syd1-mainnet-lodestar-002) - [Nimbus](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=nimbus&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686476377190&to=1686477446234&var-client_name=All&var-attestation_prefix=) ##### Set 3 Blocks 17457363 –> 17457369 Slots 6639039 -> 6639046 - [Prysm](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=xatu-sentry-syd1-mainnet-prysm-002&var-client_name=xatu-sentry-sfo3-mainnet-prysm-001&var-client_name=xatu-sentry-syd1-mainnet-prysm-001&var-client_name=xatu-sentry-ams3-mainnet-prysm-002&var-consensus_implementation=prysm&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686491634545&to=1686493443338) - [Lighthouse](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=All&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686491634545&to=1686493443338&var-attestation_prefix=) - [Teku](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=All&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686491985578&to=1686493106683&var-client_name=xatu-sentry-sfo3-mainnet-teku-002&var-client_name=xatu-sentry-sfo3-mainnet-teku-003&var-client_name=xatu-sentry-syd1-mainnet-teku-001&var-client_name=xatu-sentry-ams3-mainnet-teku-003&var-client_name=xatu-sentry-ams3-mainnet-teku-002&var-client_name=xatu-sentry-ams3-mainnet-teku-001&var-client_name=xatu-sentry-syd1-mainnet-teku-002&var-client_name=xatu-sentry-syd1-mainnet-teku-003) - [Lodestar](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lodestar&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686491992826&to=1686493167654&var-client_name=xatu-sentry-ams3-mainnet-lodestar-002&var-client_name=xatu-sentry-ams3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-002&var-client_name=xatu-sentry-syd1-mainnet-lodestar-003&var-client_name=xatu-sentry-syd1-mainnet-lodestar-002) - [Nimbus](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=All&var-consensus_implementation=nimbus&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1686491634545&to=1686493443338&var-attestation_prefix=) ## Sentry Nodes All data is derived from Sentry nodes running [Xatu](https://github.com/ethpandaops/xatu). Some sentries were not synced or had other issues not related to the test, and have been filtered out of the data. Running 3 of every CL in Amsterdam, San Francisco & Sydney. 1. Using a multiplexed EL (Nethermind) while subscribing to all attestation subnets 2. Using a multiplexed EL (Nethermind) while subscribing to the default amount of attestation subnets 3. Using a stubbed EL (Stubbies) while subscribing to the default amount of attestation subnets ### Xatu Dashboards **Note: May need to hit refresh (top right) on some of the snapshot panels if they are erroring** - [Prysm](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=xatu-sentry-syd1-mainnet-prysm-002&var-client_name=xatu-sentry-sfo3-mainnet-prysm-001&var-client_name=xatu-sentry-sfo3-mainnet-prysm-002&var-client_name=xatu-sentry-syd1-mainnet-prysm-001&var-client_name=xatu-sentry-ams3-mainnet-prysm-002&var-consensus_implementation=prysm&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253016820&to=1685259017758&var-attestation_prefix=2023_05_bigblocks_) - [Publicly accessible snapshot 28/05/23](https://grafana.observability.ethpandaops.io/dashboard/snapshot/BR6rVAbjDHxVl1xzs4MYpWmP6Rqj5ulK) - [Publicly accessible snapshot day 2 (earlier)](https://grafana.observability.ethpandaops.io/dashboard/snapshot/VMaTEevrp0cp2YN0S8jt8WPDrKM4gvtU) - [Publicly accessible snapshot day 2 (later)](https://grafana.observability.ethpandaops.io/dashboard/snapshot/JsHWocLqMG3FIXdYccX9j3Jn5yJlFwM9) - [Teku](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=All&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685252940297&to=1685259053897&var-client_name=xatu-sentry-sfo3-mainnet-teku-002&var-client_name=xatu-sentry-sfo3-mainnet-teku-003&var-client_name=xatu-sentry-syd1-mainnet-teku-001&var-client_name=xatu-sentry-ams3-mainnet-teku-003&var-client_name=xatu-sentry-ams3-mainnet-teku-002&var-client_name=xatu-sentry-ams3-mainnet-teku-001&var-client_name=xatu-sentry-syd1-mainnet-teku-002&var-client_name=xatu-sentry-syd1-mainnet-teku-003&var-attestation_prefix=2023_05_bigblocks_) - [Publicly accessible snapshot 28/05/23](https://grafana.observability.ethpandaops.io/dashboard/snapshot/t7P9uHnVo8tiWFBWZbmBV7eamaJVcRfN?orgId=1) - [Publicly accessible snapshot day 2 (earlier)](https://grafana.observability.ethpandaops.io/dashboard/snapshot/6o9tP3HCvtGerVRRVo15hM7Xa2SDYpzG) - [Publicly accessible snapshot day 2 (later)](https://grafana.observability.ethpandaops.io/dashboard/snapshot/WmHFXM5sCNasoxWMb3NTGwx2nA5PwSU4) - [Lighthouse](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253013835&to=1685259012742&var-client_name=All&var-attestation_prefix=2023_05_bigblocks_) - [Publicly accessible snapshot 28/05/23](https://grafana.observability.ethpandaops.io/dashboard/snapshot/JHHtPDrNROWyKIwo0jXlkA2WU774P290) - [Publicly accessible snapshot day 2 (earlier)](https://grafana.observability.ethpandaops.io/dashboard/snapshot/33K6tzvcj3qD1t4wixIsm1RAtvwJqCYr) - [Publicly accessible snapshot day 2 (later)](https://grafana.observability.ethpandaops.io/dashboard/snapshot/YJUZaT6wfqsnAk7WQa35KUigp3Ft3rCQ) - [Lodestar](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lodestar&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253176391&to=1685258784609&var-client_name=xatu-sentry-ams3-mainnet-lodestar-002&var-client_name=xatu-sentry-ams3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-002&var-client_name=xatu-sentry-syd1-mainnet-lodestar-003&var-client_name=xatu-sentry-syd1-mainnet-lodestar-002&var-attestation_prefix=2023_05_bigblocks_) - [Publicly accessible snapshot 28/05/23](https://grafana.observability.ethpandaops.io/dashboard/snapshot/uVLwwKjssNR7xR4aJmSt0i1asQMv1HJJ?orgId=1) - [Publicly accessible snapshot day 2 (earlier)](https://grafana.observability.ethpandaops.io/dashboard/snapshot/XFPo3ESwJAItknZsdKzu9Ho5uF7dLAMK) - [Publicly accessible snapshot day 2 (later)](https://grafana.observability.ethpandaops.io/dashboard/snapshot/SGVv2qYIxpGr192p5q1atXmoRVTsVU1R) - [Nimbus](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=nimbus&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685252997208&to=1685258975626&var-client_name=All&var-attestation_prefix=2023_05_bigblocks_) - [Publicly accessible snapshot 28/05/23](https://grafana.observability.ethpandaops.io/dashboard/snapshot/bXvXELX1iWeXcdgqXMLmAf64X1Ki5N7I?orgId=1) - [Publicly accessible snapshot day 2 (earlier)](https://grafana.observability.ethpandaops.io/dashboard/snapshot/P1IUiwvauwBmxst1UeDwM9D6ZWZF1trr) - [Publicly accessible snapshot day 2 (later)](https://grafana.observability.ethpandaops.io/dashboard/snapshot/daWUYqHyZm1Om2Rnr2EjeGw2OSvFSzOm) ### Initial receiving time for each test block The most obvious display of the big block test in action. Definite increase in arrival time for larger blocks across the board. ##### Prysm [Prysm](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=xatu-sentry-syd1-mainnet-prysm-002&var-client_name=xatu-sentry-sfo3-mainnet-prysm-001&var-client_name=xatu-sentry-sfo3-mainnet-prysm-002&var-client_name=xatu-sentry-syd1-mainnet-prysm-001&var-client_name=xatu-sentry-ams3-mainnet-prysm-002&var-consensus_implementation=prysm&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253016820&to=1685259017758&viewPanel=39&var-attestation_prefix=2023_05_bigblocks_) slight increase. Possibly the most accurate representation of the p2p layer as I believe Prysm emits `block` events immediately once the block is seen and sound enough to start verifying.  ##### Teku [Teku](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=All&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685252940297&to=1685259053897&var-client_name=xatu-sentry-sfo3-mainnet-teku-002&var-client_name=xatu-sentry-sfo3-mainnet-teku-003&var-client_name=xatu-sentry-syd1-mainnet-teku-001&var-client_name=xatu-sentry-ams3-mainnet-teku-003&var-client_name=xatu-sentry-ams3-mainnet-teku-002&var-client_name=xatu-sentry-ams3-mainnet-teku-001&var-client_name=xatu-sentry-syd1-mainnet-teku-002&var-client_name=xatu-sentry-syd1-mainnet-teku-003&viewPanel=39&var-attestation_prefix=2023_05_bigblocks_) saw a similar story  ##### Lighthouse [Lighthouse](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253013835&to=1685259012742&var-client_name=All&viewPanel=39&var-attestation_prefix=2023_05_bigblocks_) perhaps the most obvious client to see the increase, but Lighthouse won't emit `block` events until the entire block handling process has completed (including any EL interaction).  ##### Lodestar [Lodestar](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lodestar&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253176391&to=1685258784609&var-client_name=xatu-sentry-ams3-mainnet-lodestar-002&var-client_name=xatu-sentry-ams3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-002&var-client_name=xatu-sentry-syd1-mainnet-lodestar-003&var-client_name=xatu-sentry-syd1-mainnet-lodestar-002&viewPanel=39&var-attestation_prefix=2023_05_bigblocks_) similar to others.  ##### Nimbus [Nimbus](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=nimbus&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685252997208&to=1685258975626&var-client_name=All&var-attestation_prefix=2023_05_bigblocks_) saw propogation blow outs for the 2nd set of 1MB tests (may be an implementation detail of Nimbus if they emit beacon events are completely handling the incoming block). All other sentries saw a general increase in block propogation time: e.g. [Lighthouse Sentries](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253013835&to=1685259012742&var-client_name=All&var-attestation_prefix=2023_05_bigblocks_) ### Bandwidth usage changes  Can be clearly seen on nodes that aren't subscribed to all subnets: - [Eleel EL, default attestation subnets](https://grafana.observability.ethpandaops.io/d/GhtVssw4z/mainnet-big-blocks-testing?orgId=1&viewPanel=3) - [Stubbed EL, default attestation subnets](https://grafana.observability.ethpandaops.io/d/GhtVssw4z/mainnet-big-blocks-testing?orgId=1&viewPanel=2) - [Eleel EL, all attestation subnets](https://grafana.observability.ethpandaops.io/d/GhtVssw4z/mainnet-big-blocks-testing?orgId=1&viewPanel=4) Toggle `Slot annotations` at the top to see times of the test. ### Attestation propagation time Hard to determine definitively what the actual effect was due to implementation differences. Peer counts may also play a part - some clients have higher/lower peer counts by default. Lighthouse was the most interesting client here. ##### Prysm [Prysm](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=xatu-sentry-syd1-mainnet-prysm-002&var-client_name=xatu-sentry-sfo3-mainnet-prysm-001&var-client_name=xatu-sentry-sfo3-mainnet-prysm-002&var-client_name=xatu-sentry-syd1-mainnet-prysm-001&var-client_name=xatu-sentry-ams3-mainnet-prysm-002&var-consensus_implementation=prysm&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253016820&to=1685259017758&viewPanel=75&var-attestation_prefix=2023_05_bigblocks_) didn't really observe any increase in attestation propogation timings:  ##### Lighthouse [Lighthouse](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253013835&to=1685259012742&var-client_name=All&viewPanel=75&var-attestation_prefix=2023_05_bigblocks_) definitely saw an increase in attestation propogation through the periods of the test. Can be seen in the heatmap, as well as the `min`/`p05` for the durations of each test.  ##### Lodestar [Lodestar](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lodestar&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253176391&to=1685258784609&var-client_name=xatu-sentry-ams3-mainnet-lodestar-002&var-client_name=xatu-sentry-ams3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-002&var-client_name=xatu-sentry-syd1-mainnet-lodestar-003&var-client_name=xatu-sentry-syd1-mainnet-lodestar-002&viewPanel=75&var-attestation_prefix=2023_05_bigblocks_) saw an increase, but the only events we receive from Lodestar at the moment are attestations that were derived from the incoming beacon block (no p2p raw/aggregates). So this can be attributed directly to an increase in block propogation time.  ##### Teku [Teku](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=All&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685252940297&to=1685259053897&var-client_name=xatu-sentry-sfo3-mainnet-teku-002&var-client_name=xatu-sentry-sfo3-mainnet-teku-003&var-client_name=xatu-sentry-syd1-mainnet-teku-001&var-client_name=xatu-sentry-ams3-mainnet-teku-003&var-client_name=xatu-sentry-ams3-mainnet-teku-002&var-client_name=xatu-sentry-ams3-mainnet-teku-001&var-client_name=xatu-sentry-syd1-mainnet-teku-002&var-client_name=xatu-sentry-syd1-mainnet-teku-003&viewPanel=75&var-attestation_prefix=2023_05_bigblocks_) very similar story to Prysm - nothing really observed  ##### Nimbus [Nimbus](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=nimbus&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685252997208&to=1685258975626&var-client_name=All&viewPanel=75&var-attestation_prefix=2023_05_bigblocks_) Similar to Prysm/Teku, nothing really observed  ## Chain Data ### Blocks making it into canonical chain Missed/orphaned blocks in the 2nd set of 1mb tests: - [Block-seen timings for each sentry in that slot range](https://docs.google.com/spreadsheets/d/1M_SRbIKMtlDwAMpaUoBIO7I8ywgxKAru-oPNtJNCXeo/edit#gid=0) - [Forky timestamp](https://forky.mainnet.ethpandaops.io/?t=1685258422268) ### Attestations making it into canonical chain #### Correctness (head, source, target) Can be roughly derived from the Xatu "Attestation agreement" panel, which finds the percentage share of attestations that each `beacon block root` contributes to the total number of attestations at each specific point in time. "For a given slot, what beacon block roots had attestations against them" ##### Prysm [Prysm](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=xatu-sentry-syd1-mainnet-prysm-002&var-client_name=xatu-sentry-sfo3-mainnet-prysm-001&var-client_name=xatu-sentry-sfo3-mainnet-prysm-002&var-client_name=xatu-sentry-syd1-mainnet-prysm-001&var-client_name=xatu-sentry-ams3-mainnet-prysm-002&var-consensus_implementation=prysm&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253016820&to=1685259017758&viewPanel=117&var-attestation_prefix=2023_05_bigblocks_) definitely saw a slight increase in attestation disagreement during the periods of the test, although nothing outside of the normal noise range  ##### Teku [Teku](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=All&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685252940297&to=1685259053897&var-client_name=xatu-sentry-sfo3-mainnet-teku-002&var-client_name=xatu-sentry-sfo3-mainnet-teku-003&var-client_name=xatu-sentry-syd1-mainnet-teku-001&var-client_name=xatu-sentry-ams3-mainnet-teku-003&var-client_name=xatu-sentry-ams3-mainnet-teku-002&var-client_name=xatu-sentry-ams3-mainnet-teku-001&var-client_name=xatu-sentry-syd1-mainnet-teku-002&var-client_name=xatu-sentry-syd1-mainnet-teku-003&viewPanel=117&var-attestation_prefix=2023_05_bigblocks_) similar to Prysm. Definite increase in attestation disagreement during the tests, but nothing outside of the normal noise range  ##### Lighthouse [Lighthouse](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253013835&to=1685259012742&var-client_name=All&viewPanel=117&var-attestation_prefix=2023_05_bigblocks_) Same as above  ##### Lodestar [Lodestar](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lodestar&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253176391&to=1685258784609&var-client_name=xatu-sentry-ams3-mainnet-lodestar-002&var-client_name=xatu-sentry-ams3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-002&var-client_name=xatu-sentry-syd1-mainnet-lodestar-003&var-client_name=xatu-sentry-syd1-mainnet-lodestar-002&viewPanel=117&var-attestation_prefix=2023_05_bigblocks_) provides a slightly more interesting view since we are only seeing attestations derived from the beacon block. This view of the world shows us what was actually packed in to the chain. Fairly similar end result to the other clients however.  ##### Nimbus [Nimbus](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=nimbus&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685252997208&to=1685258975626&var-client_name=All&viewPanel=117&var-attestation_prefix=2023_05_bigblocks_) fairly similar view to Prysm/Teku/Lighthouse.  #### Inclusion delay Can be derived from the Xatu panel "Attestation seen distance". "Similar to inclusion distance but instead of when it was included in a block, it's instead when a sentry saw a attestation and the "distance" between its target and the current slot. "0" being the attestation for the current slot, "1" being current slot - 1 etc." ##### Prysm [Prysm](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-client_name=xatu-sentry-syd1-mainnet-prysm-002&var-client_name=xatu-sentry-sfo3-mainnet-prysm-001&var-client_name=xatu-sentry-sfo3-mainnet-prysm-002&var-client_name=xatu-sentry-syd1-mainnet-prysm-001&var-client_name=xatu-sentry-ams3-mainnet-prysm-002&var-consensus_implementation=prysm&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253016820&to=1685259017758&viewPanel=151&var-attestation_prefix=2023_05_bigblocks_) saw nothing out of the ordinary  ##### Teku [Teku](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=All&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685252940297&to=1685259053897&var-client_name=xatu-sentry-sfo3-mainnet-teku-002&var-client_name=xatu-sentry-sfo3-mainnet-teku-003&var-client_name=xatu-sentry-syd1-mainnet-teku-001&var-client_name=xatu-sentry-ams3-mainnet-teku-003&var-client_name=xatu-sentry-ams3-mainnet-teku-002&var-client_name=xatu-sentry-ams3-mainnet-teku-001&var-client_name=xatu-sentry-syd1-mainnet-teku-002&var-client_name=xatu-sentry-syd1-mainnet-teku-003&viewPanel=151&var-attestation_prefix=2023_05_bigblocks_) same as above  ##### Lighthouse [Lighthouse](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lighthouse&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253013835&to=1685259012742&var-client_name=All&viewPanel=151&var-attestation_prefix=2023_05_bigblocks_) same story  ##### Lodestar [Lodestar](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=lodestar&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685253176391&to=1685258784609&var-client_name=xatu-sentry-ams3-mainnet-lodestar-002&var-client_name=xatu-sentry-ams3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-003&var-client_name=xatu-sentry-sfo3-mainnet-lodestar-002&var-client_name=xatu-sentry-syd1-mainnet-lodestar-003&var-client_name=xatu-sentry-syd1-mainnet-lodestar-002&viewPanel=151&var-attestation_prefix=2023_05_bigblocks_) again provides a different view of the world only showing us attestations from the beacon block, but doesn't really reveal anything out of the ordinary.  ##### Nimbus [Nimbus](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&var-network_name=mainnet&var-consensus_implementation=nimbus&var-consensus_version=All&var-geo_continent_code=All&var-heatmap_interval=250&var-interval_tight=$__auto_interval_interval_tight&var-interval=$__auto_interval_interval&var-interval_loose=$__auto_interval_interval_loose&var-table_prefix=&from=1685252997208&to=1685258975626&var-client_name=All&viewPanel=151&var-attestation_prefix=2023_05_bigblocks_) slightly different to the other clients, likely just showing a different in implementation of when these events are emitted.