# Block observability
With the push towards 100M gas, knowing processing times for blocks is critical. Getting execution clients to expose this data will be a useful tool for observing performance on a larger scale.
## The problem
Currently there are limited ways to observe how long a block takes to process;
- logs - have to parse logs to extract the data
- metrics - over time, can't be used to observe a single block
- distributed tracing - does any client implement this?
## Possible solution
Whatever the solution may be, we need to be able to observe the following;
- total block processing time eg. `engine_newPayload*` call
- total head processing time eg. `engine_forkchoiceUpdated*` call
These are good to have;
- total execution time (per block)
- total validation time (per block)
- total persist time (per block)
These are nice to have;
- total execution time (per transaction)
### Extend `ethstats`
[`ethstats`](https://geth.ethereum.org/docs/monitoring/ethstats) is a way for clients to push data to a remote server. It's a simple websocket connection that pushes various events that is lightweight and opt in for clients.
#### Who currently supports ethstats
All clients (pending reth PR)
- Geth
- Nethermind
- Erigon
- Besu
- Reth
#### Existing event `block`
We have the existing event `block` that is emitted when `engine_forkchoiceUpdated*` is called and the new head is set.
Example payload (can differ per client);
```json
{
"id": "some-node",
"number": 2,
"hash": "0x0b703bbb27159582ca31033cc56c43bc6649d960a75e4d4adf2b7d5b6ff5b0d2",
"parentHash": "0x5919146caaa3e4d89081f92856ddfc83ec1d7271b31a03f200820deba6ae0331",
"timestamp": 1750997318,
"miner": "0x8943545177806ed17b9f23f0a21ee5948ecaa776",
"gasUsed": 7832767,
"gasLimit": 60000000,
"difficulty": "0",
"totalDifficulty": "0",
"transactions": [
{
"hash": "0x3412b3fe28546657e7da297a70375c0191b967507a4bb118f0ae0f85157b4d66"
},
{
"hash": "0xcdb5e325f69fcc180c590984427454c864410394df987c6a1c272147ef6c2c48"
}
],
"transactionsRoot": "0xe8bf6ddde7488abcbd1666ee2d6a093ca42017b87592ae891ba2b71ed52fe600",
"stateRoot": "0xd45793c9d59d39c5149e3a47c7fd9d18543a032f26955aa16fc6199d00e363f0",
"uncles": [],
}
```
We could extend this event to include the processing time here to make the block canonical eg.
- `processingTimeMs` - the total time to set the block as canonical
#### New event `block_new_payload`
This event is emitted when `engine_newPayload*` is called and the block is processed.
Example payload;
```json
{
"id": "some-node",
"number": 112,
"number": 2,
"hash": "0x0b703bbb27159582ca31033cc56c43bc6649d960a75e4d4adf2b7d5b6ff5b0d2",
"timestamp": 1750997318,
"gasUsed": 7832767,
"gasLimit": 60000000,
"transactions": [
{
"hash": "0x3412b3fe28546657e7da297a70375c0191b967507a4bb118f0ae0f85157b4d66"
},
{
"hash": "0xcdb5e325f69fcc180c590984427454c864410394df987c6a1c272147ef6c2c48"
}
],
"transactionsRoot": "0xe8bf6ddde7488abcbd1666ee2d6a093ca42017b87592ae891ba2b71ed52fe600",
"stateRoot": "0xd45793c9d59d39c5149e3a47c7fd9d18543a032f26955aa16fc6199d00e363f0",
"executionTimeMs": 1000,
"validationTimeMs": 1000,
"writeTimeMs": 1000,
"totalTimeMs": 3000
}
```
The key fields here are;
- `executionTimeMs` - the time to execute all transactions
- `validationTimeMs` - the time to validate the final state after execution
- `writeTimeMs` - the time persist the block
- `processingTimeMs` - the total time to process the block
#### Implementation
- `processingTimeMs` for `block` and `block_new_payload` should be ok to implement as it could just wrap the `engine_newPayload*`/`engine_forkchoiceUpdated*` calls
- `executionTimeMs`/`validationTimeMs`/`writeTimeMs` will be hard to implement unless the client already has this data available and exposed
#### Backwards compatibility
- `block` event should be fine as it's just adding a new field
- `block_new_payload` event should be fine as its just adding a new event
## How will this data be consumed
- Any existing ethstats servers can be updated to handle the new data
- [Contributoor](https://github.com/ethpandaops/contributoor), which currently can be run by home stakers beside a beacon node, could be extended to include an `ethstats` server. This data could then be added to the [public](https://ethpandaops.io/data/xatu/) dataset for analysis (if the *contributoor* **opts in**). There is also an added benefit here where we could connect data from a *contributoors* beacon node client and execution client, giving a better view into the processing time of beacon block.