# Testing overview doc This document is meant to be a guide to all the tools used across the various testing, devops and security teams. ### Ethereum stack: #### Each node: - EVM - Spec tests - State transition - (JSON/Engine/Builder)-API spec compliance - Block building #### Network level: - Client interop - P2P - Syncing - Network level agreement on events (deposits/withdrawals/slashings) - MEV workflow ## Tools: ### Execution spec tests (EEST): **Summary**: Execution layer test repository and tools framework **Description**: Contains EVM test vectors written in python, and tools to fill them into several fixture types that can be consumed by either Hive or the clients themselves. The filling engine is powered by the [pytest library](https://docs.pytest.org/en/8.0.x/) which allows for test vector parameterization with a few lines of code, with improved readability (when compared to the [previous JSON/YAML fillers](https://github.com/ethereum/tests/tree/develop/src)). **Target layers**: EVM, Spec tests **Repository**: https://github.com/ethereum/execution-spec-tests/ **Point of contact**: [Dan](https://github.com/danceratopz), [Spencer](https://github.com/spencer-tb), [Dimitry](https://github.com/winsvega), [Mario](https://github.com/marioevz) **Integrations**: [Hive](#Hive) **Note**: ### Consensus spec tests: **Summary**: Consensus layer specification files **Description**: Consensus layer executable specification, which consists of the beacon and networking logic written in markdown files with embedded python code inside. The [markdown files](https://github.com/ethereum/consensus-specs/tree/dev/specs) are automatically parsed to generate a python scripts per each fork with the appropriate iterative logic modifications. The repository also contains test logic that is used in conjunction with the beacon chain executable logic to produce test fixtures in different [formats](https://github.com/ethereum/consensus-specs/tree/dev/tests/formats). **Target layers**: Spec tests, State transition (Consensus) **Repository**: https://github.com/ethereum/consensus-specs **Point of contact**: [Hsiao-Wei](https://github.com/hwwhww), [Alex](https://github.com/ralexstokes), [Mikhail](https://github.com/mkalinin) **Integrations**: **Note**: ### EVM Transition Tool: **Summary**: Execution layer state transition function tool **Description**: Performs a single state transition calculation based on a given input, conformed by an environment (fork schedule, block number, timestamp, etc) and a list of transactions, and generates the state root, receipts, and logs calculations, all of which can be later used to fill tests. **Target layers**: State transition **Repository**: - Geth: https://github.com/ethereum/go-ethereum/tree/master/cmd/evm - EELS: https://github.com/ethereum/execution-specs/tree/master/src/ethereum_spec_tools/evm_tools - evmone: https://github.com/ethereum/evmone **Point of contact**: [Martin](https://github.com/holiman) (Geth), [Guru](https://github.com/gurukamath) (EELS), [Paweł](https://github.com/chfast) (evmone) **Integrations**: [EEST](#Execution-spec-tests-EEST) **Note**: ### Hive: **Summary**: End-to-end multi-client test harness **Description**: Composed by two parts: (1) the simulators are self-contained programs that run within a docker instance and orchestrate different test scenarios that involve 1 or muiltiple Ethereum clients, execution or consensus (2) the server that listens to requests by the simulators to start/end test suites/cases, start/end client instances, and configure them. Hive is more adequate for deterministic tests that have a known testing target, and they do not involve chaos testing and any kind of stochastic behavior is reduced to a minimum. **Target layers**: ~All **Repository**: https://github.com/ethereum/hive **Point of contact**: [Felix](https://github.com/fjl), [Mario](https://github.com/marioevz) **Integrations**: [EEST](#Execution-spec-tests-EEST) **Note**: ### FuzzyVM: **Summary**: Differential fuzzer for the EVM **Description**: The fuzzer generates state tests that can be used to differential fuzz EVM implimentations. The text execution however is handled by goevmlab. **Target layers**: EVM **Repository**: https://github.com/MariusVanDerWijden/FuzzyVM **Point of contact**: [Marius](https://github.com/MariusVanDerWijden) **Integrations**: **Note**: ### Kurtosis: **Summary**: Easy to configure Ethereum network tool **Description**: Kurtosis allows us to define the network parameters as a simple YAML file and it deploys the test network to a docker(local) or kubernetes(remote) backend allowing us to have semi-reproducible tests both locally and at scale. Kurtosis also allows us to integrate with a lot of tools to enable a wider range of testing without the need for redefining commong layers. The definitions are written in starlark and can be extended based on the need of the user. The YAML files can be passed amongst users to get a semi-reproducible error for debugging. **Target layers**: ~All **Repository**: https://github.com/kurtosis-tech/ethereum-package **Point of contact**: [Barnabas](https://github.com/barnabasbusa),[Pari](https://github.com/parithosh) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis),[Blobber](https://notes.ethereum.org/@parithosh/testing-overview-doc#Blobber),[Mock-builder](https://notes.ethereum.org/@parithosh/testing-overview-doc#Mock-builder),[Assertoor](https://notes.ethereum.org/@parithosh/testing-overview-doc#Assertoor),[Forky](https://notes.ethereum.org/@parithosh/testing-overview-doc#Forky),[Xatu](https://notes.ethereum.org/@parithosh/testing-overview-doc#Xatu),[Tracoor](https://notes.ethereum.org/@parithosh/testing-overview-doc#Tracoor),[Dora](https://notes.ethereum.org/@parithosh/testing-overview-doc#Dora),[Goomy-blob](https://notes.ethereum.org/@parithosh/testing-overview-doc#Goomy-blob) **Note**: ### Public testnets: **Summary**: Publically accessible network running the latest fork we want to test **Description**: The public testnets are a deployment that mimics a real network spread across cloud instances across the world. We can modify every parameter for the testing need at the time. These machines are setup purely with ansible and docker, to facilitate easy debugging. Since the instances are public, anyone can perform sync/spam/other tests on their own. This is also an extremely useful resource for integrations to the wider ecosystem. **Target layers**: ~All **Repository**: https://github.com/ethpandaops/template-devnets **Point of contact**: [Pari](https://github.com/parithosh), [Barnabas]([https:](https://github.com/barnabasbusa)) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis),[Blobber](https://notes.ethereum.org/@parithosh/testing-overview-doc#Blobber),[Mock-builder](https://notes.ethereum.org/@parithosh/testing-overview-doc#Mock-builder),[Assertoor](https://notes.ethereum.org/@parithosh/testing-overview-doc#Assertoor),[Forky](https://notes.ethereum.org/@parithosh/testing-overview-doc#Forky),[Xatu](https://notes.ethereum.org/@parithosh/testing-overview-doc#Xatu),[Tracoor](https://notes.ethereum.org/@parithosh/testing-overview-doc#Tracoor),[Dora](https://notes.ethereum.org/@parithosh/testing-overview-doc#Dora),[Goomy-blob](https://notes.ethereum.org/@parithosh/testing-overview-doc#Goomy-blob) **Note**: ### Mock-builder: **Summary**: Mock implimentation of the Builder API **Description**: The mock builder is useful for testing if the clients have implimented the builder API before the relay infra is ready for testing. This is mostly used in the early days of a new fork, where the spec is still actively changing. Its also extremely useful for prototyping changes to the spec. **Target layers**: (JSON/Engine/Builder)-API spec compliance, Block building, MEV workflow **Repository**: https://github.com/marioevz/mock-builder/ **Point of contact**: [Mario](https://github.com/marioevz) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis), [Public testnets](https://notes.ethereum.org/@parithosh/testing-overview-doc#Public-testnets), [Hive](https://notes.ethereum.org/@parithosh/testing-overview-doc#Hive), [Attacknet](https://notes.ethereum.org/@parithosh/testing-overview-doc#Attacknet) **Note**: ### Blobber: **Summary**: Blob p2p edge case testing tool **Description**: The blobber acts as a [MITM](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) between the validator and the beacon client. The blobber intercepts the block production messages and executes a list of actions that it has been configured to perform. This could vary from sending conflicting blobs to delayed propagation of the blob. Blobber additionally connects directly to peer beacon node via the p2p layer, allowing it to bypass the beacon message verifications - this alows us to propagate invalid blobs into the network. **Target layers**: (JSON/Engine/Builder)-API spec compliance, Block building, p2p **Repository**: https://github.com/marioevz/blobber/ **Point of contact**: [Mario](https://github.com/marioevz) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis), [Public testnets](https://notes.ethereum.org/@parithosh/testing-overview-doc#Public-testnets), [Attacknet](https://notes.ethereum.org/@parithosh/testing-overview-doc#Attacknet) **Note**: ### Assertoor: **Summary**: Integration testing tool for Ethereum networks **Description**: Assertoor can be used to assert any network behaviour as a standalone test. This test could for example be inclusion of transactions, every client proposing a block or deposits/withdrawald(Imagine Hive, but with a full testnet underneath). Assertoor is a standalone test assertion tool, meaning it can be deployed against public networks or local networks. The tests can be configured at runtime or via an API, the results can be viewed via the webpage or the API. Assertoor can also be configured as a github CI action to run on commits. **Target layers**: Network level agreement on events, p2p, Client interop **Repository**: https://github.com/ethpandaops/assertoor **Point of contact**: [pk910](https://github.com/pk910) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis), [Public testnets](https://notes.ethereum.org/@parithosh/testing-overview-doc#Public-testnets), [Attacknet](https://notes.ethereum.org/@parithosh/testing-overview-doc#Attacknet) **Note**: https://github.com/ethpandaops/kurtosis-assertoor-github-action ### Attacknet: **Summary**: Chaos engineering framework for Ethereum networks **Description**: Attacknet uses kurtosis underneath to spin up a network on a kubernetes cluster, it then uses Chaos mesh to orchestrate chaos engineering strategies on the cluster. This allows us to simulate real world edge cases on a test network, e.g: Delayed message passing, dropped packets, hard restarting nodes and so on. **Target layers**: p2p, client interop, network level agreement on events **Repository**: https://github.com/crytic/attacknet **Point of contact**: [Pari](https://github.com/parithosh), [Ben](https://github.com/bsamuels453) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis),[Assertoor](https://notes.ethereum.org/@parithosh/testing-overview-doc#Assertoor) **Note**: ### Forky: **Summary**: Forkchoice visualizer for the Beaconchain **Description**: Forky uses the standard Ethereum Beacon API to fetch the forkchoice of a node and visualizes it as a diagram for debugging. You can also upload a saved forkchoice output to generalize a visual. **Target layers**: p2p, network level agreement on events **Repository**: https://github.com/ethpandaops/forky **Point of contact**: [Sam](https://github.com/samcm),[Savid](https://github.com/Savid) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis),[Public testnets](https://notes.ethereum.org/@parithosh/testing-overview-doc#Public-testnets) **Note**: https://forky.mainnet.ethpandaops.io/ ### Xatu: **Summary**: Ethereum network monitoring with collection clients and a centralized server for data pipelining **Description**: Xatu is split into many components, the Xatu client connects to a beacon node and forwards the annotated event stream to the Xatu server. The Xatu server aggregates and stores these events in a database along with a data pipeline. This data can then be queried for research and for network health analysis purposes. **Target layers**: p2p **Repository**: https://github.com/ethpandaops/xatu/ **Point of contact**: [Sam](https://github.com/samcm),[Savid](https://github.com/Savid) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis),[Public testnets](https://notes.ethereum.org/@parithosh/testing-overview-doc#Public-testnets) **Note**: https://notes.ethereum.org/@ethpandaops/xatu-overview ### Tracoor: **Summary**: Collects and visualizes trace information to help the debugging process **Description**: Tracoor connects to beacon and execution nodes and collects a trace as well as a `ssz` state from every EL/CL client pair. It then offers this information in a handy website to help offer all the debug data needed during network issues. **Target layers**: network level agreement on events, client interop **Repository**: https://github.com/ethpandaops/tracoor **Point of contact**: [Sam](https://github.com/samcm),[Savid](https://github.com/Savid) **Integrations**: [Public testnets](https://notes.ethereum.org/@parithosh/testing-overview-doc#Public-testnets) **Note**: https://tracoor.mainnet.ethpandaops.io/ ### Dora: **Summary**: Lightweight beaconchain explorer **Description**: Dora is a lightweight beaconchain explorer meant to keep up with the active fork changes of Ethereum. The explorer helps make debugging easier and the database of the explorer can directly be queried to answer some network health questions and to surface issues. **Target layers**: client interop **Repository**: https://github.com/ethpandaops/dora **Point of contact**: [pk910](https://github.com/pk910) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis),[Public testnets](https://notes.ethereum.org/@parithosh/testing-overview-doc#Public-testnets) **Note**: https://dora.holesky.ethpandaops.io/ ### Goomy-blob: **Summary**: Tool to generate random blob transtactions **Description**: Goomy can be configured to generate random blob load for testing as well as certain spam tests with a focus on replacements and invalid blobs. **Target layers**: p2p, client interop **Repository**: https://github.com/ethpandaops/goomy-blob **Point of contact**: [pk910](https://github.com/pk910) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis),[Assertoor](https://notes.ethereum.org/@parithosh/testing-overview-doc#Assertoor),[Public testnets](https://notes.ethereum.org/@parithosh/testing-overview-doc#Public-testnets) **Note**: ### Snapshotter: **Summary**: Takes a snapshot of an Ethereum node for quicker shadowforks **Description**: Snapshotter takes the snapshot of a node and compresses it for storage in s3. We can use this stored file to start a fresh node from the snapshot state, allowing us to perform shadowforks really quickly. **Target layers**: client interop **Repository**: https://github.com/ethpandaops/snapshotter **Point of contact**: [Rafael](https://github.com/skylenet/) **Integrations**: [Kurtosis](https://notes.ethereum.org/@parithosh/testing-overview-doc#Kurtosis),[Assertoor](https://notes.ethereum.org/@parithosh/testing-overview-doc#Assertoor) **Note**: