-
-
owned this note
-
Published
Linked with GitHub
# Interop-tools
Find the interop event milestones [here](https://notes.ethereum.org/@timbeiko/nyota-milestones)
## Intro
Please ensure your local machine has the below tools installed and configured:
- [Docker](https://docs.docker.com/engine/install/)
- [Kurtosis](https://docs.kurtosis.com/quickstart/)
Quickstart links:
- [Docker-cache](https://notes.ethereum.org/@ethpandaops/interop-tools#Docker)
- [Kurtosis TLDR](https://notes.ethereum.org/@ethpandaops/interop-tools#Kurtosis-TLDR)
- [Builder-bot](https://notes.ethereum.org/@ethpandaops/interop-tools#Builder-bot)
- [Pectra](https://notes.ethereum.org/@ethpandaops/interop-tools#Pectra)
- [PeerDAS](https://notes.ethereum.org/@ethpandaops/interop-tools#PeerDAS)
- [Verkle-at-genesis](https://notes.ethereum.org/@ethpandaops/interop-tools#Verkle-at-genesis)
- [Verkle-transition](https://notes.ethereum.org/@ethpandaops/interop-tools#Verkle-transition)
- [EL testnet data snapshots](https://notes.ethereum.org/@ethpandaops/interop-tools#Snapshots)
- [Kurtosis-advanced](https://notes.ethereum.org/@ethpandaops/interop-tools#Advanced-kurtosis)
## Docker
### Docker registry cache
To save some internet bandwidth, we've set up a pull-through cache to limit the number of requests made to the internet when pulling images from Docker Hub.
#### Prerequisite
You'll need to update your Docker Engine/Daemon configuration to allow pulling images from an insecure registry. The registry is considered "insecure" because the images will be pulled unencrypted via HTTP. There's no TLS setup provided for HTTPS. We're not going to be setting up HTTPs purely to reduce overheads for the interop event.
Add the following config to Docker Engine/Daemon:
```json
"insecure-registries": [
"192.168.45.152:80"
]
```
If you're using Docker for Desktop, you can find the config in "Settings" > "Docker Engine"
For linux you can edit your daemon.json file to add the insecure-registries.
```bash
sudo vi /etc/docker/daemon.json
```
```json
{
"insecure-registries": [
"192.168.45.152:80"
]
}
```
```bash
sudo systemctl restart docker
```
#### Pull kurtosis related images
To avoid getting docker hub api rate limited you can pull the kurtosis images via the cache
```bash
docker pull 192.168.45.152:80/dh/timberio/vector:0.31.0-debian
docker pull 192.168.45.152:80/dh/traefik:2.10.6
docker pull 192.168.45.152:80/dh/alpine:3.17
docker pull 192.168.45.152:80/dh/kurtosistech/engine:0.89.9
docker pull 192.168.45.152:80/dh/fluent/fluent-bit:1.9.7
docker pull 192.168.45.152:80/dh/kurtosistech/core:0.89.9
docker pull 192.168.45.152:80/dh/python:3.11-alpine
docker pull 192.168.45.152:80/dh/ethpandaops/rpc-snooper:latest
docker pull 192.168.45.152:80/dh/ethpandaops/ethereum-genesis-generator:3.0.0-rc.19
docker pull 192.168.45.152:80/dh/ethpandaops/ethereum-genesis-generator:3.1.5
docker pull 192.168.45.152:80/dh/ethpandaops/ethereum-genesis-generator:2.0.12
docker pull 192.168.45.152:80/dh/ethpandaops/ethereum-genesis-generator:verkle-gen-v1.0.0
docker pull 192.168.45.152:80/dh/badouralix/curl-jq
docker pull 192.168.45.152:80/dh/ethpandaops/dora:peer-das
docker pull 192.168.45.152:80/dh/ethpandaops/dora:electra-support
docker pull 192.168.45.152:80/dh/ethpandaops/dora:master
docker pull 192.168.45.152:80/dh/ethpandaops/goomy-blob:master
docker pull 192.168.45.152:80/dh/protolambda/eth2-val-tools:latest
docker pull 192.168.45.152:80/dh/kurtosistech/files-artifacts-expander:0.89.9
docker pull 192.168.45.152:80/dh/library/httpd:latest
docker pull 192.168.45.152:80/dh/library/redis:alpine
docker pull 192.168.45.152:80/dh/library/postgres:alpine
docker pull 192.168.45.152:80/dh/ethpandaops/flashbots-builder:main
docker pull 192.168.45.152:80/dh/ethpandaops/mev-rs:main
docker pull 192.168.45.152:80/dh/ethpandaops/mev-rs:main-minimal
docker pull 192.168.45.152:80/dh/ethpandaops/tx-fuzz:master
```
### Examples
#### Example pulling `alpine:latest`
```bash
# Directly from Docker Hub (Please avoid using this)
docker pull alpine:latest
# With pull-through registry
docker pull 192.168.45.152:80/dh/library/alpine:latest
```
#### Example pulling `ethpandaops/geth:master`
```bash
# Directly from Docker Hub (Please avoid using this)
docker pull ethpandaops/geth:master
# With pull-through registry
docker pull 192.168.45.152:80/dh/ethpandaops/geth:master
```
## Kurtosis TLDR:
- Local testing tool
- Specify network definition as a YAML file
- Spin up network with `kurtosis run github.com/kurtosis-tech/ethereum-package --args-file filename.yaml --image-download always`
- Exhaustive configuration options found [here](https://github.com/kurtosis-tech/ethereum-package)
## Pectra
### Info
- [Spec sheet - pectra-devnet-0](https://notes.ethereum.org/@ethpandaops/pectra-devnet-0)
- [Interop milestones](https://notes.ethereum.org/@timbeiko/pectra-interop-milestones)
### Kurtosis config example
:::info
This config depends on the local docker registry cache being configured on your machine. See [docker](#Docker) for more info.
:::
```yaml=
participants:
- el_type: nethermind
el_image: 192.168.45.152:80/dh/nethermindeth/nethermind:pectra
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:ef-tests-electra
- el_type: nethermind
el_image: 192.168.45.152:80/dh/nethermindeth/nethermind:pectra
cl_type: teku
cl_image: 192.168.45.152:80/dh/ethpandaops/teku:master
- el_type: ethereumjs
el_image: 192.168.45.152:80/dh/ethpandaops/ethereumjs:master
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:ef-tests-electra
- el_type: ethereumjs
el_image: 192.168.45.152:80/dh/ethpandaops/ethereumjs:master
cl_type: teku
cl_image: 192.168.45.152:80/dh/ethpandaops/teku:master
- el_type: nethermind
el_image: 192.168.45.152:80/dh/nethermindeth/nethermind:pectra
cl_type: prysm
cl_image: 192.168.45.152:80/dh/ethpandaops/prysm-beacon-chain:electra-devnet0
vc_image: 192.168.45.152:80/dh/ethpandaops/prysm-validator:electra-devnet0
- el_type: ethereumjs
el_image: 192.168.45.152:80/dh/ethpandaops/ethereumjs:master
cl_type: prysm
cl_image: 192.168.45.152:80/dh/ethpandaops/prysm-beacon-chain:electra-devnet0
vc_image: 192.168.45.152:80/dh/ethpandaops/prysm-validator:electra-devnet0
network_params:
electra_fork_epoch: 1
additional_services:
- dora
- apache
- assertoor
snooper_enabled: true
assertoor_params:
run_stability_check: false
run_block_proposal_check: false
```
#### Advanced config:
```yaml=
participants:
- el_type: nethermind
el_image: 192.168.45.152:80/dh/nethermindeth/nethermind:pectra
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:ef-tests-electra
- el_type: nethermind
el_image: 192.168.45.152:80/dh/nethermindeth/nethermind:pectra
cl_type: teku
cl_image: 192.168.45.152:80/dh/ethpandaops/teku:master
- el_type: nethermind
el_image: 192.168.45.152:80/dh/nethermindeth/nethermind:pectra
cl_type: prysm
cl_image: 192.168.45.152:80/dh/ethpandaops/prysm-beacon-chain:electra-devnet0
vc_image: 192.168.45.152:80/dh/ethpandaops/prysm-validator:electra-devnet0
- el_type: nethermind
el_image: 192.168.45.152:80/dh/nethermindeth/nethermind:pectra
cl_type: grandine
cl_image: 192.168.45.152:80/dh/ethpandaops/grandine:feature-electra
- el_type: ethereumjs
el_image: 192.168.45.152:80/dh/ethpandaops/ethereumjs:master
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:ef-tests-electra
- el_type: ethereumjs
el_image: 192.168.45.152:80/dh/ethpandaops/ethereumjs:master
cl_type: teku
cl_image: 192.168.45.152:80/dh/ethpandaops/teku:master
- el_type: ethereumjs
el_image: 192.168.45.152:80/dh/ethpandaops/ethereumjs:master
cl_type: prysm
cl_image: 192.168.45.152:80/dh/ethpandaops/prysm-beacon-chain:electra-devnet0
vc_image: 192.168.45.152:80/dh/ethpandaops/prysm-validator:electra-devnet0
- el_type: ethereumjs
el_image: 192.168.45.152:80/dh/ethpandaops/ethereumjs:master
cl_type: grandine
cl_image: 192.168.45.152:80/dh/ethpandaops/grandine:feature-electra
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:lightclient-prague-devnet-0
cl_type: prysm
cl_image: 192.168.45.152:80/dh/ethpandaops/prysm-beacon-chain:electra-devnet0
vc_image: 192.168.45.152:80/dh/ethpandaops/prysm-validator:electra-devnet0
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:lightclient-prague-devnet-0
cl_type: teku
cl_image: 192.168.45.152:80/dh/ethpandaops/teku:master
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:lightclient-prague-devnet-0
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:ef-tests-electra
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:lightclient-prague-devnet-0
cl_type: grandine
cl_image: 192.168.45.152:80/dh/ethpandaops/grandine:feature-electra
network_params:
electra_fork_epoch: 1
additional_services:
- dora
- apache
- assertoor
snooper_enabled: true
assertoor_params:
run_stability_check: false
run_block_proposal_check: false
tests:
- { file: "https://raw.githubusercontent.com/ethpandaops/assertoor-test/master/assertoor-tests/pectra-dev/wait-for-slot.yaml", config: { slot: 43 }}
- { file: "https://raw.githubusercontent.com/ethpandaops/assertoor-test/master/assertoor-tests/pectra-dev/bls-changes.yaml", config: { validatorCount: 300 }}
- { file: "https://raw.githubusercontent.com/ethpandaops/assertoor-test/master/assertoor-tests/pectra-dev/voluntary-exits.yaml", config: { validatorCount: 10 }}
- https://raw.githubusercontent.com/ethpandaops/assertoor-test/master/assertoor-tests/pectra-dev/massive-deposit.yaml
```
## PeerDAS
### Info
- [Spec sheet - peerdas-devnet-0](https://notes.ethereum.org/@ethpandaops/peerdas-devnet-0)
- [Interop milestones](https://notes.ethereum.org/@ethpandaops/peerdas-devnet-0)
### Kurtosis config example
:::info
This config depends on the local docker registry cache being configured on your machine. See [docker](#Docker) for more info.
:::
```yaml=
participants:
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:master
cl_type: prysm
cl_image: 192.168.45.152:80/dh/ethpandaops/prysm-beacon-chain:peerDAS
cl_max_mem: 2048
count: 2
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:master
cl_type: lighthouse
cl_extra_args: [
--subscribe-all-data-column-subnets,
]
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:das
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:master
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:das
network_params:
eip7594_fork_epoch: 0
eip7594_fork_version: "0x50000038"
snooper_enabled: true
global_log_level: debug
additional_services:
- dora
- goomy_blob
- apache
```
## Verkle
### Info
Verkle testing is divided into two sections:
- Verkle at genesis
- Verkle fork with transition
We support testing both, please refer to the respective section for configs. There is no spec sheet for the transition testing.
### Verkle-at-genesis
#### Info
- [Spec sheet - verkle-gen-devnet-6](https://notes.ethereum.org/@ethpandaops/verkle-gen-devnet-6)
- [Interop milestones](https://notes.ethereum.org/@gballet/verke_interop_milestones)
#### Kurtosis config example
:::info
This config depends on the local docker registry cache being configured on your machine. See [docker](#Docker) for more info.
:::
```yaml=
participants:
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:kaustinen-with-shapella
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:verkle-trees-capella
count: 2
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:kaustinen-with-shapella
cl_type: lodestar
cl_image: 192.168.45.152:80/dh/ethpandaops/lodestar:g11tech-verge
network_params:
network: verkle-gen-devnet-6
```
### Verkle-transition
#### Info
Please note, due to the testing tools deprecating capella testing - verkle transition testing relies on an archived fork of kurtosis.
#### Kurtosis config example
:::info
This config depends on the local docker registry cache being configured on your machine. See [docker](#Docker) for more info.
:::
```yaml=
participants:
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:gballet-transition-post-genesis-bb3a39d
cl_type: lodestar
cl_image: 192.168.45.152:80/dh/ethpandaops/lodestar:g11tech-verge-628c35a
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:gballet-transition-post-genesis-bb3a39d
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:verkle-trees-capella-2ffb8a9
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:gballet-transition-post-genesis-bb3a39d
cl_type: teku
cl_image: 192.168.45.152:80/dh/ethpandaops/teku:verkle-trees-ab99e2d
- el_type: besu
el_image: 192.168.45.152:80/dh/ethpandaops/besu:matkt-fork-verkle-f2e19f9
cl_type: lodestar
cl_image: 192.168.45.152:80/dh/ethpandaops/lodestar:g11tech-verge-628c35a
network_params:
electra_fork_epoch: 1
additional_services:
- el_forkmon
- dora
- tx_spammer
- apache
```
Run this file with:
```bash
kurtosis run github.com/kurtosis-tech/ethereum-package@bbusa/verkle-archive --args-file=./<insert-filename>.yaml --image-download always
```
## Builder-bot
We have added a builder bot to the telegram interop server, this bot can build and push a docker image to the `ethpandaops` dockerhub repo. In conjunction with the cache server, we should be able to build images quickly and save bandwidth at the same time.
To use the bot:
- Navigate to the `Builder-bot` topic on the telegram interop chap
- Type in `/build https://<full-github-url-of-repo-branch>`
E.g:
`/build https://github.com/lightclient/go-ethereum/tree/devnet-patches-2` will build the `devnet-patches-2` branch of `lightclient`'s fork of `go-ethereum`.
Even if you use the official fork and the `main` branch, please still stick to the above notation.
## Snapshots
### EL client data directories
We've taken some snapshots of the data directories of execution clients for the Sepolia and Holesky testnets.
### Prerequesites
We've used [`zstandard`](https://github.com/facebook/zstd) as the compression algorithm, so if you haven't installed it yet, you might need to.
Examples on how to install it:
- Mac: `brew install zstd`
- Ubuntu/Debian: `apt get install zstd`
- Alpine: `apk add zstd`
#### Versions
Block height during snapshot:
- Sepolia: `5650000`
- Holesky: `1310000`
Client versions used when taking the snapshot:
Client | Version
------ | -----
Geth | `Geth/v1.13.12-stable-02eb36af/linux-amd64/go1.21.7`
Nethermind | `Nethermind/v1.25.3+e64bee70/linux-x64/dotnet8.0.1`
Besu | `besu/v24.1.2/linux-x86_64/openjdk-java-17`
Erigon | `erigon/2.57.3/linux-amd64/go1.20.12`
Reth | `reth/v0.1.0-alpha.17-aac72f4/x86_64-unknown-linux-gnu`
#### Links
The snapshots can be pulled using the following links:
Client | Network | Size | Download link
--- | --- | --- | --
Besu | Sepolia | 212 GB | [Download](http://192.168.45.152:8080/snapshots/sepolia/besu/latest/snapshot.tar.zst)
Erigon | Sepolia | 245 GB | [Download](http://192.168.45.152:8080/snapshots/sepolia/erigon/latest/snapshot.tar.zst)
Geth | Sepolia | 199 GB | [Download](http://192.168.45.152:8080/snapshots/sepolia/geth/latest/snapshot.tar.zst)
Nethermind | Sepolia | 229 GB | [Download](http://192.168.45.152:8080/snapshots/sepolia/nethermind/latest/snapshot.tar.zst)
Reth | Sepolia | 215 GB | [Download](http://192.168.45.152:8080/snapshots/sepolia/reth/latest/snapshot.tar.zst)
Besu | Holesky | 43 GB | [Download](http://192.168.45.152:8080/snapshots/holesky/besu/latest/snapshot.tar.zst)
Erigon | Holesky | 41 GB | [Download](http://192.168.45.152:8080/snapshots/holesky/erigon/latest/snapshot.tar.zst)
Geth | Holesky | 39 GB | [Download](http://192.168.45.152:8080/snapshots/holesky/geth/latest/snapshot.tar.zst)
Nethermind | Holesky | 77 GB | [Download](http://192.168.45.152:8080/snapshots/holesky/nethermind/latest/snapshot.tar.zst)
Reth | Holesky | 39 GB | [Download](http://192.168.45.152:8080/snapshots/holesky/reth/latest/snapshot.tar.zst)
We also provide a pre-deneb data dirs with preimages (Useful for Verkle)
Block height during that snapshot:
- Holesky: `793312`
Client | Network | Size | Download link
--- | --- | --- | --
Geth | Sepolia | 14 GB | [Download](http://192.168.45.152:8080/snapshots/holesky/geth/793312/snapshot.tar.zst)
### Example
#### Downloading a Holesky Geth datadir to `$PWD/data/geth-holesky`:
```
mkdir $PWD/data/geth-holesky
curl -s -L http://192.168.45.152:8080/snapshots/holesky/geth/latest/snapshot.tar.zst | tar -I zstd -xvf - -C $PWD/data/geth-holesky
```
#### Or... importing with docker, if you don't want to install zstd on your machine:
```
docker run --rm -it -v $PWD/data/geth-holesky:/data --entrypoint "/bin/sh" alpine -c "apk add --no-cache curl tar zstd && curl -s -L http://192.168.45.152:8080/snapshots/holesky/geth/latest/snapshot.tar.zst | tar -I zstd -xvf - -C /data"
```
## Advanced kurtosis
Please make sure that you have all prerequisites installed before you come to the event.
* Have Docker [installed](https://github.com/docker/docker-install?tab=readme-ov-file#usage)
* Have a Kurtosis [installed](https://docs.kurtosis.com/install)
* Familirize yourself (at least a little bit) with [Kurtosis](https://ethpandaops.io/posts/kurtosis-deep-dive/)
* Run kurtosis run at least once to make sure everything is working as expected (`kurtosis run github.com/kurtosis-tech/ethereum-package`)
* Verify you have kurtosis verison 0.89.0 or higher (`kurtosis version`)
Advanced Kurtosis usage and a deep dive can be found [here](https://ethpandaops.io/posts/kurtosis-deep-dive/).
A test assertion tool integrated to Kurtosis (similar to Hive) can be found [here](https://ethpandaops.io/posts/assertoor-introduction/).
### Replace all images in your kurtosis config with the caching server's IP
* Replace all images in your kurtosis config with the caching server's IP. For example, if your kurtosis config looks like this:
```yaml
participants:
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:master
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:master
```
Replace it with
```yaml
participants:
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:master
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:master
```
### Running a client outside of Kurtosis
It's possible to run a network within Kurtosis and then run an EL/CL outside of it for easier debugging.
- Determine your private ipv4 address: e.g. 192.168.48.2
- Add `port_publisher` to your Kurtosis config:
```yaml
port_publisher:
nat_exit_ip: 192.168.48.2
public_port_start: 30000
```
- Create your Kurtosis enclave
```kurtosis run --enclave my-enclave github.com/kurtosis-tech/ethereum-package --args-file=network_params.yaml --image-download always```
- Find the `apache` port.
```kurtosis service inspect my-enclave apache```
E.g. `http://127.0.0.1:61972`
- Download network config files which can be found here:
- In a `.tar` file: `http://127.0.0.1:61972/network-configs.tar`
- Or individually: `http://127.0.0.1:61972/network-configs/`
- `http://127.0.0.1:61972/network-configs/config.yaml`
- `http://127.0.0.1:61972/network-configs/boot_enr.yaml`
- `http://127.0.0.1:61972/network-configs/bootnode.txt`
- etc
- Use these bootnodes/configs to connect to the network running within Kurtosis
### Verkle holesky shadowfork
:::info
This config depends on the local docker registry cache being configured on your machine. See [docker](#Docker) for more info.
:::
`kurtosis run github.com/kurtosis-tech/ethereum-package@bbusa-verkle-archive --args-file this.yaml --image-download always`
```yaml
participants:
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:kaustinen-with-shapella
cl_type: lighthouse
cl_image: 192.168.45.152:80/dh/ethpandaops/lighthouse:verkle-trees-capella-2ffb8a9
- el_type: geth
el_image: 192.168.45.152:80/dh/ethpandaops/geth:kaustinen-with-shapella
cl_type: lodestar
cl_image: 192.168.45.152:80/dh/ethpandaops/lodestar:g11tech-verge-815364b
network_params:
electra_fork_epoch: 1
genesis_delay: 60
network: holesky-shadowfork-verkle
network_sync_base_url: http://192.168.45.152:5000/snapshots/
additional_services:
- dora
- el_forkmon
- tx_spammer
tx_spammer_params:
tx_spammer_extra_args:
- "--txcount=100"
snooper_enabled: true
persistent: true
```