How to setup local, multi-client testnets using Kurtosis

Co-ordinating releases between clients can be hard to orchestrate, especially when testing new features that have no devnet.

Kurtosis is a tool that allows us to create testing environments using just a few commands. We can additionally setup various scenario’s and write tests for them using the kurtosis module.
This guide explores the use of kurtosis to help setup a local dev environment.

Link to Kurtosis docs: here

The doc below has a TLDR section for testing your feature, a verkle section and a mev section. All the steps are exactly the same in every scenario, just the config file changes.

TLDR:

• 1. Ensure Docker is installed locally
• 2. Dowload kurtosis with the instructions provided here
• 3. Save the below JSON in a file called example.json

{
  "participants": [
    {
      "el_client_type": "geth",
      "el_client_image": "ethpandaops/geth:dencun-devnet-8-946a2da",
      "cl_client_type": "lighthouse",
      "cl_client_image": "sigp/lighthouse:deneb-modern",
      "count": 3
    }
  ],
  "network_params": {
    "capella_fork_epoch": 2,
    "deneb_fork_epoch": 3
  },
  "launch_additional_services": false,
  "wait_for_finalization": false,
  "wait_for_verifications": false,
  "global_client_log_level": "info",
  "snooper_enabled": true
}

• 4. Start the module with the following command:

kurtosis run --enclave ethTestnet github.com/kurtosis-tech/ethereum-package  "$(cat ./example.json)"

• 5. Testnet will start in the background. You can use the following commands to get more information:

kurtosis enclave inspect ethTestnet
kurtosis service logs ethTestnet <nane> -f 

Config Explanation:

Kurtosis has built a module for the devnets. The module can be seen here. It’s written in starlark and the core “genesis” tooling is its own package here.

The module takes in parameters and starts a testnet accordingly. We can specify the client combinations, number of clients, fork epoch and so on.

Naturally the larger the number of clients, the more CPU/RAM is needed. Most systems should be able to handle a 3-4 node setup locally. For more elaborate setups, please use a cloud instance or get a fast computer to run it.
We will just assume a 3 node setup for this example.

Let node 1 be the EL:CL combination you wish to test(feature branch for example). node 2 and node 3 can be tested, stable clients(lighthouse-geth and nimbus-nethermind for example).

Since our node 1 is used for testing new features or cli flags, we can specify them under elExtraParams,beaconExtraParams or validatorExtraParams as needed.

Kurtosis works on the basis of docker images, so your feature branch would need to have a docker image built.

The below config describes what that looks like in JSON:

{
  "participants": [
    {
      "elType": "<your-EL-client>",
      "elImage": "<your-EL-docker-image",
      "elLogLevel": "<log-level-you-want>",
      "elExtraParams": [<EL-extra-params-if-needed>],
      "clType": "<your-CL-client>",
      "clImage": "<your-CL-docker-image",
      "beaconExtraParams": [<beacon-extra-params-if-needed>],
      "validatorExtraParams": [<val-extra-params-if-needed>],
    },
    {
      "el_client_type": "geth",
      "el_client_image": "ethpandaops/geth:dencun-devnet-8-946a2da",
      "cl_client_type": "lighthouse",
      "cl_client_image": "sigp/lighthouse:deneb-modern"
    },
    {
      "el_client_type": "geth",
      "el_client_image": "ethpandaops/geth:dencun-devnet-8-946a2da",
      "cl_client_type": "lighthouse",
      "cl_client_image": "sigp/lighthouse:deneb-modern"
    }
  ],
  "network": {
    "networkId": "3151908",
    "depositContractAddress": "0x4242424242424242424242424242424242424242",
    "secondsPerSlot": 12,
    "slotsPerEpoch": 32,
    "capella_fork_epoch": 2,
    "deneb_fork_epoch": 5,
    "numValidatorKeysPerNode": 64,
    "preregisteredValidatorKeysMnemonic": "giant issue aisle success illegal bike spike question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy very lucky have athlete"
  },
  "waitForMining": true,
  "waitForFinalization": true,
  "logLevel": "info"
}

Verbose Steps:

1. Dowload kurtosis with the instructions provided here
2. Ensure docker is installed and your user can execute docker commands (i.e, run docker ps -a)
3. Copy the JSON file shown above and place it in a file locally accessible, make the required changes to the JSON config. You’d probably want to reduce the number of clients you’re running if you’re on a laptop.
4. Start the module with the following command:

kurtosis run --enclave ethTestnet github.com/kurtosis-tech/ethereum-package  "$(cat ./example.json)"

4. Check the logs for the URL to forkmon, alternatively check the module logs
5. You can use the kurtosis module to fetch logs, e.g:

kurtosis enclave inspect ethTestnet
kurtosis service logs ethTestnet <enter-GUID-of-service-from-previous-step>

6. If you need shell access, you can achieve that with:

kurtosis service shell ethTestnet <enter-GUID-of-service-from-previous-step>

7. When done, run kurtosis enclave rm -f ethTestnet to delete the testnet

Verkle config example:

{
  "participants": [
    {
      "el_client_type": "geth",
      "el_client_image": "ethpandaops/geth:<VERKLE_IMAGE>",
      "elExtraParams": ["--override.verkle=<UNIXTIMESTAMP>"],
      "cl_client_type": "lighthouse",
      "cl_client_image": "sigp/lighthouse:latest"
    },
    {
      "el_client_type": "geth",
      "el_client_image": "ethpandaops/geth:<VERKLE_IMAGE>",
      "elExtraParams": ["--override.verkle=<UNIXTIMESTAMP>"],
      "cl_client_type": "lighthouse",
      "cl_client_image": "sigp/lighthouse:latest"
    },
    {
      "el_client_type": "geth",
      "el_client_image": "ethpandaops/geth:<VERKLE_IMAGE>",
      "elExtraParams": ["--override.verkle=<UNIXTIMESTAMP>"],
      "cl_client_type": "lighthouse",
      "cl_client_image": "sigp/lighthouse:latest"
    }
  ],
  "network_params": {
    "capella_fork_epoch": 2,
    "deneb_fork_epoch": 5
  },
  "launch_additional_services": false,
  "wait_for_finalization": false,
  "wait_for_verifications": false,
  "global_client_log_level": "info"
}

mock-mev example

Useful for testing mev-boost and the client implimentations without adding the complexity of the relay. This can be enabled by a single config command and would deploy the mock-builder from Mario instead of the relay infrastructure.

{
  "participants": [
    {
      "el_client_type": "geth",
      "el_client_image": "ethpandaops/geth:dencun-devnet-8-946a2da",
      "cl_client_type": "lighthouse",
      "cl_client_image": "sigp/lighthouse:deneb-modern",
      "count": 3
    }
  ],
  "network_params": {
    "capella_fork_epoch": 2,
    "deneb_fork_epoch": 4
  },
  "launch_additional_services": true,
  "global_client_log_level": "debug",
  "snooper_enabled": true,
  "mev_type": "mock"
}

Debugging

• To grab the genesis files, you can run:

kurtosis files download <enclave name> <service name> <path to copy to>

e.g. to get the EL genesis data:

kurtosis files download ethTestnet el-genesis-data ~/Downloads