-
-
Published
Linked with GitHub
[TOC]
# Debugging Eth2 clients with Rumor
[Rumor](https://github.com/protolambda/rumor) is an interactive shell, scripting environment, and toolchain for Eth2 debugging and testing.
This doc describes why, what, and how this all works.
## Background
Testing Eth2 clients can be incredibly difficult:
- Clients are autonomous, tests are not
- Clients are protective, tests need to instrument
- Clients are different, tests are shared
- Clients are functionally wide, tests are functionally focused
Clients are not changing this, tests will need to be different
So what do we do? We translate testing to fit clients:
- Commands for all communication, all clients talk the protocol
- Commands to surround a client, all clients work with peers
- Commands to inject data, all clients sync to the same state
- Commands to mock / enable functionality, all clients value a "complete" peer
And why "commands"? Because code is already too much:
- Commands separate the intention from the implementation
- Commands are short and descriptive
- Commands can be scripted and used interactively
So this is why Rumor started, and why I expanded it beyond basic networking, with the new v0.2.x version.
## Functionality
TLDR: everything you know, love or hate about a shell, but with the p2p sugar to script most Eth2 protocol interactions.
The main concepts are:
- **Actors**: to run lots of **different p2p hosts** is incredibly easy, and low overhead.
And may sybil clients on attack-nets later...
- **Log**: Every command formats in log, optionally JSON format.
And automagically: **log vars -> env vars**
- **Shell**: Commands can be **stateful** and easily **share data** between eachother, run in the **background**, etc.
- **Coverage**: There are currently over a 100 commands, each with options, and common patterns, to do what you want with libp2p or Eth2.
There are a ton of commands, but you don't need them all at the same time!
See [How to use it](#How-to-use-it) for quick-start!
A full overview, of v0.2:
```
host Manage the libp2p host
start Start the host node. See flags for security, transport, mux etc. options
stop Stop the host node.
view View local peer ID, listening addresses, etc.
listen Start listening on given address (see option flags).
event command was not recognized
sleep Sleep for given amount of time.
enr Ethereum Name Record (ENR) utilities
view View ENR contents
gen-key Make a private key for an ENR.
make Make a private key for an ENR. Optionally override current ENR settings.
peer Manage peers
connect Connect to peer.
disconnect Close all open connections with the given peer
protect Protect a peer by giving it a tag
unprotect Unprotect a peer by removing a tag
add Add the address of a peer to the peerstore. If an ENR is provided, update ENR values as well.
trim Trim peers, with timeout.
list List peers.
identify Identify peer (requires host start --identify=true).
track Manage peerstore trackers
tee Track a part of the peerstore and tee it elsewhere
list List active peerstore tracker tees
info Read info about a specific peer from the peerstore.
addrs View known addresses of [peerID]. Defaults to local addresses if no peer id is specified.
status Manage and track peer status
get Get current status and if following the chain or not.
set Set (a part of) the current status and if following the chain or not.
req Fetch status of connected peer.
poll Fetch status of all connected peers, repeatedly on the given interval.
serve Serve incoming status requests
follow Enable or disable automatic status updating based on followed chain
metadata Manage and track peer metadata
ping Ping a connected peer to get their seq nr, optionally req metadata after.
pong Serve incoming ping requests: pong back, and optionally request metadata back
get Get current metadata and if following automatically or not.
set Set (a part of) the current status and if following the chain or not.
req Fetch metadata of connected peer (without pinging first).
poll Ping all connected peers, repeatedly on the given interval. Optionally update if seq nr is new.
serve Serve incoming metadata requests
follow Enable or disable automatic metadata updating
peerstore Manage peerstores
create Create and activate a peerstore
switch Switch peerstore
list List peerstores and identify current peerstore
dv5 Discv5 needs an ENR first. Use 'enr make'.
run Run discv5, spawned as a background process.
ping Run discv5-ping
resolve Resolve target address and try to find latest record for it.
request Request target address directly.
lookup Get list of nearby nodes. If no target node is provided, then find nodes nearby to self.
random Get random multi addrs, keep going until stopped
self get local discv5 ENR
gossip Manage Libp2p GossipSub
start Start GossipSub
list List joined gossip topics
join Join a gossip topic. This only sets up the topic, it does not actively find peers. See `gossip log start` and `gossip publish`.
events Listen for events (not messages) on this topic. Events: 'join=<peer-ID>', 'leave=<peer-ID>'
list-peers List the peers known for the given topic
blacklist Blacklist a peer from GossipSub
leave Leave a gossip topic.
log Log the messages of a gossip topic. Messages are hex-encoded. Join a topic first.
publish Publish a message to the topic. The message should be hex-encoded.
rpc Manage Eth2 RPC
goodbye Manage goodbye RPC
status Manage status RPC
ping Manage ping RPC
metadata Manage metadata RPC
blocks-by-range Manage blocks-by-range RPC
blocks-by-root Manage blocks-by-root RPC
->
req Make requests
raw Make raw requests
listen Listen for new requests
resp Respond to requests
chunk Respond a chunk to a request
invalid-request Respond a raw chunk to a request
server-error Respond a raw chunk to a request
close Close open requests
blocks Manage eth2 beacon blocks
on <db name> # Select by name
db # Use current preferred DB
import Import a SignedBeaconBlock
export Export a SignedBeaconBlock by its block root
get Get a summary of a SignedBeaconBlock by its block root
rm Remove a block from the managed blocks collection
stats Show stats of currently managed blocks
list List known block roots
states Manage eth2 beacon States
on <db name> # Select by name
db # Use current preferred DB
import Import a BeaconState
export Export a BeaconState by its state root
get Get a summary of a SignedBeaconBlock by its block root
rm Remove a block from the managed States collection
stats Show stats of currently managed states
list List known state roots
# Warning 'chain' and subcommands are *experimental and being worked on*
chain Manage eth2 chains
create Create a new eth2 chain from a pre-state
copy Copy an eth2 chain, used to maintain a different view of a chain than other actors on the original chain.
switch Switch actor to another eth2 chain
rm Remove an eth2 chain, cannot remove when other actors are following the chain
list List chains and identify current chain
on current chain was not found. Use 'chain create' to create chains
attestation Add an attestation from the attestations tracker to the chain view.
block Add a block from the blocks tracker to the chain view.
hot Manage the hot part of the chain
cold Manage the cold part of the chain
head Manage the head of the chain, either manually or with forkchoice
get Get the head of the chain.
set Override the head of the chain.
follow Follow the head of the chain or not.
serve Serve the chain to peers
by-range Serve the chain by slot range.
by-root Serve the chain by block root.
sync
by-range Sync the chain by slot range.
by-root Sync the chain by requesting blocks by root.
votes List the latest votes of validators
```
Fair warning: Forkchoice/chain testing is incomplete and being worked on. All the p2p/networking commands should work though, give it a try.
## How it works
The stack is layered as follows:
- A main starts the shell / script parser / server / whatever Rumor mode
- A session processor opens a session (concurrent sessions are possible!)
- It tracks sessions
- It tracks actors
- It tracks global state (collection of peerstores, states, blocks, chains, etc.)
- It tracks ongoing calls from sessions
- It can redirect/fork logging
- A session has a:
- Interpreter to run your inputs (nicely separated from the parser, thanks mvdan). Either a rumor command, or proxy to subprocess.
- Dynamic environment to resolve any variables of your scripts to log outputs. Never hardcode anything in your script.
- An actor which holds things like:
- libp2p host
- discv5 local node
- choices of peerstore, chain, head, status, etc.
- buffered RPC requests to respond to
- A suite of commands, each a simple struct that either:
- Resolves subcommands, forwarding state
- Runs what you request, with flags
- Any command
- Runs in sync within session
- Can continue in background (think dv5, sync, etc.), until `cancel`led
## How to use it
### Install Rumor
This requires Go:
```
GO111MODULE=on go get github.com/protolambda/rumor
```
### Run it
Then, pick how you want to use Rumor, starting the program with a subcommand:
```
attach Attach to a running rumor server
bare Rumor as a bare JSON-formatted input/output process
file Run rom a file
help Help about any command
serve Rumor as a server to attach to (IPC/TCP/WS socket)
shell Rumor as a human-readable shell
```
Get started, run:
```shell
rumor shell
```
You then end up in an interactive shell, a lot like `bash`:
- It has history (CTRL+R, navigation keys)
- It can deal with multi-line statements. Start typing a bash statement.
- ```bash
for i in {0..3}; do
echo "heyyy"
echo "yaa"
done
```
- Or continue multi-line commands with `\`
- You can use pipes, call commands from your system, etc.
You can exit the shell with `CTRL+C` or with the `exit` command. It will gracefully shutdown remaining open processes.
Now, let's get started with Rumor commands.
### Commons
```bash
# Start a libp2p host (has options for security, mux, nat, etc.).
# Creates a private key for p2p if you don't have one already.
host start
# Optional, if you want it to spam about *every* stream and connection
# that occurs. It will run in the background.
host notify all
# Starts libp2p listening (has options for interface, port etc.)
host listen
# You'll get a multi-address in the log to connect to from another host
```
Now, if you don't have a client at hand to connect with, try running a 2nd actor!
```bash
# If with ':', it will be used as actor name
bob: host start
# Or, to not repeat ourselves every command, change the default actor:
bob: me
# Take a different port this time
host listen --tcp=9001
# connect with the address you got previously
peer connect /ip4/127.0.0.1/tcp/9000/p2p/foobar
```
This was painful, I don't want to copy anything! Well, try this next time:
```bash
alice: host start
# Anything starting with `_` is a call ID: a reference for later.
# Every command has an auto-generated one by default, but you can name it yourself if you like
_my_ref alice: host listen
# Logged an 'addr' field to connect to
bob: host start
bob: host listen --tcp=9001
# Now '_' + ID + '_' + key = automatic environment variable
bob: peer connect $_my_ref_addr
```
Even shorter, `_` is a shorthand for the last call:
```bash
alice: host start
bob: host start
alice: host listen
bob: host listen
alice: peer connect $__addr
```
Great, you should now be able to run a multi-actor libp2p network, in less than a single tweet.
### ENRs and discovery
There's this optional step you can run before or after the `host` `start`/`listen`, but required if you need discv5. And you can customize your actor identity.
```bash
enr make
```
- It will automatically pick up on the port you already use for `host listen` (if any)
- It will automatically pick up on a NAT address, if libp2p got it for you via upnp or nat-pmp.
- You can customize your ENR data before, or while, running discv5
- You can assign a static and/or fallback IP and port
- `eth2`/`attnets` ENR entry support soon.
Now, to run discv5:
```bash
_my_dv5 dv5 run (optional bootnode enrs/enodes here)
# do other things, maybe some dv5 queries
sleep 20s
# Cancel the background task. Dv5 will go offline again
_my_dv5 cancel
```
There is a whole lot of discv5 commands, and the `random` command can run in background and add addresses. Filling the peerstore with discv5. (Improvements work in progress)
### Peers
`peer` helps you list, manage, connect with peers. As well as `status` and `metadata` serving/requesting to behave nice with Eth2 peers.
When you `host start`, the host is configured to maintain the peer count within a low and high bound by default.
This can be disabled/customized with options. To avoid a peer from getting cut off, try `peer protect somepeerid bootnode`.
`peer addrs` to show local addresses, or those of a peer.
`peer info` to dig all available information of a peer up from the peerstore:
- Peer ID, node ID, pubkey, etc.
- Address info
- ENR data (if running dv5)
- Status/metadata (if using `peer status` or `peer metadata`)
- Libp2p identify user-agent and protocol version (if enabled in `host start`)
- Latency (if using metadata pings/pongs)
And `peer list` can be helpful in scripts too:
```bash
alice: peer list
peerlist=$__peers
for peer_id in ${peerlist[@]}; do
echo "Alice is connected to peer $peer_id"
done
```
In Bash variables fro logs:
- array variables become indexed arrays
- maps become associative arrays
#### peerstore
But what is the `peerstore` command for then? To manage the *`store`*, not the *`peers`*. You can create peerstores separately, share them between multiple actors, and switch from one to another.
*If you are building some sort of crawler, you may want to look into this.*
Persistance of peerstores is planned. The peerstores are backed by the IPFS datastore library, instead of the memory-backed peerstore. Currently the "Map" backend is used, but the abstraction allows for more sophistication cached file-based stores, or complete databases.
### GossipSub
Gossipsub is like a form of pubsub, or a chat channel, but in a p2p network.
Each chat channel is called a topic, and is negotiated as a protocol by libp2p.
To get gossiping between peers, join the channel before connecting.
Negotiation of the topic support in an already open connection is planned but not currently supported.
You can `log` messages to a file (hex encoded, one per line), and publish messages (also hex encoded).
The eth2 pubsub topics to join can be found in the [Eth2 p2p spec](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/phase0/p2p-interface.md#topics-and-messages)
Rumor will apply Snappy compression/uncompression as necessary, if the topic name ends with `_snappy`
Support for message validation, and auto-import into the blocks, attestations, etc. DBs is planned but not supported yet.
### RPC
To interact with the raw Eth2 RPC (a thin protocol on top of libp2p streams), use the `rpc` commands.
Each RPC "method" has its own command, with ways to request and respond:
```
req Make requests
raw Make raw requests
listen Listen for new requests
resp Respond to requests
chunk Respond a chunk to a request
invalid-request Respond a raw chunk to a request
server-error Respond a raw chunk to a request
close Close open requests
```
Example:
```bash
alice: me
host start
host listen
host view
alice_id="$__peer_id"
alice_addr="$__addr"
_listener rpc status listen --timeout=300s
bob: me
host start
host listen --tcp=9001
peer connect $alice_addr
echo "Connected! requesting now"
# Just send an empty status (or generate one with pyspec/client)
_requester rpc status req raw --raw --timeout=300s $alice_id "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007b00000000000000"
echo "requested status!"
# Respond back
alice: me
_listener next # wait for the request
rpc status resp chunk raw $_listener_req_id "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007b00000000000000"
echo "responded to status!"
# Handle the response
bob: me
# Await and process the first response chunk
_requester next
echo "Got status response! $_requester_data"
```
Like "cancel", "next" works on an existing call. It waits for the current step to complete. E.g. a status RPC request is received. Or a RPC response chunk is received.
### More tutorial material soon
- Peer status/metadata exchange
- Blocks/states import/export
- Advanced peerstore usage
- Altona sync
## Planned UX features
- Proxy interrupt signal to end calls early
- Move calls to background via signal
- Support for switching sessions within the shell