[TOC] # Debugging Eth2 clients with Rumor [Rumor](https://github.com/protolambda/rumor) is an interactive shell, scripting environment, and toolchain for Eth2 debugging and testing. This doc describes why, what, and how this all works. ## Background Testing Eth2 clients can be incredibly difficult: - Clients are autonomous, tests are not - Clients are protective, tests need to instrument - Clients are different, tests are shared - Clients are functionally wide, tests are functionally focused Clients are not changing this, tests will need to be different So what do we do? We translate testing to fit clients: - Commands for all communication, all clients talk the protocol - Commands to surround a client, all clients work with peers - Commands to inject data, all clients sync to the same state - Commands to mock / enable functionality, all clients value a "complete" peer And why "commands"? Because code is already too much: - Commands separate the intention from the implementation - Commands are short and descriptive - Commands can be scripted and used interactively So this is why Rumor started, and why I expanded it beyond basic networking, with the new v0.2.x version. ## Functionality TLDR: everything you know, love or hate about a shell, but with the p2p sugar to script most Eth2 protocol interactions. The main concepts are: - **Actors**: to run lots of **different p2p hosts** is incredibly easy, and low overhead. And may sybil clients on attack-nets later... - **Log**: Every command formats in log, optionally JSON format. And automagically: **log vars -> env vars** - **Shell**: Commands can be **stateful** and easily **share data** between eachother, run in the **background**, etc. - **Coverage**: There are currently over a 100 commands, each with options, and common patterns, to do what you want with libp2p or Eth2. There are a ton of commands, but you don't need them all at the same time! See [How to use it](#How-to-use-it) for quick-start! A full overview, of v0.2: ``` host Manage the libp2p host start Start the host node. See flags for security, transport, mux etc. options stop Stop the host node. view View local peer ID, listening addresses, etc. listen Start listening on given address (see option flags). event command was not recognized sleep Sleep for given amount of time. enr Ethereum Name Record (ENR) utilities view View ENR contents gen-key Make a private key for an ENR. make Make a private key for an ENR. Optionally override current ENR settings. peer Manage peers connect Connect to peer. disconnect Close all open connections with the given peer protect Protect a peer by giving it a tag unprotect Unprotect a peer by removing a tag add Add the address of a peer to the peerstore. If an ENR is provided, update ENR values as well. trim Trim peers, with timeout. list List peers. identify Identify peer (requires host start --identify=true). track Manage peerstore trackers tee Track a part of the peerstore and tee it elsewhere list List active peerstore tracker tees info Read info about a specific peer from the peerstore. addrs View known addresses of [peerID]. Defaults to local addresses if no peer id is specified. status Manage and track peer status get Get current status and if following the chain or not. set Set (a part of) the current status and if following the chain or not. req Fetch status of connected peer. poll Fetch status of all connected peers, repeatedly on the given interval. serve Serve incoming status requests follow Enable or disable automatic status updating based on followed chain metadata Manage and track peer metadata ping Ping a connected peer to get their seq nr, optionally req metadata after. pong Serve incoming ping requests: pong back, and optionally request metadata back get Get current metadata and if following automatically or not. set Set (a part of) the current status and if following the chain or not. req Fetch metadata of connected peer (without pinging first). poll Ping all connected peers, repeatedly on the given interval. Optionally update if seq nr is new. serve Serve incoming metadata requests follow Enable or disable automatic metadata updating peerstore Manage peerstores create Create and activate a peerstore switch Switch peerstore list List peerstores and identify current peerstore dv5 Discv5 needs an ENR first. Use 'enr make'. run Run discv5, spawned as a background process. ping Run discv5-ping resolve Resolve target address and try to find latest record for it. request Request target address directly. lookup Get list of nearby nodes. If no target node is provided, then find nodes nearby to self. random Get random multi addrs, keep going until stopped self get local discv5 ENR gossip Manage Libp2p GossipSub start Start GossipSub list List joined gossip topics join Join a gossip topic. This only sets up the topic, it does not actively find peers. See `gossip log start` and `gossip publish`. events Listen for events (not messages) on this topic. Events: 'join=<peer-ID>', 'leave=<peer-ID>' list-peers List the peers known for the given topic blacklist Blacklist a peer from GossipSub leave Leave a gossip topic. log Log the messages of a gossip topic. Messages are hex-encoded. Join a topic first. publish Publish a message to the topic. The message should be hex-encoded. rpc Manage Eth2 RPC goodbye Manage goodbye RPC status Manage status RPC ping Manage ping RPC metadata Manage metadata RPC blocks-by-range Manage blocks-by-range RPC blocks-by-root Manage blocks-by-root RPC -> req Make requests raw Make raw requests listen Listen for new requests resp Respond to requests chunk Respond a chunk to a request invalid-request Respond a raw chunk to a request server-error Respond a raw chunk to a request close Close open requests blocks Manage eth2 beacon blocks on <db name> # Select by name db # Use current preferred DB import Import a SignedBeaconBlock export Export a SignedBeaconBlock by its block root get Get a summary of a SignedBeaconBlock by its block root rm Remove a block from the managed blocks collection stats Show stats of currently managed blocks list List known block roots states Manage eth2 beacon States on <db name> # Select by name db # Use current preferred DB import Import a BeaconState export Export a BeaconState by its state root get Get a summary of a SignedBeaconBlock by its block root rm Remove a block from the managed States collection stats Show stats of currently managed states list List known state roots # Warning 'chain' and subcommands are *experimental and being worked on* chain Manage eth2 chains create Create a new eth2 chain from a pre-state copy Copy an eth2 chain, used to maintain a different view of a chain than other actors on the original chain. switch Switch actor to another eth2 chain rm Remove an eth2 chain, cannot remove when other actors are following the chain list List chains and identify current chain on current chain was not found. Use 'chain create' to create chains attestation Add an attestation from the attestations tracker to the chain view. block Add a block from the blocks tracker to the chain view. hot Manage the hot part of the chain cold Manage the cold part of the chain head Manage the head of the chain, either manually or with forkchoice get Get the head of the chain. set Override the head of the chain. follow Follow the head of the chain or not. serve Serve the chain to peers by-range Serve the chain by slot range. by-root Serve the chain by block root. sync by-range Sync the chain by slot range. by-root Sync the chain by requesting blocks by root. votes List the latest votes of validators ``` Fair warning: Forkchoice/chain testing is incomplete and being worked on. All the p2p/networking commands should work though, give it a try. ## How it works The stack is layered as follows: - A main starts the shell / script parser / server / whatever Rumor mode - A session processor opens a session (concurrent sessions are possible!) - It tracks sessions - It tracks actors - It tracks global state (collection of peerstores, states, blocks, chains, etc.) - It tracks ongoing calls from sessions - It can redirect/fork logging - A session has a: - Interpreter to run your inputs (nicely separated from the parser, thanks mvdan). Either a rumor command, or proxy to subprocess. - Dynamic environment to resolve any variables of your scripts to log outputs. Never hardcode anything in your script. - An actor which holds things like: - libp2p host - discv5 local node - choices of peerstore, chain, head, status, etc. - buffered RPC requests to respond to - A suite of commands, each a simple struct that either: - Resolves subcommands, forwarding state - Runs what you request, with flags - Any command - Runs in sync within session - Can continue in background (think dv5, sync, etc.), until `cancel`led ## How to use it ### Install Rumor This requires Go: ``` GO111MODULE=on go get github.com/protolambda/rumor ``` ### Run it Then, pick how you want to use Rumor, starting the program with a subcommand: ``` attach Attach to a running rumor server bare Rumor as a bare JSON-formatted input/output process file Run rom a file help Help about any command serve Rumor as a server to attach to (IPC/TCP/WS socket) shell Rumor as a human-readable shell ``` Get started, run: ```shell rumor shell ``` You then end up in an interactive shell, a lot like `bash`: - It has history (CTRL+R, navigation keys) - It can deal with multi-line statements. Start typing a bash statement. - ```bash for i in {0..3}; do echo "heyyy" echo "yaa" done ``` - Or continue multi-line commands with `\` - You can use pipes, call commands from your system, etc. You can exit the shell with `CTRL+C` or with the `exit` command. It will gracefully shutdown remaining open processes. Now, let's get started with Rumor commands. ### Commons ```bash # Start a libp2p host (has options for security, mux, nat, etc.). # Creates a private key for p2p if you don't have one already. host start # Optional, if you want it to spam about *every* stream and connection # that occurs. It will run in the background. host notify all # Starts libp2p listening (has options for interface, port etc.) host listen # You'll get a multi-address in the log to connect to from another host ``` Now, if you don't have a client at hand to connect with, try running a 2nd actor! ```bash # If with ':', it will be used as actor name bob: host start # Or, to not repeat ourselves every command, change the default actor: bob: me # Take a different port this time host listen --tcp=9001 # connect with the address you got previously peer connect /ip4/ ``` This was painful, I don't want to copy anything! Well, try this next time: ```bash alice: host start # Anything starting with `_` is a call ID: a reference for later. # Every command has an auto-generated one by default, but you can name it yourself if you like _my_ref alice: host listen # Logged an 'addr' field to connect to bob: host start bob: host listen --tcp=9001 # Now '_' + ID + '_' + key = automatic environment variable bob: peer connect $_my_ref_addr ``` Even shorter, `_` is a shorthand for the last call: ```bash alice: host start bob: host start alice: host listen bob: host listen alice: peer connect $__addr ``` Great, you should now be able to run a multi-actor libp2p network, in less than a single tweet. ### ENRs and discovery There's this optional step you can run before or after the `host` `start`/`listen`, but required if you need discv5. And you can customize your actor identity. ```bash enr make ``` - It will automatically pick up on the port you already use for `host listen` (if any) - It will automatically pick up on a NAT address, if libp2p got it for you via upnp or nat-pmp. - You can customize your ENR data before, or while, running discv5 - You can assign a static and/or fallback IP and port - `eth2`/`attnets` ENR entry support soon. Now, to run discv5: ```bash _my_dv5 dv5 run (optional bootnode enrs/enodes here) # do other things, maybe some dv5 queries sleep 20s # Cancel the background task. Dv5 will go offline again _my_dv5 cancel ``` There is a whole lot of discv5 commands, and the `random` command can run in background and add addresses. Filling the peerstore with discv5. (Improvements work in progress) ### Peers `peer` helps you list, manage, connect with peers. As well as `status` and `metadata` serving/requesting to behave nice with Eth2 peers. When you `host start`, the host is configured to maintain the peer count within a low and high bound by default. This can be disabled/customized with options. To avoid a peer from getting cut off, try `peer protect somepeerid bootnode`. `peer addrs` to show local addresses, or those of a peer. `peer info` to dig all available information of a peer up from the peerstore: - Peer ID, node ID, pubkey, etc. - Address info - ENR data (if running dv5) - Status/metadata (if using `peer status` or `peer metadata`) - Libp2p identify user-agent and protocol version (if enabled in `host start`) - Latency (if using metadata pings/pongs) And `peer list` can be helpful in scripts too: ```bash alice: peer list peerlist=$__peers for peer_id in ${peerlist[@]}; do echo "Alice is connected to peer $peer_id" done ``` In Bash variables fro logs: - array variables become indexed arrays - maps become associative arrays #### peerstore But what is the `peerstore` command for then? To manage the *`store`*, not the *`peers`*. You can create peerstores separately, share them between multiple actors, and switch from one to another. *If you are building some sort of crawler, you may want to look into this.* Persistance of peerstores is planned. The peerstores are backed by the IPFS datastore library, instead of the memory-backed peerstore. Currently the "Map" backend is used, but the abstraction allows for more sophistication cached file-based stores, or complete databases. ### GossipSub Gossipsub is like a form of pubsub, or a chat channel, but in a p2p network. Each chat channel is called a topic, and is negotiated as a protocol by libp2p. To get gossiping between peers, join the channel before connecting. Negotiation of the topic support in an already open connection is planned but not currently supported. You can `log` messages to a file (hex encoded, one per line), and publish messages (also hex encoded). The eth2 pubsub topics to join can be found in the [Eth2 p2p spec](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/phase0/p2p-interface.md#topics-and-messages) Rumor will apply Snappy compression/uncompression as necessary, if the topic name ends with `_snappy` Support for message validation, and auto-import into the blocks, attestations, etc. DBs is planned but not supported yet. ### RPC To interact with the raw Eth2 RPC (a thin protocol on top of libp2p streams), use the `rpc` commands. Each RPC "method" has its own command, with ways to request and respond: ``` req Make requests raw Make raw requests listen Listen for new requests resp Respond to requests chunk Respond a chunk to a request invalid-request Respond a raw chunk to a request server-error Respond a raw chunk to a request close Close open requests ``` Example: ```bash alice: me host start host listen host view alice_id="$__peer_id" alice_addr="$__addr" _listener rpc status listen --timeout=300s bob: me host start host listen --tcp=9001 peer connect $alice_addr echo "Connected! requesting now" # Just send an empty status (or generate one with pyspec/client) _requester rpc status req raw --raw --timeout=300s $alice_id "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007b00000000000000" echo "requested status!" # Respond back alice: me _listener next # wait for the request rpc status resp chunk raw $_listener_req_id "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007b00000000000000" echo "responded to status!" # Handle the response bob: me # Await and process the first response chunk _requester next echo "Got status response! $_requester_data" ``` Like "cancel", "next" works on an existing call. It waits for the current step to complete. E.g. a status RPC request is received. Or a RPC response chunk is received. ### More tutorial material soon - Peer status/metadata exchange - Blocks/states import/export - Advanced peerstore usage - Altona sync ## Planned UX features - Proxy interrupt signal to end calls early - Move calls to background via signal - Support for switching sessions within the shell