# Vitalik's Annotated Ethereum 2.0 Spec, Part 2 ## Helper functions This first set of functions is made up of relatively simple "helper" functions that are then used in the rest of the spec. *Note*: The definitions below are for specification purposes and are not necessarily optimal implementations. ### Math #### `integer_squareroot` ```python def integer_squareroot(n: uint64) -> uint64: """ Return the largest integer ``x`` such that ``x**2 <= n``. """ x = n y = (x + 1) // 2 while y < x: x = y y = (x + n // x) // 2 return x ``` A square root function, using [the Babylonian method](https://en.wikipedia.org/wiki/Methods_of_computing_square_roots#Babylonian_method) for efficiency. Guaranteed to provide a precise integer result: the largest integer `x` such that `x**2 <= n` (eg. sqrt(14) = 3, sqrt(15) = 3, sqrt(16) = 4, sqrt(17) = 4). Actual implementations can use other algorithms if needed; only this precise numerical property in the outputs is mandatory. #### `xor` ```python def xor(bytes_1: Bytes32, bytes_2: Bytes32) -> Bytes32: """ Return the exclusive-or of two 32-byte strings. """ return Bytes32(a ^ b for a, b in zip(bytes_1, bytes_2)) ``` Does a bit-by-bit [XOR](https://en.wikipedia.org/wiki/Exclusive_or) on the inputs. #### `uint_to_bytes` `def uint_to_bytes(n: uint) -> bytes` is a function for serializing the `uint` type object to bytes in ``ENDIANNESS``-endian. The expected length of the output is the byte-length of the `uint` type. #### `bytes_to_uint64` ```python def bytes_to_uint64(data: bytes) -> uint64: """ Return the integer deserialization of ``data`` interpreted as ``ENDIANNESS``-endian. """ return uint64(int.from_bytes(data, ENDIANNESS)) ``` Converts 8 bytes into a 64-bit integer. ### Crypto #### `hash` `def hash(data: bytes) -> Bytes32` is SHA256. #### `hash_tree_root` `def hash_tree_root(object: SSZSerializable) -> Root` is a function for hashing objects into a single root by utilizing a hash tree structure, as defined in the [SSZ spec](../../ssz/simple-serialize.md#merkleization). #### BLS Signatures Eth2 makes use of BLS signatures as specified in the [IETF draft BLS specification draft-irtf-cfrg-bls-signature-02](https://tools.ietf.org/html/draft-irtf-cfrg-bls-signature-02) but uses [Hashing to Elliptic Curves - draft-irtf-cfrg-hash-to-curve-07](https://tools.ietf.org/html/draft-irtf-cfrg-hash-to-curve-07) instead of draft-irtf-cfrg-hash-to-curve-06. Specifically, eth2 uses the `BLS_SIG_BLS12381G2_XMD:SHA-256_SSWU_RO_POP_` ciphersuite which implements the following interfaces: - `def Sign(SK: int, message: Bytes) -> BLSSignature` - `def Verify(PK: BLSPubkey, message: Bytes, signature: BLSSignature) -> bool` - `def Aggregate(signatures: Sequence[BLSSignature]) -> BLSSignature` - `def FastAggregateVerify(PKs: Sequence[BLSPubkey], message: Bytes, signature: BLSSignature) -> bool` - `def AggregateVerify(PKs: Sequence[BLSPubkey], messages: Sequence[Bytes], signature: BLSSignature) -> bool` Within these specifications, BLS signatures are treated as a module for notational clarity, thus to verify a signature `bls.Verify(...)` is used. *Note*: The non-standard configuration of the BLS and hash to curve specs is temporary and will be resolved once IETF releases BLS spec draft 3. BLS is used [because of its aggregation-friendliness](https://ethresear.ch/t/pragmatic-signature-aggregation-with-bls/2105): many BLS signatures can be aggregated into a single signature, and if the signatures are of the same message this aggregation is extremely fast to do and the aggregate signatures are extremely cheap to verify (one elliptic curve addition (!!) per participant, plus one pairing to verify the signature no matter how many participants there are). This is the key magic that allows eth2 to support very high numbers of validators. ### Predicates <a id="lifecycle" /> #### `[Aside: note on a validator's life cycle]` _(This story begins right after the end of the previous aside on the deposit process)_ When a validator deposit is processed, the validator record is added to the validator registry (`state.validators`) (or, if it's a deposit with a pubkey that is already in the validator set, it is treated as a top-up to that existing validator' balance). If, after the deposit or top-up, the validator's balance is >= 32 ETH, the validator is placed into an **eligible for activation** stage. Validators in the eligible for activation stage are automatically put into a queue for activation (the queue doesn't literally exist as an in-consensus data structure; rather, the consensus rules just say to activate validators in order of when they became eligible for activation). See the [discussion on churn above](#churn) for why the queue exists and how many validators can get activated per epoch. Note that when the validator gets to the front of the queue their activation time gets set to 4 epochs in the future; this is to ensure committees are predictable that far ahead, as the calculation of the committees depends on the active validator set. When a validator is active, they get assigned the full set of validator responsibilities. These responsibilities are: * Making an attestation in every epoch, which includes (i) a vote on the most recent head of the beacon chain, (ii) the Casper FFG source and target checkpoint blocks, and (iii) in phase 1+ votes on shard blocks * Occasionally being selected as the proposer of a beacon block or (in phase 1+) a shard block A validator remains active until they either (i) sign a `VoluntaryExit` message that is included on chain, (ii) fall below the minimum balance of 16 ETH or (iii) get slashed. Note that in all three cases, the exiting step is done with the `initiate_validator_exit` function, which _puts the validator in a queue_ for exiting. Hence, even a slashed validator can remain active temporarily. This is awkward, but was done for three reasons: 1. To protect hard invariants about how quickly the validator set changes 2. Even in a situation where so many validators are exiting that the exit queue is more than 4 eeks long (4 eeks being the time until a slashed validator can withdraw), validators never have the incentive to self-slash to exit more quickly. 3. To prevent mass slashings from decreasing the validator set size (if that did happen, it would reduce the size of the slashable intersection needed for a successful attack against a chain). Note that currently, being slashed _does_ immediately reduce a validator's balance by 1/32, which effects the denominator in the 2/3 finality calculation, but the effect of this is very small. The exit queue is processed at the same rate as the activation queue, and exiting has a similar 4 epoch delay. After a validator exits, they are eligible to withdraw after `MIN_VALIDATOR_WITHDRAWABILITY_DELAY` (~1 day) if they exit without being slashed, and `EPOCHS_PER_SLASHINGS_VECTOR` (4 eeks) if they exit due to being slashed. Slashed validators suffer three penalties: * The minimum penalty (`1/MIN_SLASHING_PENALTY_QUOTIENT` of their balance) * A penalty proportional to the portion of other validators that get slashed (see the [`process_slashings` function](#Slashings)) * Penalties for being "offline", as though the validator was active but failing to make any attestations during the entire 4-eek period The third is included to prevent self-slashing from being a way to escape inactivity leaks. Once a validator withdraws, in phase 0 they are effectively inert from the point of view of the protocol. In later phases, an explicit "withdraw" functionality will be added, which would move the validator's balance to the appropriate account in the appropriate shard on eth2. With this background (see also the [Validator struct definition](#Validator), hopefully the next four functions are reasonably self-explanatory: #### `is_active_validator` ```python def is_active_validator(validator: Validator, epoch: Epoch) -> bool: """ Check if ``validator`` is active. """ return validator.activation_epoch <= epoch < validator.exit_epoch ``` #### `is_eligible_for_activation_queue` ```python def is_eligible_for_activation_queue(validator: Validator) -> bool: """ Check if ``validator`` is eligible to be placed into the activation queue. """ return ( validator.activation_eligibility_epoch == FAR_FUTURE_EPOCH and validator.effective_balance == MAX_EFFECTIVE_BALANCE ) ``` #### `is_eligible_for_activation` ```python def is_eligible_for_activation(state: BeaconState, validator: Validator) -> bool: """ Check if ``validator`` is eligible for activation. """ return ( # Placement in queue is finalized validator.activation_eligibility_epoch <= state.finalized_checkpoint.epoch # Has not yet been activated and validator.activation_epoch == FAR_FUTURE_EPOCH ) ``` Note that the activation queue only processes activations that were registered before the last finalized block that the beacon chain knows about; see [the section on registry updates](#Registry-updates) for more information on this. #### `is_slashable_validator` ```python def is_slashable_validator(validator: Validator, epoch: Epoch) -> bool: """ Check if ``validator`` is slashable. """ return (not validator.slashed) and (validator.activation_epoch <= epoch < validator.withdrawable_epoch) ``` #### `is_slashable_attestation_data` ```python def is_slashable_attestation_data(data_1: AttestationData, data_2: AttestationData) -> bool: """ Check if ``data_1`` and ``data_2`` are slashable according to Casper FFG rules. """ return ( # Double vote (data_1 != data_2 and data_1.target.epoch == data_2.target.epoch) or # Surround vote (data_1.source.epoch < data_2.source.epoch and data_2.target.epoch < data_1.target.epoch) ) ``` This function determines if the two `AttestationData` objects conflict with each other and so count as a self-contradiction (aka equivocation, aka double-voting) under Casper FFG rules. If they are, then any validator that signed both can be slashed. #### `is_valid_indexed_attestation` ```python def is_valid_indexed_attestation(state: BeaconState, indexed_attestation: IndexedAttestation) -> bool: """ Check if ``indexed_attestation`` is not empty, has sorted and unique indices and has a valid aggregate signature. """ # Verify indices are sorted and unique indices = indexed_attestation.attesting_indices if len(indices) == 0 or not indices == sorted(set(indices)): return False # Verify aggregate signature pubkeys = [state.validators[i].pubkey for i in indices] domain = get_domain(state, DOMAIN_BEACON_ATTESTER, indexed_attestation.data.target.epoch) signing_root = compute_signing_root(indexed_attestation.data, domain) return bls.FastAggregateVerify(pubkeys, signing_root, indexed_attestation.signature) ``` Verifies the validity of an attestation (mainly extracting the pubkeys of the signers and then verifying the signature). This function works with indexed attestations, but note that regular attestation verification goes through this function after it converts the bitfield into a list of validator indices. #### `is_valid_merkle_branch` ```python def is_valid_merkle_branch(leaf: Bytes32, branch: Sequence[Bytes32], depth: uint64, index: uint64, root: Root) -> bool: """ Check if ``leaf`` at ``index`` verifies against the Merkle ``root`` and ``branch``. """ value = leaf for i in range(depth): if index // (2**i) % 2: value = hash(branch[i] + value) else: value = hash(value + branch[i]) return value == root ``` A generic Merkle branch validity checker. ### Misc #### `compute_shuffled_index` ```python def compute_shuffled_index(index: uint64, index_count: uint64, seed: Bytes32) -> uint64: """ Return the shuffled index corresponding to ``seed`` (and ``index_count``). """ assert index < index_count # Swap or not (https://link.springer.com/content/pdf/10.1007%2F978-3-642-32009-5_1.pdf) # See the 'generalized domain' algorithm on page 3 for current_round in range(SHUFFLE_ROUND_COUNT): pivot = bytes_to_uint64(hash(seed + uint_to_bytes(uint8(current_round)))[0:8]) % index_count flip = (pivot + index_count - index) % index_count position = max(index, flip) source = hash( seed + uint_to_bytes(uint8(current_round)) + uint_to_bytes(uint32(position // 256)) ) byte = uint8(source[(position % 256) // 8]) bit = (byte >> (position % 8)) % 2 index = flip if bit else index return index ``` Eth2 needs some form of "random sampling" in order to assign validators to committees; if each validator could choose which committee they are on, a small portion of malicious validators could target one specific shard to attack and make false attestations for that shard. We can model this as a shuffling algorithm, taking an array of length N (filled with the active validator indices in that epoch) and pseudorandomly shuffling it (eg. `[0, 1, 2, 3, 5, 6] -> [3, 2, 0, 5, 6, 1]`); the committees can then just be consecutive slices of the desired length of the output array. There are a few desiderata for this shuffling algorithm: 1. **It's a shuffle**: that is, every value in the input appears in the output exactly once. 2. **Committee sizes are precise, not approximate**: technically the fact that we're doing committee selection using this shuffle-and-slice method achieves this goal, but it's still important to mention; it's the reason why eg. methods based on privately pre-committed values are not acceptable. Precise committee sizes are needed because it turns out that if you have approximate committee sizes then the committee size needed to achieve the same low failure probability increases by ~2x. 3. **Efficient forward calculation**: given a single `i`, it's easy to compute `shuffle(i)`. This is needed so that individual validators can efficiently determine what their responsibilities are. 4. **Efficient backward calculation**: given a single `shuffle(i)`, it's easy to determine `i`. This is needed so that light clients can efficiently determine the validators in any single committee. Note that some of the same desiderata apply also to proposer selection: * **We want exactly one proposer per slot**, and not a Poisson-like process that sometimes outputs one proposer but sometimes zero or two or more. This is for two reasons: (i) multiple proposers per slot creates competition, which creates incentives for validators to be highly connected to beat out competitors, which creates centralization pressures, and (ii) sometimes having zero proposers in a slot would mean unpredictable times between proposers, which lead to longer average wait times for transaction inclusion. * **We want it to be possible to efficiently calculate who the proposer is**, particularly for the benefit of light clients. We use the "swap-or-not" algorithm from https://link.springer.com/content/pdf/10.1007%2F978-3-642-32009-5_1.pdf, which efficiently satisfies all of the above properties. The swap-or-not algorithm works by performing 90 rounds of the following procedure: * Choose a random "pivot" `p` * For every index `x`, maybe swap the value at position `x` with the value at `p-x` (wrapping around the list if necessary). The "maybe" is determined by using a hash function to pseudorandomly generate `N` bits (`N` being the size of the list being shuffled) and checking if the `max(x, p-x)`'th bit equals one (this trickery is done to ensure that you get the same answer for `x` and `p-x`) It can be efficiently run forwards or backwards (you don't have to generate all `N` bits at each round, just the chunk that contains `max(x, p-x)`), and is fairly low-overhead. #### `compute_proposer_index` ```python def compute_proposer_index(state: BeaconState, indices: Sequence[ValidatorIndex], seed: Bytes32) -> ValidatorIndex: """ Return from ``indices`` a random index sampled by effective balance. """ assert len(indices) > 0 MAX_RANDOM_BYTE = 2**8 - 1 i = uint64(0) total = uint64(len(indices)) while True: candidate_index = indices[compute_shuffled_index(i % total, total, seed)] random_byte = hash(seed + uint_to_bytes(uint64(i // 32)))[i % 32] effective_balance = state.validators[candidate_index].effective_balance if effective_balance * MAX_RANDOM_BYTE >= MAX_EFFECTIVE_BALANCE * random_byte: return candidate_index i += 1 ``` Computes the proposer index. This function is somewhat involved; the idea is that it chooses a proposer, accepts them with `BALANCE/32` probability, and if it fails it keeps trying. This is done so that the probability of being selected as a proposer remains proportional to balance. #### `compute_committee` ```python def compute_committee(indices: Sequence[ValidatorIndex], seed: Bytes32, index: uint64, count: uint64) -> Sequence[ValidatorIndex]: """ Return the committee corresponding to ``indices``, ``seed``, ``index``, and committee ``count``. """ start = (len(indices) * index) // count end = (len(indices) * (index + 1)) // count return [indices[compute_shuffled_index(uint64(i), uint64(len(indices)), seed)] for i in range(start, end)] ``` Take a slice of a validator index list (assumed to be the list of active validator indices), and returns the `index`'th slice (out of a total `count` slices) of the shuffle. #### `compute_epoch_at_slot` ```python def compute_epoch_at_slot(slot: Slot) -> Epoch: """ Return the epoch number at ``slot``. """ return Epoch(slot // SLOTS_PER_EPOCH) ``` #### `compute_start_slot_at_epoch` ```python def compute_start_slot_at_epoch(epoch: Epoch) -> Slot: """ Return the start slot of ``epoch``. """ return Slot(epoch * SLOTS_PER_EPOCH) ``` #### `compute_activation_exit_epoch` ```python def compute_activation_exit_epoch(epoch: Epoch) -> Epoch: """ Return the epoch during which validator activations and exits initiated in ``epoch`` take effect. """ return Epoch(epoch + 1 + MAX_SEED_LOOKAHEAD) ``` This function takes as input an epoch (always in practice the current epoch) and outputs the epoch in which a validator that is scheduled for activation in that epoch will get activated. The delay of 4 epochs is used to keep committees predictable. #### `compute_fork_data_root` ```python def compute_fork_data_root(current_version: Version, genesis_validators_root: Root) -> Root: """ Return the 32-byte fork data root for the ``current_version`` and ``genesis_validators_root``. This is used primarily in signature domains to avoid collisions across forks/chains. """ return hash_tree_root(ForkData( current_version=current_version, genesis_validators_root=genesis_validators_root, )) ``` The root hash of the genesis validator set gets mixed into the fork version to add further domain separation, allowing chains with different genesises to automatically have different versions. This makes it easier to have many testnets with replay protection. #### `compute_fork_digest` ```python def compute_fork_digest(current_version: Version, genesis_validators_root: Root) -> ForkDigest: """ Return the 4-byte fork digest for the ``current_version`` and ``genesis_validators_root``. This is a digest primarily used for domain separation on the p2p layer. 4-bytes suffices for practical separation of forks/chains. """ return ForkDigest(compute_fork_data_root(current_version, genesis_validators_root)[:4]) ``` The first four bytes of the fork digest are used on the p2p layer to separate validators of different chains out into different networks. #### `compute_domain` ```python def compute_domain(domain_type: DomainType, fork_version: Version=None, genesis_validators_root: Root=None) -> Domain: """ Return the domain for the ``domain_type`` and ``fork_version``. """ if fork_version is None: fork_version = GENESIS_FORK_VERSION if genesis_validators_root is None: genesis_validators_root = Root() # all bytes zero by default fork_data_root = compute_fork_data_root(fork_version, genesis_validators_root) return Domain(domain_type + fork_data_root[:28]) ``` A helper function used by [`get_domain`](#get_domain). Combines together domain type and fork version (see [the section on forks](#Fork)) into a `Domain` object. See also [the section on domain separation](#domain_separation). #### `compute_signing_root` ```python def compute_signing_root(ssz_object: SSZObject, domain: Domain) -> Root: """ Return the signing root for the corresponding signing data. """ return hash_tree_root(SigningData( object_root=hash_tree_root(ssz_object), domain=domain, )) ``` Computes the hash that is being signed when an SSZ container is being signed. This is done by creating an ephemeral SSZ container that puts the original container and the domain together, and outputting the root of that. ### Beacon state accessors This set of functions accesses the beacon chain state. #### `get_current_epoch` ```python def get_current_epoch(state: BeaconState) -> Epoch: """ Return the current epoch. """ return compute_epoch_at_slot(state.slot) ``` #### `get_previous_epoch` ```python def get_previous_epoch(state: BeaconState) -> Epoch: """` Return the previous epoch (unless the current epoch is ``GENESIS_EPOCH``). """ current_epoch = get_current_epoch(state) return GENESIS_EPOCH if current_epoch == GENESIS_EPOCH else Epoch(current_epoch - 1) ``` #### `get_block_root` ```python def get_block_root(state: BeaconState, epoch: Epoch) -> Root: """ Return the block root at the start of a recent ``epoch``. """ return get_block_root_at_slot(state, compute_start_slot_at_epoch(epoch)) ``` #### `get_block_root_at_slot` ```python def get_block_root_at_slot(state: BeaconState, slot: Slot) -> Root: """ Return the block root at a recent ``slot``. """ assert slot < state.slot <= slot + SLOTS_PER_HISTORICAL_ROOT return state.block_roots[slot % SLOTS_PER_HISTORICAL_ROOT] ``` #### `get_randao_mix` ```python def get_randao_mix(state: BeaconState, epoch: Epoch) -> Bytes32: """ Return the randao mix at a recent ``epoch``. """ return state.randao_mixes[epoch % EPOCHS_PER_HISTORICAL_VECTOR] ``` In the state, we store an array of historical randao mixes (aka pseudorandomness seeds). This is needed because for many reasons we want to be able to calculate historical committees. Sometimes we care about very recent history (eg. attestations from epoch N can get included in epoch N+1, so the end-of-epoch processing of epoch N+1 needs to know what the randomness seed used in epoch N was so that it can compute the committes of that epoch), but sometimes we want to look far back, eg. we want to be able to calculate committees from months ago to verify slashings. Having a 32-eek historical store of randomness seeds helps us do this. #### `get_active_validator_indices` ```python def get_active_validator_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ Return the sequence of active validator indices at ``epoch``. """ return [ValidatorIndex(i) for i, v in enumerate(state.validators) if is_active_validator(v, epoch)] ``` Returns the subset of all validator indices that is active in the given epoch (note that this method can also get the historic active validator index set for any earlier epoch, as the state stores activation and exit epochs of all validators) #### `get_validator_churn_limit` ```python def get_validator_churn_limit(state: BeaconState) -> uint64: """ Return the validator churn limit for the current epoch. """ active_validator_indices = get_active_validator_indices(state, get_current_epoch(state)) return max(MIN_PER_EPOCH_CHURN_LIMIT, uint64(len(active_validator_indices)) // CHURN_LIMIT_QUOTIENT) ``` See the [section on churn](#churn). #### `get_seed` ```python def get_seed(state: BeaconState, epoch: Epoch, domain_type: DomainType) -> Bytes32: """ Return the seed at ``epoch``. """ mix = get_randao_mix(state, Epoch(epoch + EPOCHS_PER_HISTORICAL_VECTOR - MIN_SEED_LOOKAHEAD - 1)) # Avoid underflow return hash(domain_type + uint_to_bytes(epoch) + mix) ``` Returns the randomness seed for the given epoch. Note the precise way the wiring is done here: the seed _relevant in_ the given epoch is the seed _generated_ 5 epochs ago. For simplicity, you should mentally think of this as just `get_randao_mix(state, Epoch(epoch - MIN_SEED_LOOKAHEAD - 1))`. The technical subtlety here is that the historical randomness seeds are stored in an array that overwrites itself in a cyclic fashion, eg. if `EPOCHS_PER_HISTORICAL_VECTOR` were equal to 10, and the current epoch was 53, then `state.randao_mixes` would contain 10 seeds from epochs `[50, 51, 52, 53, 44, 45, 46, 47, 48, 49]`. So if you were to to call `get_seed` during epoch 53, if would return the value in the array at position `(53 - 1 - 4) % 10 = 8`, wrapping back around to the end. The `+ EPOCHS_PER_HISTORICAL_VECTOR` is added to ensure that in the exceptional case where `epoch < 5`, the "wrap back around to the end" behavior would still work, but the spec would avoid having any negative numbers even in the middle of the computation (this was an agreed-upon goal for the spec; the simplicity of being able to represent almost all integers with `uint64` outweighed the small complexity increases such as here). See also: the [section on randomness seeds](#seeds). #### `get_committee_count_per_slot` ```python def get_committee_count_per_slot(state: BeaconState, epoch: Epoch) -> uint64: """ Return the number of committees in each slot for the given ``epoch``. """ return max(uint64(1), min( MAX_COMMITTEES_PER_SLOT, uint64(len(get_active_validator_indices(state, epoch))) // SLOTS_PER_EPOCH // TARGET_COMMITTEE_SIZE, )) ``` Returns the number of committees in each slot (and in phase 1+, the number of shards crosslinked in each slot). If there are at least enough validators to fill up a full committee (128 validators) for each shard (\*64) for each slot in the epoch (\*32), ie. >= 262,144 validators or 8,388,608 ETH, then we get the full 64 committees per slot, and every shard gets crosslinked in every slot. If the number of validators is less than this, then we decrease the number of committees per slot to ensure that each committee remains a safe size, though at the cost of not crosslinking every shard in every slot (instead it would rotate: eg. if there were only 25 committees per slot, then slot 1 would process shards 0...24, slot 2 would process 25...49, slot 3 would process 50...63 and wrap around to also process 0...10, etc.). If there are not enough validators for even a single full committee (that is, less than 128 \* 32 = 4,096 validators, or 124,288 ETH), then committee sizes begin to drop, though in that case low committee sizes would arguably ve small issue relative to the much larger problem that there would exist many actors that could unilaterally launch a 51% attack. #### `get_beacon_committee` ```python def get_beacon_committee(state: BeaconState, slot: Slot, index: CommitteeIndex) -> Sequence[ValidatorIndex]: """ Return the beacon committee at ``slot`` for ``index``. """ epoch = compute_epoch_at_slot(slot) committees_per_slot = get_committee_count_per_slot(state, epoch) return compute_committee( indices=get_active_validator_indices(state, epoch), seed=get_seed(state, epoch, DOMAIN_BEACON_ATTESTER), index=(slot % SLOTS_PER_EPOCH) * committees_per_slot + index, count=committees_per_slot * SLOTS_PER_EPOCH, ) ``` Gets the i'th committee for the given slot. #### `get_beacon_proposer_index` ```python def get_beacon_proposer_index(state: BeaconState) -> ValidatorIndex: """ Return the beacon proposer index at the current slot. """ epoch = get_current_epoch(state) seed = hash(get_seed(state, epoch, DOMAIN_BEACON_PROPOSER) + uint_to_bytes(state.slot)) indices = get_active_validator_indices(state, epoch) return compute_proposer_index(state, indices, seed) ``` Gets the current block proposer. Note that `compute_proposer_index` is maintained separately from this code because in phase 1 we plan to add shard proposer selection code that also uses that function. #### `get_total_balance` ```python def get_total_balance(state: BeaconState, indices: Set[ValidatorIndex]) -> Gwei: """ Return the combined effective balance of the ``indices``. ``EFFECTIVE_BALANCE_INCREMENT`` Gwei minimum to avoid divisions by zero. Math safe up to ~10B ETH, afterwhich this overflows uint64. """ return Gwei(max(EFFECTIVE_BALANCE_INCREMENT, sum([state.validators[index].effective_balance for index in indices]))) ``` Gets the total balance of the given set of validator indices (this is a helper function; we use it to get the total active balance and the total balance approving some FFG vote or shard block). #### `get_total_active_balance` ```python def get_total_active_balance(state: BeaconState) -> Gwei: """ Return the combined effective balance of the active validators. Note: ``get_total_balance`` returns ``EFFECTIVE_BALANCE_INCREMENT`` Gwei minimum to avoid divisions by zero. """ return get_total_balance(state, set(get_active_validator_indices(state, get_current_epoch(state)))) ``` #### `get_domain` ```python def get_domain(state: BeaconState, domain_type: DomainType, epoch: Epoch=None) -> Domain: """ Return the signature domain (fork version concatenated with domain type) of a message. """ epoch = get_current_epoch(state) if epoch is None else epoch fork_version = state.fork.previous_version if epoch < state.fork.epoch else state.fork.current_version return compute_domain(domain_type, fork_version, state.genesis_validators_root) ``` Returns the domain hash (data which gets mixed in with a message being signed) for a particular `DomainType`. This is used to implement domain separation; see [the section on domain separation](#domain_separation) for more info. #### `get_indexed_attestation` ```python def get_indexed_attestation(state: BeaconState, attestation: Attestation) -> IndexedAttestation: """ Return the indexed attestation corresponding to ``attestation``. """ attesting_indices = get_attesting_indices(state, attestation.data, attestation.aggregation_bits) return IndexedAttestation( attesting_indices=sorted(attesting_indices), data=attestation.data, signature=attestation.signature, ) ``` Converts an attestation in the regular format, where the set of signers is defined by a bitfield determining which members of the committee participated, into an attestation that directly contains the validator indices of the participants (ie. the type used in slashings). We have logic to convert from one type of attestation to the other so that the methods for verifying regular attestations and for verifying attestations in slashings can share most of the same code. #### `get_attesting_indices` ```python def get_attesting_indices(state: BeaconState, data: AttestationData, bits: Bitlist[MAX_VALIDATORS_PER_COMMITTEE]) -> Set[ValidatorIndex]: """ Return the set of attesting indices corresponding to ``data`` and ``bits``. """ committee = get_beacon_committee(state, data.slot, data.index) return set(index for i, index in enumerate(committee) if bits[i]) ``` Computes the committee that needed to sign an attestation with particular `AttestationData`, and uses that and the bitfield in the attestation to determine the raw list of validator indices that participated in the attestation. ### Beacon state mutators These methods (no longer pure functions) modify the beacon chain state. #### `increase_balance` ```python def increase_balance(state: BeaconState, index: ValidatorIndex, delta: Gwei) -> None: """ Increase the validator balance at index ``index`` by ``delta``. """ state.balances[index] += delta ``` #### `decrease_balance` ```python def decrease_balance(state: BeaconState, index: ValidatorIndex, delta: Gwei) -> None: """ Decrease the validator balance at index ``index`` by ``delta``, with underflow protection. """ state.balances[index] = 0 if delta > state.balances[index] else state.balances[index] - delta ``` #### `initiate_validator_exit` ```python def initiate_validator_exit(state: BeaconState, index: ValidatorIndex) -> None: """ Initiate the exit of the validator with index ``index``. """ # Return if validator already initiated exit validator = state.validators[index] if validator.exit_epoch != FAR_FUTURE_EPOCH: return # Compute exit queue epoch exit_epochs = [v.exit_epoch for v in state.validators if v.exit_epoch != FAR_FUTURE_EPOCH] exit_queue_epoch = max(exit_epochs + [compute_activation_exit_epoch(get_current_epoch(state))]) exit_queue_churn = len([v for v in state.validators if v.exit_epoch == exit_queue_epoch]) if exit_queue_churn >= get_validator_churn_limit(state): exit_queue_epoch += Epoch(1) # Set validator exit epoch and withdrawable epoch validator.exit_epoch = exit_queue_epoch validator.withdrawable_epoch = Epoch(validator.exit_epoch + MIN_VALIDATOR_WITHDRAWABILITY_DELAY) ``` This function initiates the procedure for a validator to exit, and is called by (i) `VoluntaryExit` processing, (ii) the code enforcing the "eject if under 16 ETH balance" rule, and (iii) slashing. The code here enforces both (i) the "minimum 4 epoch delay rule" and (ii) the exit queue (in the case that too many validators are trying to exit at the same time). The implementation is as follows. Start off with the current epoch + 5 (the current epoch is already partially over so we need +5 to guarantee the delay is >=4 epochs). See if there are already too many validators exiting at that epoch; if there are not, then exit at that epoch, but if there are, instead try the next epoch. This creates a de-facto first-in-first-out queue for exits in the case of congestion. #### `slash_validator` ```python def slash_validator(state: BeaconState, slashed_index: ValidatorIndex, whistleblower_index: ValidatorIndex=None) -> None: """ Slash the validator with index ``slashed_index``. """ epoch = get_current_epoch(state) initiate_validator_exit(state, slashed_index) validator = state.validators[slashed_index] validator.slashed = True validator.withdrawable_epoch = max(validator.withdrawable_epoch, Epoch(epoch + EPOCHS_PER_SLASHINGS_VECTOR)) state.slashings[epoch % EPOCHS_PER_SLASHINGS_VECTOR] += validator.effective_balance decrease_balance(state, slashed_index, validator.effective_balance // MIN_SLASHING_PENALTY_QUOTIENT) # Apply proposer and whistleblower rewards proposer_index = get_beacon_proposer_index(state) if whistleblower_index is None: whistleblower_index = proposer_index whistleblower_reward = Gwei(validator.effective_balance // WHISTLEBLOWER_REWARD_QUOTIENT) proposer_reward = Gwei(whistleblower_reward // PROPOSER_REWARD_QUOTIENT) increase_balance(state, proposer_index, proposer_reward) increase_balance(state, whistleblower_index, Gwei(whistleblower_reward - proposer_reward)) ``` Slashes a validator (ie. forcibly exits and penalizes the validator if they did something provably illegal, eg. signing two conflicting messages in the same epoch). Slashing performs the following actions: * Forcibly exits the validator * Sets the `slashed` flag of that validator to true * Sets a withdrawal delay of 4 eeks (as opposed to the normal ~1 day) * Increments the value in the desired position `state.slashings` array (this is one of those cyclically-rewriting arrays where in the i'th epoch position `i % EPOCHS_PER_SLASHINGS_VECTOR` gets rewritten). This array is used to track the total number of validators slashed, which is used to compute total slashing penalties (often called "anti-correlation penalties") * Decreases their balance by the minimum penalty (1/32 of their balance) * Rewards whoever published the slashing * Rewards the block proposer for including the slashing ## Genesis The main function defined here, `initialize_beacon_state_from_eth1`, takes an eth1 block hash and timestamp and a list of deposits, and generates an eth2 genesis state. All clients will run this function to compute the genesis state when the chain launches for the first time. ### . Before the Ethereum 2.0 genesis has been triggered, and for every Ethereum 1.0 block, let `candidate_state = initialize_beacon_state_from_eth1(eth1_block_hash, eth1_timestamp, deposits)` where: - `eth1_block_hash` is the hash of the Ethereum 1.0 block - `eth1_timestamp` is the Unix timestamp corresponding to `eth1_block_hash` - `deposits` is the sequence of all deposits, ordered chronologically, up to (and including) the block with hash `eth1_block_hash` Eth1 blocks must only be considered once they are at least `SECONDS_PER_ETH1_BLOCK * ETH1_FOLLOW_DISTANCE` seconds old (i.e. `eth1_timestamp + SECONDS_PER_ETH1_BLOCK * ETH1_FOLLOW_DISTANCE <= current_unix_time`). Due to this constraint, if `GENESIS_DELAY < SECONDS_PER_ETH1_BLOCK * ETH1_FOLLOW_DISTANCE`, then the `genesis_time` can happen before the time/state is first known. Values should be configured to avoid this case. ```python def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, eth1_timestamp: uint64, deposits: Sequence[Deposit]) -> BeaconState: fork = Fork( previous_version=GENESIS_FORK_VERSION, current_version=GENESIS_FORK_VERSION, epoch=GENESIS_EPOCH, ) state = BeaconState( genesis_time=eth1_timestamp + GENESIS_DELAY, fork=fork, eth1_data=Eth1Data(block_hash=eth1_block_hash, deposit_count=len(deposits)), latest_block_header=BeaconBlockHeader(body_root=hash_tree_root(BeaconBlockBody())), randao_mixes=[eth1_block_hash] * EPOCHS_PER_HISTORICAL_VECTOR, # Seed RANDAO with Eth1 entropy ) # Process deposits leaves = list(map(lambda deposit: deposit.data, deposits)) for index, deposit in enumerate(deposits): deposit_data_list = List[DepositData, 2**DEPOSIT_CONTRACT_TREE_DEPTH](*leaves[:index + 1]) state.eth1_data.deposit_root = hash_tree_root(deposit_data_list) process_deposit(state, deposit) # Process activations for index, validator in enumerate(state.validators): balance = state.balances[index] validator.effective_balance = min(balance - balance % EFFECTIVE_BALANCE_INCREMENT, MAX_EFFECTIVE_BALANCE) if validator.effective_balance == MAX_EFFECTIVE_BALANCE: validator.activation_eligibility_epoch = GENESIS_EPOCH validator.activation_epoch = GENESIS_EPOCH # Set genesis validators root for domain separation and chain versioning state.genesis_validators_root = hash_tree_root(state.validators) return state ``` *Note*: The ETH1 block with `eth1_timestamp` meeting the minimum genesis active validator count criteria can also occur before `MIN_GENESIS_TIME`. ### Genesis state Let `genesis_state = candidate_state` whenever `is_valid_genesis_state(candidate_state) is True` for the first time. ```python def is_valid_genesis_state(state: BeaconState) -> bool: if state.genesis_time < MIN_GENESIS_TIME: return False if len(get_active_validator_indices(state, GENESIS_EPOCH)) < MIN_GENESIS_ACTIVE_VALIDATOR_COUNT: return False return True ``` *Note*: The `is_valid_genesis_state` function (including `MIN_GENESIS_TIME` and `MIN_GENESIS_ACTIVE_VALIDATOR_COUNT`) is a placeholder for testing. It has yet to be finalized by the community, and can be updated as necessary. The idea here is that you can think of a client as repeatedly attempting to create a genesis state using the algorithm above, but only accepting the state when it satisfies the function above. In reality, clients will not work this way because it is too inefficient (better just track valid eth1 deposits and the timestamp from eth1, and activate when both hit the target). ### Genesis block Let `genesis_block = BeaconBlock(state_root=hash_tree_root(genesis_state))`. ## Beacon chain state transition function Here, we finally get to defining the main function in the spec, which defines how the state is to be modified when a block is processed. The function also has the ability to declare that the block is invalid (this is typically done with either `assert`s, though anything that causes the code to throw an exception, eg. out-of-range list accessed, as well as uint64 overflow or underflow, counts as the block being invalid). We start off with a high-level definition, that breaks it up into two parts: (i) a per-slot state transition (`process_slots`) that takes place in each slot regardless of whether or not there was a block there, and (ii) a per-block state transition that takes the block as input. For example, if a block has slot 66 and its parent has slot 62, then the `process_slot` function would be called four all four slots in between (and `process_slot` would in turn call the epoch-boundary processing function `process_epoch`, because slot 64 is an epoch boundary, between epoch 1 [slots 32...63] and epoch 2 [slots 64...95]). ### . The post-state corresponding to a pre-state `state` and a signed block `signed_block` is defined as `state_transition(state, signed_block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause a `uint64` overflow or underflow are also considered invalid. ```python def state_transition(state: BeaconState, signed_block: SignedBeaconBlock, validate_result: bool=True) -> BeaconState: block = signed_block.message # Process slots (including those with no blocks) since block process_slots(state, block.slot) # Verify signature if validate_result: assert verify_block_signature(state, signed_block) # Process block process_block(state, block) # Verify state root if validate_result: assert block.state_root == hash_tree_root(state) # Return post-state return state ``` ```python def verify_block_signature(state: BeaconState, signed_block: SignedBeaconBlock) -> bool: proposer = state.validators[signed_block.message.proposer_index] signing_root = compute_signing_root(signed_block.message, get_domain(state, DOMAIN_BEACON_PROPOSER)) return bls.Verify(proposer.pubkey, signing_root, signed_block.signature) ``` ```python def process_slots(state: BeaconState, slot: Slot) -> None: assert state.slot < slot while state.slot < slot: process_slot(state) # Process epoch on the start slot of the next epoch if (state.slot + 1) % SLOTS_PER_EPOCH == 0: process_epoch(state) state.slot = Slot(state.slot + 1) ``` Processes all slots between the slot of the parent block and the input slot (which is the current slot), applying the `process_epoch` function if the slot progression crosses an epoch-boundary. <a id="process_slot_notes" /> ```python def process_slot(state: BeaconState) -> None: # Cache state root previous_state_root = hash_tree_root(state) state.state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_state_root # Cache latest block header state root if state.latest_block_header.state_root == Bytes32(): state.latest_block_header.state_root = previous_state_root # Cache block root previous_block_root = hash_tree_root(state.latest_block_header) state.block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_block_root ``` The main function of the `process_slot` function is to update the historical `block_roots` and `state_roots` arrays. The state root manipulation is needed as a clever trick to get around a challenging issue. Namely, we want to include the root of the block at slot `n` into the history in slot `n`. The most natural time to do this is, well, when we are actually processing the block. But this poses a problem to the block creator: the post-state root of the block can only be generated after the state transition is fully processed, but including the block root into the history during slot `n` would require the block's post-state root to be known during the state transition! We get around this via the following tactic. While processing the block at slot `n` (in `process_block`), we add the block header, but zero out the state root. Then, at the beginning of the `process_slot` function of slot N+1 (at which point the state has not yet been modified after it was processed in slot `n`), we edit the saved block header and fill in the post-state root. Note that this requires an extra data structure, `state.latest_block_header`, even though we only _really_ care about storing historical roots; the complexity increase here was deemed worth it to keep the state transition function itself a clean `state_transition(state, block) -> new_state` (as opposed to requiring the previous block as an explicit argument). ### Epoch processing ```python def process_epoch(state: BeaconState) -> None: process_justification_and_finalization(state) process_rewards_and_penalties(state) process_registry_updates(state) process_slashings(state) process_final_updates(state) ``` At the epoch boundary (ie. after the end of the last slot of an epoch), we perform a set of procedures, largely around processing the `PendingAttestations` that have been saved up in the current and previous epoch, though there is also some other work that gets done. First, we define some helper functions: #### Helper functions ```python def get_matching_source_attestations(state: BeaconState, epoch: Epoch) -> Sequence[PendingAttestation]: assert epoch in (get_previous_epoch(state), get_current_epoch(state)) return state.current_epoch_attestations if epoch == get_current_epoch(state) else state.previous_epoch_attestations ``` When [processing attestations](#Attestations), we already only accept attestations that have the correct Casper FFG source checkpoint (specifically, the most recent justified checkpoint that the chain knows about). The goal of this function is to get all attestations that have a correct Casper FFG source. Hence, it can safely just return all the `PendingAttestation`s for the desired epoch (current or previous). ```python def get_matching_target_attestations(state: BeaconState, epoch: Epoch) -> Sequence[PendingAttestation]: return [ a for a in get_matching_source_attestations(state, epoch) if a.data.target.root == get_block_root(state, epoch) ] ``` Returns the subset of `PendingAttestation`s that have the correct Casper FFG target (ie. the checkpoint that is part of the current chain). ```python def get_matching_head_attestations(state: BeaconState, epoch: Epoch) -> Sequence[PendingAttestation]: return [ a for a in get_matching_target_attestations(state, epoch) if a.data.beacon_block_root == get_block_root_at_slot(state, a.data.slot) ] ``` Returns the subset of `PendingAttestation`s that have the correct head (ie. they voted for a head that ended up actually being the head of the chain). ```python def get_unslashed_attesting_indices(state: BeaconState, attestations: Sequence[PendingAttestation]) -> Set[ValidatorIndex]: output = set() # type: Set[ValidatorIndex] for a in attestations: output = output.union(get_attesting_indices(state, a.data, a.aggregation_bits)) return set(filter(lambda index: not state.validators[index].slashed, output)) ``` Gets the list of attesting indices from a set of attestations, filtering out the indices that have been slashed. The idea here is that if you get slashed, you are still "technically" part of the validator set (see the [note on the validator life cycle](#lifecycle) for reasoning why), but your attestations do not get counted. ```python def get_attesting_balance(state: BeaconState, attestations: Sequence[PendingAttestation]) -> Gwei: """ Return the combined effective balance of the set of unslashed validators participating in ``attestations``. Note: ``get_total_balance`` returns ``EFFECTIVE_BALANCE_INCREMENT`` Gwei minimum to avoid divisions by zero. """ return get_total_balance(state, get_unslashed_attesting_indices(state, attestations)) ``` Gets the total attesting balance (excluding slashed validators) from a list of attestations. In the functions below, we'll see a pattern. There are four main properties of an attestation that eth2 is concerned with, both for internal recordkeeping, and for reward/penalty accounting: * Having the correct FFG source * Having the correct FFG target * Having the correct chain head * Having the correct shard block (in phase 1+ only) For each one of these, we'll use one of the helpers defined above to determine the set of all validator indices that have that property in their attestations. We will then use this information to (i) reward or penalize them, and (ii) count their balance toward a total. The total is sometimes itself used when calculating rewards/penalties, but also to determine if 2/3 thresholds for Casper FFG or for shard committees have been met. We care about justifications included in the current _and_ the previous block, because it's possible that some attestations of a slot in the previous block were included in the current epoch, and so we need to combine together attestations on both sides of the boundary. If this were not done, a few proposers at the end of an epoch being malicious could easily prevent the chain from detecting justification and finality. #### Justification and finalization ```python def process_justification_and_finalization(state: BeaconState) -> None: if get_current_epoch(state) <= GENESIS_EPOCH + 1: return previous_epoch = get_previous_epoch(state) current_epoch = get_current_epoch(state) old_previous_justified_checkpoint = state.previous_justified_checkpoint old_current_justified_checkpoint = state.current_justified_checkpoint # Process justifications state.previous_justified_checkpoint = state.current_justified_checkpoint state.justification_bits[1:] = state.justification_bits[:JUSTIFICATION_BITS_LENGTH - 1] state.justification_bits[0] = 0b0 matching_target_attestations = get_matching_target_attestations(state, previous_epoch) # Previous epoch if get_attesting_balance(state, matching_target_attestations) * 3 >= get_total_active_balance(state) * 2: state.current_justified_checkpoint = Checkpoint(epoch=previous_epoch, root=get_block_root(state, previous_epoch)) state.justification_bits[1] = 0b1 matching_target_attestations = get_matching_target_attestations(state, current_epoch) # Current epoch if get_attesting_balance(state, matching_target_attestations) * 3 >= get_total_active_balance(state) * 2: state.current_justified_checkpoint = Checkpoint(epoch=current_epoch, root=get_block_root(state, current_epoch)) state.justification_bits[0] = 0b1 # Process finalizations bits = state.justification_bits # The 2nd/3rd/4th most recent epochs are justified, the 2nd using the 4th as source if all(bits[1:4]) and old_previous_justified_checkpoint.epoch + 3 == current_epoch: state.finalized_checkpoint = old_previous_justified_checkpoint # The 2nd/3rd most recent epochs are justified, the 2nd using the 3rd as source if all(bits[1:3]) and old_previous_justified_checkpoint.epoch + 2 == current_epoch: state.finalized_checkpoint = old_previous_justified_checkpoint # The 1st/2nd/3rd most recent epochs are justified, the 1st using the 3rd as source if all(bits[0:3]) and old_current_justified_checkpoint.epoch + 2 == current_epoch: state.finalized_checkpoint = old_current_justified_checkpoint # The 1st/2nd most recent epochs are justified, the 1st using the 2nd as source if all(bits[0:2]) and old_current_justified_checkpoint.epoch + 1 == current_epoch: state.finalized_checkpoint = old_current_justified_checkpoint ``` This function processes the beacon chain's own recordkeeping of which justified and finalized blocks in its own history it knows about. Roughly the first half of this function checks if the checkpoint at the beginning of the current epoch has been justified, meaning 2/3 of active validators voted for it (remember: that's the epoch we're currently at the very end of), and also does that check for the previous epoch. This data gets saved in the `justification_bits` array, which keeps track of which recent epochs have been justified. The second half of the code uses this justification history, as well as what epoch was used as a source in the current or previous epoch, to determine whether or not a block is finalized (see [the Gasper paper](https://arxiv.org/abs/2003.03052) for how this works). #### Rewards and penalties ##### Helpers ```python def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: total_balance = get_total_active_balance(state) effective_balance = state.validators[index].effective_balance return Gwei(effective_balance * BASE_REWARD_FACTOR // integer_squareroot(total_balance) // BASE_REWARDS_PER_EPOCH) ``` This is the reward that almost all other rewards in ethereum are computed as a multiple of. Particularly, note that it's a desired goal of the spec that `effective_balance * BASE_REWARD_FACTOR // integer_squareroot(total_balance)` is the average per-epoch reward received by a validator under theoretical best-case conditions; to achieve this, the base reward equals that amount divided by `BASE_REWARDS_PER_EPOCH`, which is the number of times that a reward of this size will be applied. ```python def get_proposer_reward(state: BeaconState, attesting_index: ValidatorIndex) -> Gwei: return Gwei(get_base_reward(state, attesting_index) // PROPOSER_REWARD_QUOTIENT) ``` Proposers get a reward equal to up to 1/8 of the base reward for every attester in an attestation they include (though they also get other rewards for including slashings, and in phase 1+ other kinds of objects too) ```python def get_finality_delay(state: BeaconState) -> uint64: return get_previous_epoch(state) - state.finalized_checkpoint.epoch ``` Gets the number of blocks since the chain was last finalized. ```python def is_in_inactivity_leak(state: BeaconState) -> bool: return get_finality_delay(state) > MIN_EPOCHS_TO_INACTIVITY_PENALTY ``` If the chain has not been finalized for >4 epochs, the chain enters an "inactivity leak" mode, where inactive validators get progressively penalized more and more, to reduce their influence until blocks get finalized again. See [here](#inactivity-quotient) for what the inactivity leak is, what it's for and how it works. ```python def get_eligible_validator_indices(state: BeaconState) -> Sequence[ValidatorIndex]: previous_epoch = get_previous_epoch(state) return [ ValidatorIndex(index) for index, v in enumerate(state.validators) if is_active_validator(v, previous_epoch) or (v.slashed and previous_epoch + 1 < v.withdrawable_epoch) ] ``` Both active validators and slashed-but-not-yet-withdrawn validators are eligible to receive penalties. This is done to prevent self-slashing from being a way to escape inactivity leaks. ```python def get_attestation_component_deltas(state: BeaconState, attestations: Sequence[PendingAttestation] ) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ Helper with shared logic for use by get source, target, and head deltas functions """ rewards = [Gwei(0)] * len(state.validators) penalties = [Gwei(0)] * len(state.validators) total_balance = get_total_active_balance(state) unslashed_attesting_indices = get_unslashed_attesting_indices(state, attestations) attesting_balance = get_total_balance(state, unslashed_attesting_indices) for index in get_eligible_validator_indices(state): if index in unslashed_attesting_indices: increment = EFFECTIVE_BALANCE_INCREMENT # Factored out from balance totals to avoid uint64 overflow if is_in_inactivity_leak(state): # Since full base reward will be canceled out by inactivity penalty deltas, # optimal participation receives full base reward compensation here. rewards[index] += get_base_reward(state, index) else: reward_numerator = get_base_reward(state, index) * (attesting_balance // increment) rewards[index] += reward_numerator // (total_balance // increment) else: penalties[index] += get_base_reward(state, index) return rewards, penalties ``` This is a helper function that outputs a list of rewards and penalties for validators; it is used for correct-source, correct-target and correct-head rewards. The general approach is: if portion `p` (eg. `p=0.9` for 90%) of validators achieve some property in their attestations, then those validators get a reward of `base_reward * p`, and the validators that did not achieve that property get a penalty of `base_reward`. We need penalties to ensure that validating is only net-profitable if you are online at least ~2/3 of the time (in reality the numbers are _slightly_ more forgiving than that, but not by much). We don't want validators that cannot meet that minimum level of liveness, as such validators would hurt more than they help by hindering finality (which requires 2/3 online). This rule that your rewards decrease if other validators do less well was included to disincentivize harming other validators; see [my writing on discouragement attacks](https://github.com/ethereum/research/blob/master/papers/discouragement/discouragement.pdf) (and [Barnabe's summary](https://hackingresear.ch/discouragement-attacks/)) for reasoning why this is a good idea. ##### Components of attestation deltas ```python def get_source_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ Return attester micro-rewards/penalties for source-vote for each validator. """ matching_source_attestations = get_matching_source_attestations(state, get_previous_epoch(state)) return get_attestation_component_deltas(state, matching_source_attestations) ``` ```python def get_target_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ Return attester micro-rewards/penalties for target-vote for each validator. """ matching_target_attestations = get_matching_target_attestations(state, get_previous_epoch(state)) return get_attestation_component_deltas(state, matching_target_attestations) ``` ```python def get_head_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ Return attester micro-rewards/penalties for head-vote for each validator. """ matching_head_attestations = get_matching_head_attestations(state, get_previous_epoch(state)) return get_attestation_component_deltas(state, matching_head_attestations) ``` The above three functions just use the `get_attestation_component_deltas` helper to compute rewards and penalties for correct FFG source, correct FFG target and correct head, respectively. ```python def get_inclusion_delay_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ Return proposer and inclusion delay micro-rewards/penalties for each validator. """ rewards = [Gwei(0) for _ in range(len(state.validators))] matching_source_attestations = get_matching_source_attestations(state, get_previous_epoch(state)) for index in get_unslashed_attesting_indices(state, matching_source_attestations): attestation = min([ a for a in matching_source_attestations if index in get_attesting_indices(state, a.data, a.aggregation_bits) ], key=lambda a: a.inclusion_delay) rewards[attestation.proposer_index] += get_proposer_reward(state, index) max_attester_reward = get_base_reward(state, index) - get_proposer_reward(state, index) rewards[index] += Gwei(max_attester_reward // attestation.inclusion_delay) # No penalties associated with inclusion delay penalties = [Gwei(0) for _ in range(len(state.validators))] return rewards, penalties ``` This function processes rewards for getting your attestation included quickly: a full base reward if it gets included in the next slot, and `1/k` of a base reward if it gets included after `k` slots. This incentivizes promptness, reducing incentive to wait more than a slot to make sure you have the correct target or head. ```python def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ Return inactivity reward/penalty deltas for each validator. """ penalties = [Gwei(0) for _ in range(len(state.validators))] if is_in_inactivity_leak(state): matching_target_attestations = get_matching_target_attestations(state, get_previous_epoch(state)) matching_target_attesting_indices = get_unslashed_attesting_indices(state, matching_target_attestations) for index in get_eligible_validator_indices(state): # If validator is performing optimally this cancels all rewards for a neutral balance base_reward = get_base_reward(state, index) penalties[index] += Gwei(BASE_REWARDS_PER_EPOCH * base_reward - get_proposer_reward(state, index)) if index not in matching_target_attesting_indices: effective_balance = state.validators[index].effective_balance penalties[index] += Gwei(effective_balance * get_finality_delay(state) // INACTIVITY_PENALTY_QUOTIENT) # No rewards associated with inactivity penalties rewards = [Gwei(0) for _ in range(len(state.validators))] return rewards, penalties ``` This code implements the [inactivity leak](#inactivity-quotient). ##### `get_attestation_deltas` ```python def get_attestation_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ Return attestation reward/penalty deltas for each validator. """ source_rewards, source_penalties = get_source_deltas(state) target_rewards, target_penalties = get_target_deltas(state) head_rewards, head_penalties = get_head_deltas(state) inclusion_delay_rewards, _ = get_inclusion_delay_deltas(state) _, inactivity_penalties = get_inactivity_penalty_deltas(state) rewards = [ source_rewards[i] + target_rewards[i] + head_rewards[i] + inclusion_delay_rewards[i] for i in range(len(state.validators)) ] penalties = [ source_penalties[i] + target_penalties[i] + head_penalties[i] + inactivity_penalties[i] for i in range(len(state.validators)) ] return rewards, penalties ``` This function combines together rewards and penalties from all of the above sources into the total rewards and penalties. ##### `process_rewards_and_penalties` ```python def process_rewards_and_penalties(state: BeaconState) -> None: if get_current_epoch(state) == GENESIS_EPOCH: return rewards, penalties = get_attestation_deltas(state) for index in range(len(state.validators)): increase_balance(state, ValidatorIndex(index), rewards[index]) decrease_balance(state, ValidatorIndex(index), penalties[index]) ``` This function combines all of the above logic together, and actually processes these rewards and penalties. #### Registry updates ```python def process_registry_updates(state: BeaconState) -> None: # Process activation eligibility and ejections for index, validator in enumerate(state.validators): if is_eligible_for_activation_queue(validator): validator.activation_eligibility_epoch = get_current_epoch(state) + 1 if is_active_validator(validator, get_current_epoch(state)) and validator.effective_balance <= EJECTION_BALANCE: initiate_validator_exit(state, ValidatorIndex(index)) # Queue validators eligible for activation and not yet dequeued for activation activation_queue = sorted([ index for index, validator in enumerate(state.validators) if is_eligible_for_activation(state, validator) # Order by the sequence of activation_eligibility_epoch setting and then index ], key=lambda index: (state.validators[index].activation_eligibility_epoch, index)) # Dequeued validators for activation up to churn limit for index in activation_queue[:get_validator_churn_limit(state)]: validator = state.validators[index] validator.activation_epoch = compute_activation_exit_epoch(get_current_epoch(state)) ``` This function processes (i) the validator activation queue, and (ii) the rule that validators with <= 16 ETH get ejected. Note that the validator activation queue is implemented in a more complex way than the exit queue, which simply immediately assigns exit epochs. The reason why we cannot do that here is that we want to only process activations if the activation was initiated in a block that is already finalized. This is done to ensure that, except in the extreme case where two conflicting blocks have been finalized, any validator that has is active on one chain must also have been at least assigned an index on the other chain (and the same index on both sides). This is done to make sure `indexed_attestations` produced by one chain can be processed on the other chain for slashings. If one chain could contain validators that were completely unknown on the other chain, slashing processing would break, as the other chain would not know the public key for those validators (and including the public key would have been more space-inefficient; 48 bytes per validator instead of 3 bytes). Note that if two conflicting blocks _do_ get finalized, the first time that happens must have been done by attestations that shared a common last finalized block, and so at that point slashing for the double-finalization can happen. #### Slashings ```python def process_slashings(state: BeaconState) -> None: epoch = get_current_epoch(state) total_balance = get_total_active_balance(state) for index, validator in enumerate(state.validators): if validator.slashed and epoch + EPOCHS_PER_SLASHINGS_VECTOR // 2 == validator.withdrawable_epoch: increment = EFFECTIVE_BALANCE_INCREMENT # Factored out from penalty numerator to avoid uint64 overflow penalty_numerator = validator.effective_balance // increment * min(sum(state.slashings) * 3, total_balance) penalty = penalty_numerator // total_balance * increment decrease_balance(state, ValidatorIndex(index), penalty) ``` This is the code that processes the anti-correlation penalty rule (that if you get slashed, you can penalized a portion of your balance equal to three times the portion of all depositors that got slashed around the same time period; see [here](https://notes.ethereum.org/@vbuterin/rkhCgQteN?type=view#Slashing-and-anti-correlation-penalties) for why this is done). The idea is that `state.slashings` is a cyclically-overwriting array that stores the last 4 eeks worth of total slashings in each epoch (eg. if the current epoch is 53 and if `EPOCHS_PER_SLASHINGS_VECTOR` were equal to 10, its elements would store the total slashings in epochs `[50, 51, 52, 53, 44, 45, 46, 47, 48, 49]` respectively). Hence, if we simply take its sum, we get the total slashings in the last `EPOCHS_PER_SLASHINGS_VECTOR` epochs. Note that we do this _halfway through_ the 4-eek period that is both the mandatory withdrawal delay for slashed validators and the length of the slashings vector. This means that if you get slashed, your penalty is calculated based on the portion of validator slashed in the period (2 eeks before you were slashed ... 2 eeks after you were slashed). #### Final updates ```python def process_final_updates(state: BeaconState) -> None: current_epoch = get_current_epoch(state) next_epoch = Epoch(current_epoch + 1) # Reset eth1 data votes if next_epoch % EPOCHS_PER_ETH1_VOTING_PERIOD == 0: state.eth1_data_votes = [] # Update effective balances with hysteresis for index, validator in enumerate(state.validators): balance = state.balances[index] HYSTERESIS_INCREMENT = EFFECTIVE_BALANCE_INCREMENT // HYSTERESIS_QUOTIENT DOWNWARD_THRESHOLD = HYSTERESIS_INCREMENT * HYSTERESIS_DOWNWARD_MULTIPLIER UPWARD_THRESHOLD = HYSTERESIS_INCREMENT * HYSTERESIS_UPWARD_MULTIPLIER if ( balance + DOWNWARD_THRESHOLD < validator.effective_balance or validator.effective_balance + UPWARD_THRESHOLD < balance ): validator.effective_balance = min(balance - balance % EFFECTIVE_BALANCE_INCREMENT, MAX_EFFECTIVE_BALANCE) # Reset slashings state.slashings[next_epoch % EPOCHS_PER_SLASHINGS_VECTOR] = Gwei(0) # Set randao mix state.randao_mixes[next_epoch % EPOCHS_PER_HISTORICAL_VECTOR] = get_randao_mix(state, current_epoch) # Set historical root accumulator if next_epoch % (SLOTS_PER_HISTORICAL_ROOT // SLOTS_PER_EPOCH) == 0: historical_batch = HistoricalBatch(block_roots=state.block_roots, state_roots=state.state_roots) state.historical_roots.append(hash_tree_root(historical_batch)) # Rotate current/previous epoch attestations state.previous_epoch_attestations = state.current_epoch_attestations state.current_epoch_attestations = [] ``` This function does a few miscellaneous operations, particularly: * Resetting eth1 data votes at the end of every 1024-slot (32-epoch) voting period * Updating validators' effective balances based on changes to their exact balances (see [the section on hysteresis](#hysteresis) for more details) * Resets the value for the next epoch in the slashings vector to zero (so that by the end of the next epoch, the number in the slashings vector in that position reflects _just_ validators slashed in that epoch) * Updates some storage of historical variables (randomness seeds and historical batch roots) * Shifts the list of `PendingAttestations` for the "current" epoch to the list meant for the "previous" epoch ### Block processing This next section, finally, deals with the procedure for processing a block itself. This part is actually surprisingly not-that-complicated; the bulk of the complexity is in either the helpers or in end-of-epoch processing. ```python def process_block(state: BeaconState, block: BeaconBlock) -> None: process_block_header(state, block) process_randao(state, block.body) process_eth1_data(state, block.body) process_operations(state, block.body) ``` There are four main components that we process: * The block header * Updating the randomness seed * Eth1 data voting * "Operations" (attestations, slashings, `VoluntaryExit`s.....) #### Block header ```python def process_block_header(state: BeaconState, block: BeaconBlock) -> None: # Verify that the slots match assert block.slot == state.slot # Verify that the block is newer than latest block header assert block.slot > state.latest_block_header.slot # Verify that proposer index is the correct index assert block.proposer_index == get_beacon_proposer_index(state) # Verify that the parent matches assert block.parent_root == hash_tree_root(state.latest_block_header) # Cache current block as the new latest block state.latest_block_header = BeaconBlockHeader( slot=block.slot, proposer_index=block.proposer_index, parent_root=block.parent_root, state_root=Bytes32(), # Overwritten in the next process_slot call body_root=hash_tree_root(block.body), ) # Verify proposer is not slashed proposer = state.validators[block.proposer_index] assert not proposer.slashed ``` This is fairly self-explanatory; just checking a few basic correctness properties of the block, and storing the block header in the cache without its state root (as we don't know its state roots yet; see [the section on `process_slot`](#process_slot_notes) to understand more fully what's going on there). #### RANDAO ```python def process_randao(state: BeaconState, body: BeaconBlockBody) -> None: epoch = get_current_epoch(state) # Verify RANDAO reveal proposer = state.validators[get_beacon_proposer_index(state)] signing_root = compute_signing_root(epoch, get_domain(state, DOMAIN_RANDAO)) assert bls.Verify(proposer.pubkey, signing_root, body.randao_reveal) # Mix in RANDAO reveal mix = xor(get_randao_mix(state, epoch), hash(body.randao_reveal)) state.randao_mixes[epoch % EPOCHS_PER_HISTORICAL_VECTOR] = mix ``` See [the section on seeds](#seeds) to understand what's going on here. #### Eth1 data ```python def process_eth1_data(state: BeaconState, body: BeaconBlockBody) -> None: state.eth1_data_votes.append(body.eth1_data) if state.eth1_data_votes.count(body.eth1_data) * 2 > EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH: state.eth1_data = body.eth1_data ``` Store vote counts for every eth1 block that has votes; if any eth1 block wins majority support within a 1024-slot voting period, formally accept that eth1 block and set it as the official "latest known eth1 block" in the eth2 state. #### Operations ```python def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: # Verify that outstanding deposits are processed up to the maximum number of deposits assert len(body.deposits) == min(MAX_DEPOSITS, state.eth1_data.deposit_count - state.eth1_deposit_index) def for_ops(operations: Sequence[Any], fn: Callable[[BeaconState, Any], None]) -> None: for operation in operations: fn(state, operation) for_ops(body.proposer_slashings, process_proposer_slashing) for_ops(body.attester_slashings, process_attester_slashing) for_ops(body.attestations, process_attestation) for_ops(body.deposits, process_deposit) for_ops(body.voluntary_exits, process_voluntary_exit) ``` Basically, for each type of operation in the block, run its associated function. Also verify that the maximum possible number of deposits is included. Note that there are maximums on all operation types, though they do not need to be explicitly enforced here because they are already included in the [beacon block body SSZ data type](#BeaconBlockBody). ##### Proposer slashings ```python def process_proposer_slashing(state: BeaconState, proposer_slashing: ProposerSlashing) -> None: header_1 = proposer_slashing.signed_header_1.message header_2 = proposer_slashing.signed_header_2.message # Verify header slots match assert header_1.slot == header_2.slot # Verify header proposer indices match assert header_1.proposer_index == header_2.proposer_index # Verify the headers are different assert header_1 != header_2 # Verify the proposer is slashable proposer = state.validators[header_1.proposer_index] assert is_slashable_validator(proposer, get_current_epoch(state)) # Verify signatures for signed_header in (proposer_slashing.signed_header_1, proposer_slashing.signed_header_2): domain = get_domain(state, DOMAIN_BEACON_PROPOSER, compute_epoch_at_slot(signed_header.message.slot)) signing_root = compute_signing_root(signed_header.message, domain) assert bls.Verify(proposer.pubkey, signing_root, signed_header.signature) slash_validator(state, header_1.proposer_index) ``` Slashes a validator that proposed two different blocks in the same slot. ##### Attester slashings ```python def process_attester_slashing(state: BeaconState, attester_slashing: AttesterSlashing) -> None: attestation_1 = attester_slashing.attestation_1 attestation_2 = attester_slashing.attestation_2 assert is_slashable_attestation_data(attestation_1.data, attestation_2.data) assert is_valid_indexed_attestation(state, attestation_1) assert is_valid_indexed_attestation(state, attestation_2) slashed_any = False indices = set(attestation_1.attesting_indices).intersection(attestation_2.attesting_indices) for index in sorted(indices): if is_slashable_validator(state.validators[index], get_current_epoch(state)): slash_validator(state, index) slashed_any = True assert slashed_any ``` Given two attestations (contained in an `AttesterSlashing`): * Verifies that the two attestations conflict (ie. they trigger the Casper FFG slashing rules) * Verifies both attestations are correct * Computes the intersection of the participant sets of the two attestations. Verifies that the intersection is nonempty, and slashes anyone in the intersection. ##### Attestations ```python def process_attestation(state: BeaconState, attestation: Attestation) -> None: data = attestation.data assert data.target.epoch in (get_previous_epoch(state), get_current_epoch(state)) assert data.target.epoch == compute_epoch_at_slot(data.slot) assert data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot <= data.slot + SLOTS_PER_EPOCH assert data.index < get_committee_count_per_slot(state, data.target.epoch) committee = get_beacon_committee(state, data.slot, data.index) assert len(attestation.aggregation_bits) == len(committee) pending_attestation = PendingAttestation( data=data, aggregation_bits=attestation.aggregation_bits, inclusion_delay=state.slot - data.slot, proposer_index=get_beacon_proposer_index(state), ) if data.target.epoch == get_current_epoch(state): assert data.source == state.current_justified_checkpoint state.current_epoch_attestations.append(pending_attestation) else: assert data.source == state.previous_justified_checkpoint state.previous_epoch_attestations.append(pending_attestation) # Verify signature assert is_valid_indexed_attestation(state, get_indexed_attestation(state, attestation)) ``` In order to ensure the chain actually finalizes, we force attesters to (i) use the latest justified block as their source, and (ii) use the correct epoch for their target (though possibly the wrong block, as the target block may not be stabilized as part of the chain yet). We do some basic sanity-checking (the attestation is not from the future, and the attestation committee index is not >= the number of committees in that slot). We then verify the attestation, and save it as a `PendingAttestation`, leaving more detailed processing of all attestations until the end of the epoch. ##### Deposits ```python def get_validator_from_deposit(state: BeaconState, deposit: Deposit) -> Validator: amount = deposit.data.amount effective_balance = min(amount - amount % EFFECTIVE_BALANCE_INCREMENT, MAX_EFFECTIVE_BALANCE) return Validator( pubkey=deposit.data.pubkey, withdrawal_credentials=deposit.data.withdrawal_credentials, activation_eligibility_epoch=FAR_FUTURE_EPOCH, activation_epoch=FAR_FUTURE_EPOCH, exit_epoch=FAR_FUTURE_EPOCH, withdrawable_epoch=FAR_FUTURE_EPOCH, effective_balance=effective_balance, ) ``` Converts a `Deposit` record (created by the eth1 deposit contract) into a `Validator` object that goes into the eth2 state. ```python def process_deposit(state: BeaconState, deposit: Deposit) -> None: # Verify the Merkle branch assert is_valid_merkle_branch( leaf=hash_tree_root(deposit.data), branch=deposit.proof, depth=DEPOSIT_CONTRACT_TREE_DEPTH + 1, # Add 1 for the List length mix-in index=state.eth1_deposit_index, root=state.eth1_data.deposit_root, ) # Deposits must be processed in order state.eth1_deposit_index += 1 pubkey = deposit.data.pubkey amount = deposit.data.amount validator_pubkeys = [v.pubkey for v in state.validators] if pubkey not in validator_pubkeys: # Verify the deposit signature (proof of possession) which is not checked by the deposit contract deposit_message = DepositMessage( pubkey=deposit.data.pubkey, withdrawal_credentials=deposit.data.withdrawal_credentials, amount=deposit.data.amount, ) domain = compute_domain(DOMAIN_DEPOSIT) # Fork-agnostic domain since deposits are valid across forks signing_root = compute_signing_root(deposit_message, domain) if not bls.Verify(pubkey, signing_root, deposit.data.signature): return #BeaconBlockBody # Add validator and balance entries state.validators.append(get_validator_from_deposit(state, deposit)) state.balances.append(amount) else: # Increase balance by deposit amount index = ValidatorIndex(validator_pubkeys.index(pubkey)) increase_balance(state, index, amount) ``` Processes a deposit; this includes (i) verifying the Merkle branch, proving the deposit is actually part of the deposit tree created by the eth1 deposit contract, (ii) verifying that deposits are being processed in order, (iii) verify the signature on the deposit, and finally (iv) adding it to the validator set. If the deposit pubkey is already in the validator set, the deposit is instead treated as a balance top-up. (Note: yes, balance top-ups do _kinda_ get around activation queues, but note that for an attacker to benefit from this, they need to have already lost the ETH that is being topped up [since depositing requires 32 ETH and 32 ETH is the maximum effective balance], so it is not actually an attack vector) ##### Voluntary exits ```python def process_voluntary_exit(state: BeaconState, signed_voluntary_exit: SignedVoluntaryExit) -> None: voluntary_exit = signed_voluntary_exit.message validator = state.validators[voluntary_exit.validator_index] # Verify the validator is active assert is_active_validator(validator, get_current_epoch(state)) # Verify exit has not been initiated assert validator.exit_epoch == FAR_FUTURE_EPOCH # Exits must specify an epoch when they become valid; they are not valid before then assert get_current_epoch(state) >= voluntary_exit.epoch # Verify the validator has been active long enough assert get_current_epoch(state) >= validator.activation_epoch + SHARD_COMMITTEE_PERIOD # Verify signature domain = get_domain(state, DOMAIN_VOLUNTARY_EXIT, voluntary_exit.epoch) signing_root = compute_signing_root(voluntary_exit, domain) assert bls.Verify(validator.pubkey, signing_root, signed_voluntary_exit.signature) # Initiate exit initiate_validator_exit(state, voluntary_exit.validator_index) ``` Validators voluntarily exiting the validator set.