# EIP-8411 payload segmentation under Shadow: a cross-check of the simulations *Technical report attached to [EIP-8411: what segmented payload diffusion is made of](https://ethresear.ch/t/eip-8411-what-segmented-payload-diffusion-is-made-of/26025). A summary of the report below also appeared as [EIP-8411 payload segmentation under the Shadow simulator](https://ethresear.ch/t/eip-8411-payload-segmentation-under-the-shadow-simulator/26070).* **TL;DR** - We reran that post's **Figure 1 and its network-size and payload-size sweeps** under a second simulator, Shadow, over QUIC and over TCP, then ran the same binary on **the real Linux network stack**. - **A-tuned reproduces** the harness's results within a few percent at every network size, and every segmented variant stays **faster than the whole message** on both transports. - The simulators disagree on the whole message and on variant C; **the order of a node's uplink queue** explains it: switching Shadow's TCP queue to a fair share reverses the whole message's advantage and halves C's, leaving a residual of about a tenth for C unexplained. - **B's completion tail** is longer only under Shadow over QUIC; capping the shared UDP send buffer removes about a third of the excess, the rest is open. - On one Linux machine, 100 processes on the kernel's stack with the links shaped by its traffic control, A-tuned and C over QUIC differ from the simulators by **as much as the simulators differ from each other**. - The whole message keeps that machine **two to four times busier** than the segmented variants, and above 100 nodes host saturation, not the network, sets the latency. - None of this tests **the bandwidth model, a full client, or a node's CPU cost on its own machine**. ## 1. The question and scope of the cross-check The earlier post's results come from an in-process harness in the Prysm research branch. It runs the real go-libp2p stack and quic-go over go-libp2p's simulated network ([simnet](https://github.com/marcopolo/simnet)), under Go's virtual clock (`testing/synctest`). This study tests whether those results survive a different simulator and transport. Figure numbers in this report are that post's: Figure 1 is its main comparison, Figure 8 its network-size sweep, Figure 9 its payload-size sweep. [Shadow](https://shadow.github.io/) runs one real process per node on real sockets, intercepts system calls, and also uses a virtual clock. Running the same code and topology there checks the path between the application and socket under real Go scheduling. Shadow's own TCP stack also allows a TCP-with-yamux comparison that the harness cannot produce. Agreement increases confidence in the harness's wiring and transport modelling. Both simulators omit CPU cost and the full client, and share the bandwidth model. Their agreement therefore provides no new evidence about hashing, proofs, coding cost or that model. The Linux check in [section 10](#10-The-Linux-check-kernel-agreement-and-shared-host-CPU-cost) tests the kernel network stack and CPU time on the wall clock. One machine carries 100 nodes at degree 70 cleanly for the segmented arms; larger cells measure host saturation. This limits the scope of the check, while making visible a whole-message CPU cost that neither simulator charges. ## 2. Method ### Protocol variants and implementation tiers Segmentation allows forwarding before the whole payload arrives. **A** carries segments on one gossipsub topic; **A-tuned** combines this with selective pushes and disciplined requests. **B** uses the partial-message extension, claiming each part from a peer. **C** uses one gossipsub topic per segment index, so its meshes forward concurrently. Coded variants add Reed–Solomon parity; the compress-first variants compress the payload before coding. The payload-size comparison distinguishes three implementation tiers: segmentation and batching; size-adaptive A with request discipline; and size-adaptive A with compress-first coding. Its reference arms separate the effects of coding and unit size. Appendix A gives the exact policies and knobs. The **phase shift** is phase forwarding without the pull discipline: it pushes every segment to the whole mesh under a budget that decays on IDONTWANT. It is included to distinguish forwarding policy from the request discipline used by A-tuned. ### Paired cells and statistics **The node.** Shadow runs the harness's own driver as one process per host, with the same arm builders, knob-to-option mapping, payload, tracer and completion rule as the post's cells. Only the network and the driver's barriers change. Shadow provides the network, and the barriers (peers up, mesh settled, publish) become fixed instants on the simulated clock. Shadow is version 3.3.0 with three local fixes of a few lines each (Appendix B); its memory manager is switched off because it cannot run this Go binary. **The cell.** Each cell uses the published configuration. The harness exports its topology draw for that seed to Shadow: 500 nodes at degree 70, the geographic latency model, 50 Mbps up and 100 down at home nodes, and the post's three scenarios. Both simulators therefore use the same graph, links and latencies. The publisher sends at a fixed instant. Receivers report completion times and byte counters, and the record has the same fields as the post's. **What each comparison tests.** Published versus twin checks code changes made for Shadow compatibility: the twin runs the exact binary used under Shadow with the same seed. Twin versus Shadow QUIC checks the simulation framework. Shadow QUIC versus Shadow TCP checks transport dependence. The Linux cells separately check the kernel stack and wall clock. | backend | clock | network stack | uplink queue | transport | comparison purpose | | --- | --- | --- | --- | --- | --- | | Published harness and harness twin | Go virtual clock | go-libp2p simnet with quic-go | FQ-CoDel | QUIC | Published versus twin checks binary changes. | | Shadow | Shadow virtual clock | Shadow sockets and TCP; quic-go over UDP | FIFO; socket round-robin in queue interventions | QUIC or TCP + yamux | Twin versus Shadow checks the simulator; QUIC versus TCP checks transport. | | Linux namespaces | Wall clock | Kernel stack; quic-go over UDP | FIFO or fq_codel | QUIC or TCP + yamux | Checks the kernel and CPU time at the scale the shared host can carry. | **Names.** An arm specifies a policy and a unit, and the report names both. `atuned_32k` is the post's A-tuned on 32 KiB segments, `atuned_16k` its 16 KiB reference, and `c_32k` variant C on 32 topics. The full table is in Appendix A. Every record includes its wire form (message count, unit, id width). Records with different wire forms are excluded from comparison. **Statistic.** For each arm, transport and point, we report the median across seeds on each side and the difference of Shadow from the twin as a percentage of the twin's median (a ratio of 2 or more is written as a multiplier). Agreement requires a difference within the harness's own seed-to-seed spread (a few percent) and the same ranking of the arms. One-seed points have no error bars and support only qualitative comparisons. <a name="assumptions"></a> ### Shared assumptions Both simulators use per-node uplink and downlink limits and fixed one-way latency per pair. Neither charges CPU time or runs the full client. The assumptions below apply to TCP and QUIC. **Cold transports.** Each connection starts carrying the payload in slow start, with the libraries' default windows. quic-go (0.59) opens with a 40 KiB congestion window (32 packets of 1 280 bytes, 3.2 times what RFC 9002 recommends). Its flow-control windows start at 512 KiB per stream and 768 KiB per connection, growing to 10 MiB and 15 MiB. Shadow's TCP is Reno with an initial window of ten segments, about 15 KB. yamux adds a 256 KiB stream window growing to 16 MiB. A 1 MiB whole message therefore spends its first round trips window-bound on every hop, while a 32 KiB segment fits QUIC's initial window. The harness has a warm-up mode that first diffuses a payload of the same shape, so every connection has left slow start before the measured publish. It also has a fork knob that changes quic-go's initial window. The post's figures and the simulator comparisons use cold connections; the warm-condition sensitivity test is separate ([section 9](#9-Warm-connections-do-not-improve-segmented-delivery)). **No cross-traffic.** Nothing else uses the links during a cell. The harness has a background-load mode that sends periodic messages of a set size on the same mesh after a lead-in. The post's figures do not use it. The post instead uses the residual-uplink sweep of its Figures 4 and 6. The link rate, 10 to 200 Mbps up and twice that down, represents the bandwidth available to the protocol after other traffic. Both simulators model this the same way, with a per-node link rate, so the sweep can be rerun under Shadow unchanged. **Under Shadow.** The Shadow node currently refuses the warm-up and background-load modes. These modes are described here but have not been measured under Shadow. ### Seed coverage | comparison | coverage | | --- | --- | | Figure 1 | Ten seeds on every side, in all three scenarios, on both transports. | | Figure 8 | Published: ten seeds at 250 and 500 nodes, three at 1 000 and above. Twin and both Shadow columns: ten at 250, 500 and 1 000. At 2 000 nodes, Shadow has four to six and the twin three to four (three to six overall). | | Figure 9 | Published: ten at every payload. Twin and Shadow: three (seeds 7 to 9), and ten at 1 MiB, where Figure 1 cells are reused. | | Linux | Three runs per seed; ten seeds (7 to 16) at 100 nodes under either queue, and one at 250; the whole message's Shadow twin stalled twice at seed 12 and is left out by the stall rule, so its FIFO rows rest on nine. Section 10 gives the paired statistic. | <a name="fig1"></a> ## 3. A-tuned agrees; three discrepancies need explanation  *Figure 1 under Shadow. Bars: receiver p99, ticks: p50, ±1 s.d. across seeds; four sides per arm.* Variant A reproduces. A-tuned is within 6 percent of its twin at both quantiles in all three scenarios and on both transports; coded A within 5 percent over QUIC, and within 13 percent over TCP in the datacenter-builder scenario. The twin is within 8 percent of the published cell on every segmented arm. The whole-message twin with 20 percent datacenter nodes is 19 percent slower than the published record (3.42 against 2.86 s), within the whole message's seed-to-seed spread. Every segmented arm meets the 3 s budget on every side and the whole message is the slowest arm on every side; at home it misses the budget at the median on both simulators. The order among the segmented arms is not the same on every side: the A family keeps its order, but C moves from the slowest segmented arm on the harness to among the fastest under Shadow. The comparison is at 500 nodes and 1 MiB. | observation | explanatory status | detail | | --- | --- | --- | | The whole message is about 34% faster under Shadow: −34% at the median at home, −36% with 20 percent datacenter nodes, −16% with a datacenter builder. | A fair-share intervention reverses the advantage over TCP, establishing queue order as the cause without reproducing the harness quantitatively. | [Section 4](#4-Queue-order-explains-the-baseline-gap-and-part-of-Cs-advantage) | | C and C + RS are about 21% and 23% faster under Shadow QUIC at home (−21% and −23%); over TCP, −37% and −41%. The pattern holds with 20 percent datacenter nodes; there is no QUIC difference with a datacenter builder. | The queue intervention explains about half the TCP advantage. More duplicates accompany an unexplained residual on both transports. | [Section 4](#4-Queue-order-explains-the-baseline-gap-and-part-of-Cs-advantage) | | B's QUIC tail is longer under Shadow; its medians agree within 7 percent and its TCP tails match. | Late receivers are served slowly. Capping the shared send buffer at 64 KB removes about 37% of the excess tail; the rest is unresolved. | [Section 5](#5-Slow-service-explains-part-of-Bs-QUIC-tail) | The numerical tables are retained in [Appendix D](#Appendix-D-detailed-scenario-comparisons). The scenario comparison distinguishes the likely causes: - The whole-message gap is a property of the home uplink. It is about 34% and 36% in the two scenarios where the builder is a home node and shrinks to 16 percent when the builder sits in a datacenter, where the first hop is not queue-bound. - C's gap follows the same scenario pattern exactly: about 18% to 21% with a home builder, nothing with a datacenter builder. That is the first reason to read the two as one mechanism before the fair-share cells confirm it. - B's tail does not follow it. The QUIC tail is long in all three scenarios and the TCP tail is normal in all three, so B's difference is not about where the queue is; it is about the transport path. <a name="queue"></a> ## 4. Queue order explains the baseline gap and part of C's advantage **Observed pattern.** On the published 500-node cell, ten seeds, the whole message completes under Shadow over QUIC at 3.22 s median and 4.04 s p99 against the twin's 4.91 s and 5.91 s. Shadow is −34% and −32% on every seed. Over TCP, ten seeds, the times are 3.63 s and 4.45 s. A-tuned and coded A are within 6% of their twins on the same cells ([section 3](#3-A-tuned-agrees-three-discrepancies-need-explanation)). ### Mechanism and queue models **The abstraction.** Both simulators model a node's network as an uplink and a downlink rate limit with a queue in front of each, and a fixed one-way latency per pair. There are no routers or shared paths. Everything a node sends passes through its uplink queue. The queue sets the order when several connections compete for it. **What each simulator puts there.** The harness's queue is FQ-CoDel (RFC 8290). Packets enter one sub-queue per connection, served round-robin by a byte quantum, with CoDel's delay control on each. Shadow uses a single FIFO, so packets leave in the order they were queued. Shadow also offers a round-robin queue that schedules sockets. go-libp2p multiplexes every QUIC connection over one UDP socket, so this queue has no effect on QUIC. Over TCP, every connection has its own socket, so the round-robin queue provides a fair share. **The mechanism.** When a node forwards a 1 MiB message to several peers at once, fair sharing advances every copy at the same rate. All copies finish together, which delays forwarding. With FIFO, the first connection to fill the queue uses the uplink first. The first peers receive the whole message early and start forwarding it while the others still wait. What matters is not the size of one message but how much a node has in its uplink queue for how many peers at once. A 32 KiB segment forwarded to eight mesh peers is gone in 40 ms, before the order can matter. Variant A is in that regime. Variant C runs 32 meshes, one per segment index, so a node forwards the whole payload's worth of segments to its peers at the same moment. That puts about 8 MiB in the queue at once, which is the whole message's load in 32 KiB pieces. ### Intervention: switch Shadow to fair sharing The same whole-message cell (seed 7) was also run under Shadow's round-robin queue. This queue provides a fair share between TCP sockets and has no effect on QUIC. | whole message, seed 7 | p50 (s) | p99 (s) | complete by 3 s (%) | | --- | ---: | ---: | ---: | | harness twin (fair share, QUIC) | 4.61 | 5.49 | 8 | | Shadow QUIC, FIFO queue | 3.10 | 3.87 | 40 | | Shadow QUIC, round-robin queue (no effect: one socket) | 3.10 | 3.87 | 40 | | Shadow TCP, FIFO queue | 3.66 | 4.43 | 18 | | Shadow TCP, round-robin queue (fair share) | 5.90 | 6.88 | 1 | Switching Shadow's queue from FIFO to a fair share changes the whole-message result over TCP from about 21% faster than the harness to slower than the harness. Over QUIC the switch changes nothing, to the byte, because all QUIC connections share one socket. The queue discipline alone accounts for the whole-message advantage under Shadow. **The same switch on variant C** (seed 7, home, TCP; the twin at this seed is a fast one): | receiver p50 (s) | harness twin | Shadow QUIC, FIFO | Shadow TCP, FIFO | Shadow TCP, round-robin | | --- | ---: | ---: | ---: | ---: | | C | 0.808 | 0.749 | 0.626 | 0.719 | | C + RS | 0.824 | 0.712 | 0.546 | 0.673 | ### Interpretation and unexplained residuals Fair sharing removes about half of C's advantage over TCP, giving latency close to Shadow QUIC. QUIC sees less of the FIFO benefit because quic-go paces every connection's packets at its congestion window over its round-trip time. Interleaving 70 paced connections into one socket already provides a partial fair share. Shadow's TCP writes a full window at once, and the FIFO serialises the flows. Linux TCP also paces sends; this suggests that the extra TCP benefit is largely an artifact of Shadow's unpaced TCP meeting FIFO, an inference from kernel behavior rather than a measurement. Queue order explains the TCP half of C's difference; pacing explains the transport dependence. The intervention leaves two quantitative differences unexplained: - C under Shadow stays 7 to 13 percent faster than its twin on both transports once the queue is equalised, C + RS 13 to 18 percent. - The whole message under the round-robin queue is about 28% slower than the twin. Shadow's round-robin is a fair share across sockets, not fq_codel, and the two transports' congestion control differs. The equalised cells are therefore not the twin's queue either. The remainder comes with more bytes. Shadow's C receives 1.01 to 1.03 MiB per node against the twin's 0.89, with nearly twice the duplicates. Shadow's C is faster while receiving more bytes. That is measured and not explained ([section 11](#11-What-this-does-not-cover-and-what-stays-open)). ### Scope of the queue comparison The simulators represent two queue configurations a node can meet. The harness's fair share corresponds to a bottleneck served by a flow scheduler such as fq_codel ([RFC 8290](https://www.rfc-editor.org/rfc/rfc8290.html)), for example a host egress qdisc or router queue configured to use it. Shadow's FIFO corresponds to a bottleneck served in arrival order, for example a FIFO in the access device in front of the host. The location alone does not specify the discipline, and fq_codel is not universal Linux behaviour. Access-link AQMs such as PIE control queue delay without themselves providing a fair share between flows ([RFC 8033](https://www.rfc-editor.org/rfc/rfc8033.html)). A single queue with PIE therefore need not reproduce the harness or fall between the two results. Read the two numbers as results for the stated configurations, with no measured basis for choosing which is more representative or treating them as bounds on deployed performance. Neither simulator models both queues at once. The queue choice affects the whole-message baseline and, by half as much, variant C. The phase shift's 13% (section 7) may be the same effect and was not tested. At 500 nodes and 1 MiB, the whole message's median is 3.2 s under FIFO and 4.9 s under fair sharing. Both exceed the 3 s budget. The gap to A-tuned (0.79 s under FIFO, 0.75 s under fair sharing) is 4.1× or 6.5×. The post uses fair sharing, so it reports the slower whole-message baseline and the larger improvement factors. Shadow gives the smaller improvement of the two, about 38% less, with the same conclusion. Variant A's results do not depend on the choice. With one mesh, a node never has enough queued at once for the order to matter. An arm that forwards to many peers at once, C or a whole message, does depend on it. A comparison between such arms and A should say which queue it assumes. <a name="btail"></a> ## 5. Slow service explains part of B's QUIC tail ### Observed pattern: late receivers continue receiving B's tail difference does not follow the queue-order pattern. Variant B's medians agree with the twin's within 7 percent in every scenario, but its p99 is longer under Shadow over QUIC. For coded B it is 1.40 of the twin at home, 1.37 and 1.23 in the datacenter scenarios. For B tuned it is 1.22, 1.18 and 1.15. Over TCP the tails match the twin (coded B 1.06 at home). This locates the difference on the path from quic-go to Shadow's UDP socket, specifically for B. **What the late nodes do.** Each Shadow node reports its completion time and its counters, so the tail can be read node by node. In the home cell at seed 7, coded B has 36 of 499 receivers finishing more than 0.4 s after the median over QUIC and one over TCP. About seven percent of receivers finish late; they receive at half the normal rate without stalling. - The late ones are not retrying. B reissues a request when its per-peer timeout lapses, and the cell's reissue count is 184 over QUIC against 169 over TCP. - They are not waiting either. Their received bytes grow in every 100 ms snapshot from the first byte at 0.2 s to completion at 1.37 s, at about half the rate of a node that finishes at the median. - They are not busy serving others (621 KiB sent against 824 for a median node). They are receivers being served slowly, by peers that answer many requests at once. **Why B and not A.** B sends each part once: a receiver claims an index from one peer and waits for it (1.2 copies of the payload per node, against A's 1.5). Slow service therefore delays B's completion directly, while A's second push of every segment and its pull discipline can compensate for a slow peer. B is the arm that exposes how fairly one uplink is shared among the connections asking for it. ### Intervention: reduce the shared send buffer Under Shadow the interface queue holds sockets, not packets, so a sender's backlog can only wait in the socket's own send buffer. Over QUIC all 70 of a node's connections share one UDP socket. quic-go asks for 7 MB of send buffer on it, which the kernel and Shadow double. At 50 Mbps that is over two seconds of uplink in one FIFO that every connection's packets enter in arrival order. Over TCP each connection has its own socket and its own, much smaller, buffer. The test is to cap the QUIC socket's send buffer and rerun the seed 7 cell against a control on the same binary. The lab's host wrapper exposes the knob. The node binary otherwise runs the report's code. | coded B, seed 7, home, Shadow QUIC | p50 (s) | p90 (s) | p99 (s) | late receivers (> p50 + 0.4 s) | | --- | ---: | ---: | ---: | ---: | | 7 MB send buffer (quic-go's request; the report's cells) | 0.843 | 1.186 | 1.549 | 37 | | 64 KB send buffer (about 10 ms of uplink) | 0.848 | 1.098 | 1.381 | 23 | | Shadow TCP, for reference | 0.726 | 0.952 | 1.095 | 1 | ### Interpretation: the buffer explains only part of the tail The cap leaves the median unchanged and removes about 37% of the excess tail. The p99 moves from 1.55 to 1.38 s against TCP's 1.10, and the late receivers from 37 to 23. The depth of the shared socket buffer is therefore part of the mechanism and not all of it. The remaining candidates and the changes needed to test them are in [section 11](#11-What-this-does-not-cover-and-what-stays-open). **Scope.** The post compares B on medians and bytes, and its median matches. The longer tail is nevertheless observed on a real transport stack under Shadow and is absent from simnet; the harness's smoother tail does not capture this behavior. <a name="tcp"></a> ## 6. Transport changes gaps and a tier ranking Every Shadow cell was run twice, once over QUIC and once over TCP with yamux, on the same topology and seed. The post's results are QUIC throughout, so this column is new rather than a reproduction. This tests whether the segmentation result depends on transport. It holds on both, although the size of the gaps and one tier ranking change. ### Observed pattern: transport effects depend on the arm The direction differs by arm. At 500 nodes and 1 MiB in the home scenario, medians across ten seeds: | arm | Shadow QUIC p50 (s) | Shadow TCP p50 (s) | TCP vs QUIC | | --- | ---: | ---: | ---: | | whole message | 3.220 | 3.629 | +13% | | A-tuned, 32 KiB | 0.793 | 0.785 | −1% | | coded A, 32 KiB | 0.699 | 0.738 | +6% | | B tuned, 32 KiB | 0.945 | 0.828 | −12% | | coded B, 32 KiB | 0.808 | 0.721 | −11% | | C, 32 KiB | 0.766 | 0.618 | −19% | | C + RS, 32 KiB | 0.715 | 0.553 | −23% | The whole message is 13% slower over TCP. Variant A is indifferent. B and C are *faster* over TCP, C by about 19% and C with the code by about 23%. The same ratios hold in the two datacenter scenarios ([Appendix D](#Appendix-D-detailed-scenario-comparisons)). Figure 9's payload-size comparison shows the opposite direction for its finer units. Its A-family tiers use 16 KiB units, and over TCP they are consistently slower than over QUIC. Across the payload-size range, tier 3 is slower by 3 to 24% and A-tuned at 16 KiB by 1 to 20%. The one 32 KiB arm in that figure, the compress-first code alone, sits near parity at 0.90 to 1.08. Figure 8 agrees for the two A-family arms it shares with Figure 1: A-tuned is within −1 to +5% of its twin at every network size and coded A within +1 to +6%. ### Interpretation: stream blocking is plausible but unmeasured The A-family pattern varies with unit size. The finer the unit, the more concurrent streams an arm opens, and the worse TCP with yamux does relative to QUIC. This is consistent with head-of-line blocking: yamux multiplexes its streams inside a single TCP connection, so a loss anywhere stalls every stream behind it, while QUIC's streams are independently retransmitted. It could also explain the one ordering change in [section 8](#8-The-baseline-gap-grows-with-payload-size). Tier 3 loses its first place to the compress-first reference over TCP, and tier 3 is the 16 KiB arm of that pair. The direction is right, but this is an inference from the ratios and the library's known behaviour. We have not instrumented head-of-line stalls to confirm it. At three seeds per point on Figure 9 the magnitudes are still coarse. ### Queue intervention: C's TCP advantage Unit size does not explain why some arms are faster over TCP. B and C use the same 32 KiB unit as A, which is indifferent. For C it is the uplink queue ([section 4](#4-Queue-order-explains-the-baseline-gap-and-part-of-Cs-advantage)). C keeps the whole payload's worth of forwards in the queue at once, and Shadow's FIFO lets the first through. TCP sees the full FIFO benefit, while quic-go's pacing turns the same FIFO into a partial fair share. Switching Shadow's TCP queue to a fair share puts C over TCP back where C over QUIC is. A similar mechanism could explain B's 11 to 12 percent advantage over TCP, but this is untested. A B server answers many requests at once from one uplink. B's QUIC tail, the third difference, is a different matter and has its own section ([section 5](#5-Slow-service-explains-part-of-Bs-QUIC-tail)). At one seed the phase shift also looked faster over TCP in Figure 8. At ten seeds it is not (TCP/QUIC 0.97 to 1.03 at every network size). **Scope.** The segmentation advantage holds on both transports, with the tier-order exception above. The whole message is the slowest arm by a wide margin on both. The transport changes the size of the gaps between arms, so a deployment on TCP would not see exactly these numbers. This matters most for arms with units smaller than 32 KiB. <a name="fig8"></a> ## 7. The baseline gap stays stable across network size At 2 000 nodes, seed coverage is lower (see the method table). A 2 000-node Shadow cell takes 65 to 107 GB, so they ran two at a time. The 4 000-node point was not run under Shadow.  *Figure 8 under Shadow: the four arms against network size at 1 MiB, one column per side.* Figure 8 tests whether the whole-message disagreement grows with network size. The gap remains stable: | nodes | 250 | 500 | 1 000 | 2 000 | | --- | ---: | ---: | ---: | ---: | | whole message, Shadow QUIC vs twin, p50 | −32% | −34% | −34% | −29% | | whole message, twin p50 (s) | 4.10 | 4.91 | 5.37 | 5.70 | | whole message, Shadow QUIC p50 (s) | 2.80 | 3.22 | 3.55 | 4.04 | The ratio is stable across an eightfold range of network size. Both simulators grow the whole message's latency at about the same rate. The twin rises from 4.1 to 5.7 s between 250 and 2 000 nodes, Shadow from 2.8 to 4.0 s, and their relative difference stays stable. This is the behaviour a queueing difference should have. It is a property of how one node empties its uplink, which does not change when the network around it grows. Had the gap widened with network size, the explanation in [section 4](#4-Queue-order-explains-the-baseline-gap-and-part-of-Cs-advantage) would be wrong, and some effect that compounds with hop count would be a better candidate. The A-family arms agree with their twins at every size. A-tuned runs +5 to +10% against its twin at the median, coded A −2 to +2%. Their absolute latencies are nearly flat in network size on both simulators, reproducing the post's point about segmentation. A-tuned's twin runs 0.708 s at 250 nodes and 0.774 s at 2 000, Shadow 0.770 s and 0.810 s. The phase shift is the only segmented arm in the figure that disagrees systematically. It runs 1.13 of its twin at 250 nodes, 1.12 at 500 and 1.13 at 1 000, then 1.03 at 2 000 (five seeds against three). The phase shift takes about 13% longer under Shadow than under the harness at 250 to 1 000 nodes, and 3% at 2 000. It lacks the pull discipline: every segment is pushed to the whole mesh under a budget that decays on IDONTWANT. At 3.3 copies per node against A-tuned's 1.5, it keeps more concurrent pushes in a node's uplink queue than A-tuned does. Whether that is the mechanism is not tested. The queue's order matters for exactly that kind of load in [section 4](#4-Queue-order-explains-the-baseline-gap-and-part-of-Cs-advantage). It does not change the figure's reading: the phase shift stays the slowest segmented arm on both simulators and the whole message stays above the budget on both. <a name="fig9"></a> ## 8. The baseline gap grows with payload size The three-seed medians are within a few percent; the ratios and line shapes are the stronger evidence at this coverage.  *Figure 9 under Shadow: the payload-size comparison's six lines, one column per side. Rows: receiver p50, p99, all bytes received per node in copies of the compressed payload, control bytes alone.* The payload-size comparison runs from 128 KiB to 2 MiB at 500 nodes. For every segmented arm the two simulators agree across the whole of it. Shadow's p50 stays between −17% and +8% of the twin's, with no trend against payload size. The lines keep their shape, their spacing and their order. The whole message does not agree. The gap grows with the payload: | payload (KiB) | 128 | 256 | 384 | 512 | 640 | 768 | 896 | 1024 | 1536 | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Shadow QUIC vs twin, p50 | −3% | −4% | −11% | −8% | −13% | −23% | −20% | −34% | −33% | The gap changes across three payload-size regimes. Up to 640 KiB the two simulators are within 13% of each other, and within 8% up to 256 KiB, which is the spread the segmented arms show. From 768 KiB the gap is 20 to 23%. At 1 MiB and above it is about 33%. That is what [section 4](#4-Queue-order-explains-the-baseline-gap-and-part-of-Cs-advantage) predicts. The queue discipline can only matter once one message occupies a node's uplink long enough for a second peer's traffic to arrive and wait behind it. A small message is gone before anything queues, so the fair-share and first-come queues behave identically. A 1 MiB message on a 50 Mbit uplink is not gone that quickly. The segmented arms never enter the regime at all, because no single segment is ever large enough to hold the link. The payload-size pattern is consistent with the queue explanation. It also locates the payload size at which the modelling difference starts to matter, somewhere between 640 and 768 KiB. That is about the point where one message's transmission time exceeds the spacing between forwards. The tier ordering from the post survives. At 1 MiB and at 2 MiB, the published, twin and Shadow QUIC columns all rank the arms identically: tier 3, then the compress-first code alone, then size-adaptive A and A-tuned, then tier 1, then the whole message. Shadow over TCP is the one exception, and it is a narrow one. Tier 3 and the compress-first reference swap the top two places (0.664 against 0.651 at 1 MiB, 0.960 against 0.928 at 2 MiB). [Section 6](#6-Transport-changes-gaps-and-a-tier-ranking) explains why that pair in particular. **The whole-message median at 2 MiB is censored.** All six cells (three seeds, both transports) ran to completion and are recorded `ok`. Only 50 to 81 of 499 receivers (QUIC) and 85 to 137 (TCP) finished inside the eight seconds of observation the cell allows after the publish. The median was therefore never reached during observation. The twin, which observes for longer, measures 14.7 s. Results at that scale need a longer observation window than this batch used. <a name="warm"></a> ## 9. Warm connections do not improve segmented delivery **Warm connections, measured in-process.** The warm-up mode was run as a condition on Figure 1's home cell, with ten seeds in the harness only. Every connection carries 320 KiB over a dummy stream before the publish (the bandwidth-delay product of 50 Mbps × 50 ms). Warm against cold, medians: | arm | p50, warm vs cold | p99, warm vs cold | bytes received, warm vs cold | | --- | ---: | ---: | ---: | | A-tuned | −4% | −4% | −6% | | B tuned | 0% | +1% | 0% | | C | +4% | +3% | −4% | | C + RS | 0% | −2% | −3% | | coded A | +42% | +30% | −17% | | coded B | +58% | +46% | −7% | | whole message | nine of ten seeds do not complete inside the cell's window | | | **Result.** Warming does not help any segmented arm. It increases the two coded arms' median latency by about 42% and 58%. **Interpretation.** The direction is consistent with the effect of a cold window on a sender. Slow start paces the first round trips, so a node that has a coded group's worth of pushes to make (twice the bytes) cannot burst them into its uplink at once. A warm window lets it. The whole message's failure to complete warm is the same effect at full size, or the warm-up's own 22 MB per node still draining at the publish. The cell cannot tell the two apart, so it is a result about the condition, not about the arm. This was not run under Shadow. <a name="realstack"></a> ## 10. The Linux check: kernel agreement and shared-host CPU cost At 100 nodes, the segmented QUIC arms complete within the simulators' own spread. The whole-message QUIC result also includes host saturation. This section separates those findings and establishes the usable scale before interpreting larger cells. Both simulators replace the operating system's network stack and charge nothing for CPU time. The Linux cells test those omissions using the same node binary, one process per network namespace, virtual links from the same topology export, and a wall-clock publish. Rates, delays and the graph match the simulated cells. The uplink queue is either FIFO or fq_codel. [Appendix C](#Appendix-C-how-the-Linux-namespace-cells-are-built) gives the namespace construction, startup schedule, ARP handling and reset procedures. ### Comparison and criterion for host interference The host's CPU is shared by every node: the run records the machine's busy fraction from the publish to the end of the hold, and, because a delivery is a burst shorter than the hold, samples it every 0.25 s through that window and records the peak and the seconds at or above 95 %. A real-stack latency counts as a network result only when that saturated time is zero. **Statistic.** Each cell is run three times on the wall clock per seed, at ten seeds (7 to 16), and each seed has its Shadow twin at the same topology draw. Tables give, on each side, the median across seeds of the per-seed medians, then the median across seeds of the per-seed difference of the real stack from Shadow, in percent, with the seeds' range in brackets. Shadow's queue is a FIFO, so the FIFO real-stack cells are compared with Shadow; the harness's queue is the fair-queueing one and the harness runs QUIC only, so the fq_codel real-stack cells are compared with the harness twin. ### Kernel comparison at the usable scale **Results at 100 nodes** (1 MiB, home links, degree 70; receivers' p50 / p99 completion time in ms, medians across seeds; the differences are the median across seeds of the per-seed difference, with the seeds' range in brackets; bytes per node likewise; the last column is the host's CPU load, mean over all 40 threads from the publish to the end of the 10 s hold). The FIFO cells against Shadow: | arm | transport | Shadow (ms) | real stack (ms) | real stack vs Shadow, p50 | p99 | bytes/node vs Shadow | host CPU (%) | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | | whole message | QUIC | 2202 / 2946 | 3609 / 4779 | +68% [+26 to +79%] | +58% [+31 to +80%] | +34% | 35 | | A-tuned 32k | QUIC | 762 / 1000 | 783 / 1070 | +4% [0 to +13%] | +9% [−15 to +20%] | −1% | 8 | | C 32k | QUIC | 715 / 926 | 854 / 1062 | +18% [+13 to +27%] | +15% [+1 to +20%] | +1% | 14 | | whole message | TCP | 2720 / 3736 | 3319 / 4369 | +23% [−7 to +58%] | +18% [+1 to +55%] | +16% | 23 | | A-tuned 32k | TCP | 748 / 967 | 966 / 1312 | +27% [−12 to +40%] | +37% [+12 to +53%] | +12% | 7 | | C 32k | TCP | 558 / 727 | 926 / 1175 | +66% [+37 to +80%] | +61% [+43 to +89%] | +24% | 14 | The fq_codel cells against the harness, which runs QUIC only, the same statistic; the TCP rows have no harness twin: | arm | transport | harness (ms) | real stack (ms) | real stack vs harness, p50 | p99 | | --- | --- | ---: | ---: | ---: | ---: | | whole message | QUIC | 3492 / 4759 | 4218 / 5529 | +25% [−13 to +53%] | +18% [−12 to +56%] | | A-tuned 32k | QUIC | 681 / 942 | 781 / 1090 | +14% [−2 to +29%] | +17% [−3 to +27%] | | C 32k | QUIC | 884 / 1100 | 890 / 1112 | −1% [−14 to +8%] | −1% [−14 to +6%] | | whole message | TCP | – | 4400 / 5598 | – | – | | A-tuned 32k | TCP | – | 932 / 1332 | – | – | | C 32k | TCP | – | 966 / 1224 | – | – | The segmented arms over QUIC run 4 to 18 % slower on the real stack than under Shadow (A-tuned +4 / +9 %, C +18 / +15 %), with the same bytes per node (within 1 %) and the same meshes (mesh sizes checked at seeds 7, 11 and 13: 6/8/11 for A-tuned and the whole message on both sides, 51–70 against 56–69 for C), and the two sides move together from seed to seed (A-tuned's p50 spans 576 to 822 ms under Shadow across the ten seeds and 640 to 856 ms on the real stack, with the same two seeds lowest on both sides). Against the harness, on the fair-queueing side, A-tuned runs 14 / 17 % slower and C is level (−1 / −1 %). The two simulators differ from each other by 7 to 56 % on the same cells (harness against Shadow −7 % / −7 % for A-tuned, +26 % / +19 % for C, +56 % / +50 % for the whole message), so the difference between the real stack and either simulator is no larger than the one between the two simulators. The TCP rows compare the Linux TCP stack with the one Shadow implements, since Shadow does not run the kernel's; the 27 % and 66 % differences belong to that comparison and are a note on Shadow's TCP fidelity rather than a protocol result. With the fair-queueing uplink the segmented arms move by −3 to +6 % and the whole message slows by 16 to 29 % (per-seed medians against the same seed's FIFO cell), the direction section 4 predicts for an arm that keeps many copies in the queue at once; the whole message is slower than both simulators on both queues, with host interference assessed separately below. ### Shared-host CPU behavior The whole-message rows are not a clean comparison over QUIC. Spreading one 1 MiB message through 100 nodes keeps the machine at 19–36 % busy averaged over the hold, two to four times the segmented arms' 6–14 %. Sampled at 0.25 s through a QUIC delivery (p50 3.0 s, last 4.2 s), the machine climbs from 14 % in the second after the publish to 67 %, then sits at 97–98 % for about 1.25 s and at 77 % for another second before falling back; the +69 % includes host saturation. Over TCP the same delivery peaks at 82–89 % for about a second and never reaches 90 %, and the difference is +23 % with a wide seed-to-seed range (−7 to +58 %): on this host the whole message costs less CPU over the kernel's TCP than over quic-go's QUIC, and its TCP row is the one whole-message comparison without host saturation. ### Evidence for the host's scale limit The same cells at 250 and 500 nodes, segmented QUIC arms, FIFO queue, p50 in seconds, with the 0.25 s host samples through the delivery window: | nodes | A-tuned p50 (s) | vs Shadow | C p50 (s) | vs Shadow | host during the delivery | | --- | ---: | ---: | ---: | ---: | --- | | 100 | 0.78 | +4% | 0.85 | +18% | 0.5 s at 80–95 % of a 0.8 s window | | 250 | 1.84 | ×2.5 | 2.64 | ×3.6 | 1.25–1.5 s at 99 % of a 1.8–2.6 s window; the sampler itself starved | | 500 | 6.3 | ×8 | – | – | 6.5 s at 99–100 %, the entire window | Shadow and the harness both give the same latency at 250 and 500 as at 100 (a mesh of degree 70 gains few hops), and the meshes and the dial graphs are the same on all three backends. Past 100 nodes the real-stack latency is limited by host throughput: at 250 the 40 threads are at 99 % for most of the delivery, so completely that the sampling process gets no CPU for that stretch, and at 500 for all of it. A control at 100 nodes with 30 busy loops occupying the machine raises A-tuned's p50 from 0.78 to 0.93–1.00 s and its p99 from 1.02–1.18 to 1.33–1.41 s, +20 to +30% from CPU contention alone; the multipliers at 250 and 500 are what that contention becomes when the delivery's demand exceeds the machine for the whole window. Duplicates received rise with the latency in every one of these cases (1.7 k at 100 idle, 2.1–2.4 k loaded, 12 k at 250, 45–50 k at 500, against 1.7 k and 4.4 k under Shadow), correlated with the delay rather than shown to cause it. The whole message was run at 250 (p50 5.6 s over QUIC, 4.0 s over TCP, the machine at 55–79 % averaged over the hold with a delivery of 8 s) and not at 500. ### Scope of the result For segmented QUIC at 100 nodes, the real stack is within the simulators' own spread: 4 to 18 % above Shadow with the same bytes and meshes, and −1 to +17 % against the harness. The simulator comparisons in sections 3 to 9 therefore use transport models that agree with the kernel to that degree. The whole message exposes CPU work the simulators omit. Forwarding one large message to many peers at once costs several times as much CPU as forwarding segments as they arrive: two to four times on one machine at 100 nodes, and at 250 nodes the host is busy through most of an 8 s delivery against about one second of saturation for the segmented arms. All nodes share one host and kernel, so these observations do not establish per-node cost on separate machines. <a name="open"></a> ## 11. What this does not cover, and what stays open The bandwidth model and the full client are outside all three backends. CPU cost is outside both simulators; section 10 measures it on one machine, with all nodes sharing one kernel, and says nothing about a node on its own host. The post's statements about the bandwidth model and the client are unchanged by this report. Points above 2 000 nodes were not run under Shadow, and the warm-up and background-load modes were not run under Shadow or on real stacks. The remaining questions and the experiments that would resolve them are: 1. **C's residual.** Once the queue's order is equalised, C under Shadow is still 7 to 13 percent faster than its twin on both transports. It receives about 15 percent more bytes with twice the duplicates. Lower latency with more bytes suggests a difference in how the stacks respond to a persistently full queue. The possible difference is simnet's CoDel drops and quic-go's response to them, against Shadow's unbounded socket buffers. This is untested; comparing drops and transport responses under matched queue settings would test it. 2. **B's remaining tail** (section 5). Two candidates remain untested. quic-go's initial flow-control windows (512 KiB per stream, 768 KiB per connection) bound how much a requester can have in flight from one server. A server answering two hundred requests over seventy connections may be constrained differently from TCP with its per-socket buffers. B's adaptive request timeout uses observed response times; a deep buffer may make it wait longer for a slow server when another peer could answer. Varying the windows and timeout would separate these effects. The initial congestion window is another candidate, but the Shadow node refuses that knob by design (it was written for the warm-up conditions); testing it needs a node change and a rebuild against the quic-go fork. 3. **The phase shift's 13%** in Figure 8: the same shape as C's difference and plausibly the same mechanism. This is untested; repeating the queue intervention on this arm would test the explanation. 4. **Figure 9 at three seeds**: the tier ordering and the whole-message threshold held from one seed to three; the TCP penalties of the 16 KiB arms (3 to 24%) would need ten seeds to be precise estimates rather than qualitative patterns. 5. **The warm condition** (section 9): ten seeds in-process say warming does not help and slows the coded arms by about 42% and 58%. The whole message does not complete warm. Whether that is a transport effect or the warm-up's own load is undecided. Separating the warm-up traffic from the measured publish would test this. None of it was run under Shadow. 6. **Real stacks beyond one machine** (section 10). The 100-node agreement is for the segmented arms over QUIC, on one host and one kernel. The 4 to 18 % over QUIC, the TCP rows, and the whole message's own numbers all include either Shadow's TCP or the host; separating them needs either several hosts or a machine with headroom during the delivery. The coded arms and variant B were not run on real stacks. The network model also leaves access-link queue configurations open. Measurements on access links would establish which queues nodes encounter. A delay-controlling single queue would test the unmeasured case discussed in [section 4](#4-Queue-order-explains-the-baseline-gap-and-part-of-Cs-advantage). Across the tested configurations, segmentation remains faster, while queueing and transport behavior explain or delimit the differences between experimental backends. The Linux evidence covers a limited set of variants on one shared host; broader deployment behavior remains open. ## Appendix A: the arms An arm is a policy at a unit, and every record carries its wire form (message count, unit, id width). A pair is compared only when both sides have the same wire form. Every cell shares `SEGMENT_STRICT=1 SEGMENT_DET_RAND=1`, degree 70, the geographic latency model, 50 Mbps up and 100 down at home nodes, a 3 s deadline, and the scenario knobs (`dc20`: 20 percent of nodes on datacenter links; `dcb20`: that plus a 1 Gbps builder). `FINAL` is the post's pull discipline: `SEGMENT_IWANT_DISCIPLINE_MS=200 SEGMENT_OFFER_TABLE=1 SEGMENT_IHAVE_PARK=1 SEGMENT_IHAVE_PARK_K=1 SEGMENT_IHAVE_PARK_PROMISE_MS=400 SEGMENT_IHAVE_PARK_TTL_MS=30000`. `K` is the payload in 32 KiB segments; `AR` is the size-adaptive push width (4 up to 256 KiB, 3 up to 1 MiB, 2 above). The table is `testing/shadowstudy/arms.sh` in the research tree, which both substrates source. | report name | published name | figure | policy and knobs | | --- | --- | --- | --- | | whole message | `wholend` | 1, 8, 9 | stock gossipsub, one message (`SEGMENT_ARMS=whole`) | | A-tuned 32k | `atuned` | 1, 8 | A's phase forwarding at r = 2, 32 KiB segments, `FINAL` | | A-tuned 16k | `am_k64` | 9 | the same at 16 KiB | | coded A 32k | `acoded` | 1, 8 | A-tuned 32k + Reed-Solomon parity K, stop-pull, structured ids | | the phase shift 32k | `lad_phase` | 8 | phase forwarding r = 2 at 32 KiB, structured ids, no discipline (Figure 2's rung) | | tier 1: segmentation + batch 16k | `lad_batch16` | 9 | plain segmented arm, full-mesh push, 16 KiB | | tier 2: size-adaptive A 16k | `aadaptds` | 9 | phase forwarding at r = AR, 16 KiB, structured ids, `FINAL` | | tier 3: size-adaptive A + compress-first code 16k | `aadaptcf` | 9 | compress first, P/16 KiB pieces + as many parity, r = AR, stop-pull; `FINAL` from 512 KiB | | tier 3 with the discipline at every size | `aadaptcfd` | 9 (below 512 KiB) | tier 3 with `FINAL` at every size | | the compress-first code alone 32k | `acodedcf` | 9 | A-tuned 32k with compress-first coding, parity K, stop-pull | | B tuned 32k | `bfixed` | 1 | variant B (the partial-message extension): phase policy, replication 2, 4 claims per peer, strike cap 1, adaptive timeout, compressed parts, 32 KiB | | coded B 32k | `bcodedfc1` | 1 | B tuned + parity K, one part per frame | | C 32k | `c32` | 1, 8 (C cells at 250 and 1 000) | variant C: one topic per segment index, phase forwarding r = 2, 32 KiB, `FINAL` | | C + RS 32k | `c32rs` | 1, 8 | C + parity K | B's records have no control-byte count because the extension carries its coordination inside the data RPCs. The tables show "–" there. ## Appendix B: the code Everything the report ran is pinned by one tag of the studies repository, [`eth-networking-studies`](https://github.com/cskiraly/eth-networking-studies) at `shadow-crosscheck`: | directory | what | commit | | --- | --- | --- | | `prysm/` | the research tree, without its notes: the node `TestShadowNode` and the topology export in `beacon-chain/p2p/segmentintegrationtest`, the arm catalogue and the cell scripts in `testing/shadowstudy`, the two library forks vendored under `third_party/` | `c4d72f7959`: the tree of `af6e41eb7d`, which sections 1 to 9 ran, plus the node's per-cell dial deadline and bound on in-flight handshakes, which section 10 ran (`0871156c89`) | | `eth-networking-lab/` | the lab: the Shadow tooling and the namespace backend in `shadowsim/` | `f108f3b`; the Shadow cells ran `0a29198` | | `shadow/` | Shadow, release 3.3.0 | `5a05740ba` | | `shadow-patches/` | the three small patches Shadow needs for a Go node | | Check out the tag, `git submodule update --init`, `./build.sh`, and follow its README: one cell on each of the three backends, and the comparison, are a few lines each. ## Appendix C: how the Linux namespace cells are built ### Network construction and run schedule Each node is the same binary as the Shadow node, one process in its own Linux network namespace. A virtual Ethernet pair joins it to a bridge in a fabric namespace. The node's upload rate is a token-bucket class (HTB) on its own interface with either a FIFO (`pfifo`, Shadow's queue) or a fair-queueing (`fq_codel`, the harness's) leaf; the download rate is the same construction on an intermediate device that ingress traffic is redirected to; the one-way delay of every pair is a separate `netem` on the receiver's bridge port, selected by source address, installed for the pairs that the dial graph connects. Rates, delays and the graph come from the same topology export the Shadow cells use, so the three backends share the topology, the links and the latencies. The node dials its edges at 1 s, subscribes at 10 s, the publisher sends at 30 s of the wall clock, and the cell holds 10 s after the publish; at 250 nodes the dial phase needs longer (connection setup on a real host runs at a few hundred endpoints per second), so the subscribe and publish instants move to 70 s and 90 s. The network is built once per cell and reset between runs (sockets drained, TCP metrics not saved, counters cleared), which keeps a 250-node cell to about seven minutes for three runs. Nothing needs root: the whole fabric lives in an unprivileged user namespace. ### Neighbour entries The kernel keeps one neighbour (ARP) table for every namespace on the machine, capped by a sysctl the namespaces cannot raise, and a cell needs one entry per connection endpoint, nodes × degree; past the cap the kernel refuses the entry and a UDP sender's packets never leave, counted as send-buffer errors while every queue reports zero drops. The plan therefore assigns each interface an address derived from its index and preloads each node's dialled peers as permanent entries, which are exempt. ### Pairing records The comparison tool pairs cells by queue, and the harness records carry none, so the harness is named queue-agnostic (`compare.py --queue-agnostic harness`) and stands beside both queues. ## Appendix D: detailed scenario comparisons These are the Figure 1 values discussed in [section 3](#3-A-tuned-agrees-three-discrepancies-need-explanation). The home scenario, QUIC (p50 / p99 in seconds, medians across ten seeds): | arm | published (s) | twin (s) | Shadow QUIC (s) | Shadow TCP (s) | Shadow QUIC vs twin, p50 / p99 | | --- | ---: | ---: | ---: | ---: | ---: | | whole message | 4.98 / 5.91 | 4.91 / 5.91 | 3.22 / 4.04 | 3.63 / 4.45 | −34% / −32% | | A-tuned, 32 KiB | 0.746 / 0.988 | 0.750 / 1.000 | 0.793 / 1.041 | 0.785 / 1.044 | +6% / +4% | | coded A, 32 KiB | 0.704 / 0.869 | 0.705 / 0.899 | 0.699 / 0.910 | 0.738 / 0.950 | −1% / +1% | | B tuned, 32 KiB | 0.897 / 1.177 | 0.938 / 1.197 | 0.945 / 1.459 | 0.828 / 1.306 | +1% / +22% | | coded B, 32 KiB | 0.742 / 1.091 | 0.754 / 1.097 | 0.808 / 1.532 | 0.721 / 1.161 | +7% / +40% | | C, 32 KiB | 0.956 / 1.169 | 0.974 / 1.198 | 0.766 / 0.973 | 0.618 / 0.778 | −21% / −19% | | C + RS, 32 KiB | 0.978 / 1.108 | 0.931 / 1.071 | 0.715 / 0.860 | 0.553 / 0.652 | −23% / −20% | The two datacenter scenarios, as differences of medians across ten seeds (Shadow QUIC against the twin, then Shadow TCP against Shadow QUIC; p50 / p99): | arm | home: QUIC vs twin | TCP vs QUIC | 20% datacenter: QUIC vs twin | TCP vs QUIC | datacenter builder: QUIC vs twin | TCP vs QUIC | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | whole message | −34% / −32% | +13% / +10% | −36% / −35% | +21% / +23% | −16% / −18% | +14% / +11% | | A-tuned | +6% / +4% | −1% / +0% | +4% / +6% | −1% / −2% | +3% / +5% | +0% / −3% | | coded A | −1% / +1% | +6% / +4% | −3% / +1% | +5% / +3% | +2% / +5% | +11% / +8% | | B tuned | +1% / +22% | −12% / −10% | +6% / +18% | −14% / −14% | +5% / +15% | −14% / −16% | | coded B | +7% / +40% | −11% / −24% | +7% / +37% | −9% / −24% | −1% / +23% | −6% / −24% | | C | −21% / −19% | −19% / −20% | −18% / −14% | −22% / −23% | +0% / +0% | −18% / −22% | | C + RS | −23% / −20% | −23% / −24% | −22% / −16% | −23% / −25% | −2% / −3% | −19% / −23% |