BMWG F. Calabria Internet-Draft Cisco Intended status: Informational C. Pignataro Expires: 13 February 2027 Blue Fern Consulting Q. Wu G. Fioccola Huawei S. Reddy Apple 12 August 2026 Benchmarking Methodology for AI Training Network Fabrics draft-calabria-bmwg-ai-fabric-training-bench-04 Abstract This document defines benchmarking terminology, methodologies, and Key Performance Indicators (KPIs) for evaluating Ethernet-based AI training network fabrics. As large-scale distributed Artificial Intelligence / Machine Learning (AI/ML) training clusters grow to tens of thousands of accelerators (GPUs or generic accelerator processing units (XPUs)), the backend network fabric determines Job Completion Time (JCT), training throughput, and accelerator utilization. This document establishes vendor-independent, reproducible test procedures for benchmarking fabric-level performance under realistic AI training workloads. The tests cover Remote Direct Memory Access (RDMA) over Converged Ethernet version 2 (RoCEv2) transport, the Ultra Ethernet Transport (UET) protocol defined by the Ultra Ethernet Consortium (UEC) Specification 1.0, congestion management (Priority Flow Control (PFC), Explicit Congestion Notification (ECN), Data Center Quantized Congestion Notification (DCQCN), Credit-Based Flow Control (CBFC)), load balancing strategies (Equal-Cost Multi-Path (ECMP), Dynamic Load Balancing (DLB), packet spraying), collective communication patterns (AllReduce, AllToAll, AllGather), and scale/ soak testing. The methodology enables direct, reproducible comparison across switch ASICs, NIC transport stacks (RoCEv2 and UET), and fabric architectures (2-tier Clos, 3-tier Clos, and rail-optimized). About This Document This note is to be removed before publishing as an RFC. Calabria, et al. Expires 13 February 2027 [Page 1] Internet-Draft AI Training Fabric Benchmarking August 2026 The latest revision of this draft can be found at https://fcalabri.github.io/bmwg-ai-fabric-training-bench/draft- calabria-bmwg-ai-fabric-training-bench.html. Status information for this document may be found at https://datatracker.ietf.org/doc/draft- calabria-bmwg-ai-fabric-training-bench/. Discussion of this document takes place on the BMWG Working Group mailing list (mailto:bmwg@ietf.org), which is archived at https://mailarchive.ietf.org/arch/browse/bmwg/. Subscribe at https://www.ietf.org/mailman/listinfo/bmwg/. Source for this draft and an issue tracker can be found at https://github.com/fcalabri/bmwg-ai-fabric-training-bench. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 13 February 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 Calabria, et al. Expires 13 February 2027 [Page 2] Internet-Draft AI Training Fabric Benchmarking August 2026 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 5 1.2. Scope and Applicability . . . . . . . . . . . . . . . . . 5 1.3. Relationship to Existing and Companion Work . . . . . . . 6 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1. Acronyms . . . . . . . . . . . . . . . . . . . . . . . . 8 3. Test Topology and Architecture . . . . . . . . . . . . . . . 8 3.1. Reference Fabric Topologies . . . . . . . . . . . . . . . 8 3.1.1. Topology A: 2-Tier Clos (Leaf-Spine) . . . . . . . . 8 3.1.2. Topology B: 3-Tier Clos (Leaf-Spine-Superspine) . . . 9 3.1.3. Topology C: Rail-Optimized . . . . . . . . . . . . . 9 3.2. Result Comparability and Normalization . . . . . . . . . 10 3.2.1. Normalization Basis . . . . . . . . . . . . . . . . . 10 3.2.2. Aggregate Switching Capacity Is Not a Normalization Factor . . . . . . . . . . . . . . . . . . . . . . . 11 3.2.3. Comparability Set . . . . . . . . . . . . . . . . . . 12 3.2.4. Comparing Fabrics of the Same Topology Class . . . . 13 3.2.5. Fabric-Visible Data Volume . . . . . . . . . . . . . 13 3.3. Device Under Test (DUT) Identification . . . . . . . . . 14 3.4. Traffic Generator Requirements . . . . . . . . . . . . . 15 3.4.1. Mandatory Functional Capabilities . . . . . . . . . . 15 3.4.2. Minimum Measurement Accuracy Requirements . . . . . . 15 3.4.3. Acceptable Implementations . . . . . . . . . . . . . 16 4. KPI Framework and Metrics Taxonomy . . . . . . . . . . . . . 17 4.1. Primary KPIs . . . . . . . . . . . . . . . . . . . . . . 17 4.2. Secondary KPIs . . . . . . . . . . . . . . . . . . . . . 18 4.3. Fabric Health Indicators . . . . . . . . . . . . . . . . 19 5. Test Category 1: RDMA Transport Benchmarks . . . . . . . . . 19 5.1. Baseline Throughput . . . . . . . . . . . . . . . . . . . 19 5.2. Latency Characterization . . . . . . . . . . . . . . . . 20 5.3. Back-to-Back Burst Absorption . . . . . . . . . . . . . . 20 6. Test Category 2: UEC Transport Protocol Benchmarks . . . . . 21 6.1. UET Throughput by Transport Service . . . . . . . . . . . 21 6.2. UET Latency Characterization . . . . . . . . . . . . . . 22 6.3. Packet Spray Efficacy Under UET RUD . . . . . . . . . . . 23 6.4. UET Congestion Control Benchmarks . . . . . . . . . . . . 24 6.5. Link-Layer and Network-Layer Enhancement Benchmarks . . . 24 6.6. UET Collective Communication Performance . . . . . . . . 25 6.7. UET PDC Scalability and Connection Setup Rate . . . . . . 26 7. Test Category 3: Congestion Management . . . . . . . . . . . 26 7.1. ECN Marking Accuracy and Threshold . . . . . . . . . . . 26 7.2. PFC Behavior Under Incast . . . . . . . . . . . . . . . . 26 7.3. DCQCN Convergence Time . . . . . . . . . . . . . . . . . 27 7.4. PFC Storm and Deadlock Resilience . . . . . . . . . . . . 27 8. Test Category 4: Load Balancing Efficacy . . . . . . . . . . 27 8.1. ECMP Entropy and Polarization . . . . . . . . . . . . . . 27 8.2. Dynamic Load Balancing (Flowlet) . . . . . . . . . . . . 27 8.3. Packet Spraying . . . . . . . . . . . . . . . . . . . . . 28 8.4. Jain's Fairness Index Measurement . . . . . . . . . . . . 28 Calabria, et al. Expires 13 February 2027 [Page 3] Internet-Draft AI Training Fabric Benchmarking August 2026 9. Test Category 5: Collective Communication Benchmarks . . . . 28 9.1. AllReduce Benchmark . . . . . . . . . . . . . . . . . . . 29 9.2. AllToAll Benchmark . . . . . . . . . . . . . . . . . . . 29 9.3. AllGather Benchmark . . . . . . . . . . . . . . . . . . . 30 9.4. Collective Communication Library Bus Bandwidth Summary . 31 10. Test Category 6: Job Completion Time (JCT) Benchmarks . . . . 31 10.1. Synthetic JCT Under Controlled Conditions . . . . . . . 32 10.2. MLPerf-Aligned JCT . . . . . . . . . . . . . . . . . . . 34 10.3. Multi-Tenant JCT Interference . . . . . . . . . . . . . 34 11. Test Category 7: Scale and Convergence . . . . . . . . . . . 34 11.1. Fabric Scale Limits . . . . . . . . . . . . . . . . . . 34 11.2. Link Failure Convergence . . . . . . . . . . . . . . . . 35 11.3. Zero-Impact Failover Measurement . . . . . . . . . . . . 35 12. Test Category 8: Soak and Stability . . . . . . . . . . . . . 35 12.1. 24-Hour Sustained Load . . . . . . . . . . . . . . . . . 35 12.2. Resource Leak Detection . . . . . . . . . . . . . . . . 36 13. Reporting Format . . . . . . . . . . . . . . . . . . . . . . 36 14. Security Considerations . . . . . . . . . . . . . . . . . . . 37 15. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 39 16. References . . . . . . . . . . . . . . . . . . . . . . . . . 39 16.1. Normative References . . . . . . . . . . . . . . . . . . 39 16.2. Informative References . . . . . . . . . . . . . . . . . 40 Appendix A. KPI-to-Test Mapping Summary . . . . . . . . . . . . 41 Appendix B. Indicative Reference Values (Non-Normative) . . . . 43 Appendix C. ASIC Feature Categories (Informational) . . . . . . 44 Appendix D. RoCEv2 Test Frame Format . . . . . . . . . . . . . . 46 Appendix E. UET (Ultra Ethernet Transport) Frame Format . . . . 47 E.1. Key Differences from RoCEv2 . . . . . . . . . . . . . . . 48 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 49 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 49 1. Introduction Distributed AI/ML training workloads impose traffic requirements that standard data center fabrics were not designed to meet. Traditional data center traffic varies in flow size and protocol mix. AI training generates synchronized, bandwidth-intensive east-west traffic dominated by collective communication operations: AllReduce, AllToAll, and AllGather. These workloads require RDMA transport with negligible application-visible loss, bounded tail latency, uniform load distribution across all fabric paths, and the ability to absorb coordinated micro-bursts from thousands of accelerators simultaneously. RoCEv2 deployments typically meet the loss requirement by operating the fabric lossless (PFC/ECN); UET is designed to tolerate wire-level loss and recover via retransmission and packet trimming (see the Zero Packet Loss definition in [TERMINOLOGY]). Calabria, et al. Expires 13 February 2027 [Page 4] Internet-Draft AI Training Fabric Benchmarking August 2026 Existing BMWG methodologies do not address AI training fabrics. [RFC2544] defines benchmarking for general network interconnect devices but does not account for RDMA transport semantics, collective communication patterns, or the congestion behavior specific to GPU- to-GPU traffic. [RFC8238] and [RFC8239] establish data center benchmarking terminology and methodology but predate large-scale RoCEv2 deployment and do not address Priority Flow Control (PFC) interactions, DCQCN congestion control convergence [DCQCN-PAPER], or the impact of load balancing strategies on Job Completion Time (JCT). Industry experience deploying RoCEv2 at scale [META-ROCE] shows the need for a standardized benchmarking methodology. The Ethernet Virtual Private Network (EVPN) benchmarking methodology [EVPN-BENCH] provides a structural template for service-oriented benchmarking but is scoped to L2VPN services rather than RDMA fabrics. This document defines a benchmarking methodology for AI training network fabrics. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 1.2. Scope and Applicability This document applies to Ethernet-based AI training backend network fabrics employing RoCEv2 and/or UEC Ultra Ethernet Transport (UET) protocols. The scope includes leaf-spine (2-tier Clos) and leaf- spine-superspine (3-tier Clos) topologies. InfiniBand fabrics are explicitly *out of scope*, though many KPIs defined herein may be adapted for IB benchmarking by future documents. The DUT is the network fabric itself (the collection of switches and interconnecting links), not individual accelerators or host NICs; host-side configuration is documented in the test report as it materially affects results. The DUT boundary for all measurements in this document is the NIC-to- NIC Ethernet fabric segment. Intra-node communication (proprietary accelerator interconnects, e.g., NVLink, Infinity Fabric/xGMI, or PCIe) and individual GPU/accelerator performance are explicitly out of scope. Collective operation measurements (AllReduce, AllGather, AllToAll) are measured at the Ethernet fabric boundary; intra-node Calabria, et al. Expires 13 February 2027 [Page 5] Internet-Draft AI Training Fabric Benchmarking August 2026 accelerator-interconnect contributions are reported separately when characterizing wide Expert Parallelism (wide-EP) or multi-node configurations. The reporting requirements that make this separation auditable, and the normalization basis that keeps it intact, are specified in Section 3.2. The methodology is designed for controlled laboratory environments per the BMWG charter; it is NOT intended for production network measurement. 1.3. Relationship to Existing and Companion Work +===================+=============================================+ | Document | Relationship | +===================+=============================================+ | [RFC1242] | Base terminology for network benchmarking; | | | terms reused herein | +-------------------+---------------------------------------------+ | [RFC2544] | Base methodology; throughput/latency/loss | | | tests adapted for RDMA | +-------------------+---------------------------------------------+ | [RFC8238] | Data center terminology; buffer, | | | congestion, and microburst terms extended | +-------------------+---------------------------------------------+ | [RFC8239] | Data center methodology; line-rate and | | | buffer tests adapted for RoCEv2 | +-------------------+---------------------------------------------+ | [RFC9004] | Back-to-back frame updates; burst | | | absorption methodology referenced | +-------------------+---------------------------------------------+ | [LLM-BENCH] | Complementary document benchmarking the | | | inference serving stack. Treats the | | | network as opaque SUT; this document | | | benchmarks the fabric itself. The two | | | documents MAY be used together but MUST NOT | | | be combined in a single benchmarking report | | | without explicit section demarcation. | +-------------------+---------------------------------------------+ | [TERMINOLOGY] | Companion document; normative source for | | | the terminology used throughout this | | | document | +-------------------+---------------------------------------------+ | [INFERENCE-BENCH] | Companion fabric-level benchmarking | | | methodology addressing AI inference serving | | | workloads | +-------------------+---------------------------------------------+ | [UEC-1.0] | UET protocol specification; transport | | | services, congestion control, and link- | Calabria, et al. Expires 13 February 2027 [Page 6] Internet-Draft AI Training Fabric Benchmarking August 2026 | | layer enhancements benchmarked in Section 6 | +-------------------+---------------------------------------------+ Table 1: Relationship to Existing and Companion Work 2. Terminology Terminology used in this document is defined in [TERMINOLOGY]. Readers should consult that document before applying the methodology defined here. Where a term overlaps with [RFC1242] or [RFC8238], the terminology document provides AI fabric context extensions; the foundational definitions in those RFCs remain authoritative for general network benchmarking. All terminology used in this document – including the AI fabric, RoCEv2, UET, RDMA transport, congestion control (PFC, DCQCN, ECN, CBFC), load balancing (ECMP, Packet Spray, DLB/Flowlet), collective communication, and KPI vocabulary (JCT, JCT Ratio, BusBW, MMR, etc.) – is defined normatively in [TERMINOLOGY] and is not redefined here. The following table lists the single bench-specific extension introduced by this document: +========+=====================================================+ | Term | Definition | +========+=====================================================+ | *PFC | A single PFC PAUSE frame with a non-zero quanta | | Pause | value transmitted on a priority class. Zero-quanta | | Event* | (X-ON / resume) frames are excluded from the count, | | | since implementations that signal resume explicitly | | | would otherwise report roughly double the event | | | rate of implementations that let the pause timer | | | expire; X-ON frames are instead used to bound the | | | paused interval for the cumulative-duration metric. | | | Used in this document as the unit of count for PFC | | | event-rate metrics (events/sec, cumulative | | | duration) reported by the methodology in Section 7. | +--------+-----------------------------------------------------+ Table 2: Bench-Specific Terminology Extensions In addition to the BusBW reporting requirements specified in [TERMINOLOGY], the runtime algorithm selected by the collective library MUST be verified via library tracing and documented as part of the test conditions for any AllReduce, AllGather, or AllToAll benchmark in this document. Calabria, et al. Expires 13 February 2027 [Page 7] Internet-Draft AI Training Fabric Benchmarking August 2026 The scope of the DUT for the tests defined in this document is the set of leaf switches, spine switches, superspine switches (if applicable), and interconnecting links forming the AI training fabric, consistent with the Fabric DUT Boundary defined in [TERMINOLOGY]. 2.1. Acronyms Acronyms used in this document are expanded in the Acronyms appendix of [TERMINOLOGY]. Acronyms unique to the methodology defined herein are expanded on first use in the body of this document. 3. Test Topology and Architecture 3.1. Reference Fabric Topologies Three reference topologies are defined. Every test report identifies which topology was used. Results obtained under different topologies are compared only under the conditions specified in Section 3.2, which defines the normalization basis used by this methodology and the parameters that MUST match before two results are compared. 3.1.1. Topology A: 2-Tier Clos (Leaf-Spine) +--------+ +--------+ +--------+ +--------+ | Spine1 | | Spine2 | | Spine3 | | SpineN | +---++---+ +---++---+ +---++---+ +---++---+ || || || || || Full Mesh Interconnect || || (ECMP / DLB / Spray) || || || || || +---++---+ +---++---+ +---++---+ +---++---+ | Leaf 1 | | Leaf 2 | | Leaf 3 | | Leaf N | +---++---+ +---++---+ +---++---+ +---++---+ || || || || [GPU/XPU] [GPU/XPU] [GPU/XPU] [GPU/XPU] Hosts w/ Hosts w/ Hosts w/ Hosts w/ RoCEv2 NIC RoCEv2 NIC Figure 1: Topology A: 2-Tier Clos (Leaf-Spine) The DUT boundary encompasses all leaf and spine switches and their interconnecting links. Traffic generators or actual GPU hosts connect at the leaf layer. Calabria, et al. Expires 13 February 2027 [Page 8] Internet-Draft AI Training Fabric Benchmarking August 2026 3.1.2. Topology B: 3-Tier Clos (Leaf-Spine-Superspine) For clusters exceeding thousands of accelerators, a superspine layer is added. Each pod consists of a leaf-spine fabric; pods interconnect via superspine switches. This topology scales to 32,000+ accelerators at 800GbE with current-generation ASICs. The DUT boundary encompasses all three tiers. 3.1.3. Topology C: Rail-Optimized SPINE LAYER +--------+ +--------+ +--------+ +--------+ | Spine1 | | Spine2 | | Spine3 | | SpineN | +--+--+--+ +--+--+--+ +--+--+--+ +--+--+--+ | Full Mesh Interconnect (ECMP/Spray) | +--+--+--+ +--+--+--+ +--+--+--+ +--+--+--+ | Rail-0 | | Rail-1 | | Rail-2 | | Rail-7 | RAIL (LEAF) LAYER | Leaf | | Leaf | | Leaf | | Leaf | one switch per NIC +--+--+--+ +--+--+--+ +--+--+--+ +--+--+--+ | | | | | | | | NIC-0 NIC-0 NIC-1 NIC-1 NIC-2 NIC-2 NIC-7 NIC-7 | | | | | | | | +--------+ +--------+ +--------+ +--------+ | Host A | | Host B | | Host C | | Host D | GPU HOSTS | GPU[0] | | GPU[0] | | GPU[0] | | GPU[0] | (each host has | GPU[1] | | GPU[1] | | GPU[1] | | GPU[1] | 8 NICs, one | ... | | ... | | ... | | ... | per rail) | GPU[7] | | GPU[7] | | GPU[7] | | GPU[7] | +--------+ +--------+ +--------+ +--------+ Figure 2: Topology C: Rail-Optimized (schematic; only 4 of N rails and hosts shown) In rail-optimized topologies, each NIC on a multi-NIC host connects to a dedicated leaf switch ("rail"); this co-optimizes network locality with the collective communications library (CCL) in use (e.g., NCCL, RCCL, oneCCL). The diagram is schematic: it depicts a representative subset of rails and hosts, not the full cross-host fan-out; in the complete topology, the Rail-N leaf switch connects to GPU[N] of every host. The DUT boundary and rail mapping are fully documented in the test report. Calabria, et al. Expires 13 February 2027 [Page 9] Internet-Draft AI Training Fabric Benchmarking August 2026 3.2. Result Comparability and Normalization Section 3.1 requires that results obtained under different topologies be compared only after normalization. This section defines what that normalization is, which parameters MUST be equal before two results are compared, and which quantities MUST NOT be used as normalization factors. 3.2.1. Normalization Basis Normalization in this document is achieved by reporting KPIs that are dimensionless or referenced to a rate observable at the Fabric DUT Boundary defined in [TERMINOLOGY]. It is not achieved by normalizing fabric hardware between the systems being compared. +==============+=============+==================================+ | KPI | Defined in | Normalizing denominator | +==============+=============+==================================+ | JCT Ratio | Section | Roofline_seq, whose | | | 10.1 | communication term is (8 × S × | | | | algo_factor) / B_acc | +--------------+-------------+----------------------------------+ | BusBW | Section 9 | Per-accelerator; made algorithm- | | | | invariant by the fixed | | | | algo_factor of [TERMINOLOGY] | +--------------+-------------+----------------------------------+ | BusBW | Section | Per-accelerator NIC line rate | | efficiency | 9.1, | | | | Section 9.3 | | +--------------+-------------+----------------------------------+ | Throughput | Section 5.1 | Theoretical rate of the Ethernet | | efficiency | | links under test | +--------------+-------------+----------------------------------+ | Aggregate | Section 4 | Fabric bisection bandwidth, | | Throughput | | computed from Ethernet link | | | | rates | +--------------+-------------+----------------------------------+ | MMR, JFI | Section 8 | Dimensionless by construction | +--------------+-------------+----------------------------------+ | JCT | Section | Baseline JCT measured on the | | Interference | 10.3 | same DUT | | Factor | | | +--------------+-------------+----------------------------------+ Table 3: Normalization Basis for Reported KPIs Calabria, et al. Expires 13 February 2027 [Page 10] Internet-Draft AI Training Fabric Benchmarking August 2026 Each denominator in Table 3 is either a workload parameter fixed by the test operator (S, algo_factor, N) or an Ethernet line rate at the DUT boundary. No denominator contains a term for switch internal capacity, accelerator interconnect capacity, or host bus capacity. Convergence and failover times (Section 11.2, Section 11.3), latency percentiles (Section 5.2), and queue occupancy are reported in absolute units and are not normalized; normalizing them would obscure the behavior they measure. 3.2.2. Aggregate Switching Capacity Is Not a Normalization Factor Aggregate switching capacity (ASIC forwarding capacity, in Tbps) MUST NOT be used as a normalization factor for any KPI defined in this document, and MUST NOT be held constant as a precondition for comparing two fabrics, for three reasons: 1. Aggregate switching capacity is a vendor-declared internal property, not a quantity observable at the Fabric DUT Boundary. Benchmarking in this document is performed on a black-box basis (Section 14). 2. Whether a given switching capacity is sufficient for the offered collective pattern is what the tests in Section 5 through Section 12 measure. Dividing a measured result by the capacity that produced it removes the effect under test: a fabric with half the switching capacity that completes the same job in 1.6 times the time would report a better result per Tbps while performing worse at the job. 3. A capacity-normalized value is a cost or efficiency figure of merit. Per the BMWG charter, and consistent with Section 13 and Appendix B, this document defines what is measured and how it is reported, and does not define figures of merit or acceptance criteria. Aggregate switching capacity, per-port speed, switch radix, and buffer architecture are reported as DUT characteristics (Section 3.3, Appendix C) so that a reader can attribute an observed difference to them. They are inputs to the interpretation of a result, not divisors applied to it. The same reasoning applies to intra-node switching and interconnect capacity (accelerator interconnects, PCIe, CXL). Were aggregate switching capacity used as a normalization factor, intra-node switching and interconnect capacity would have to be included for that normalization to be complete, which the DUT boundary of Section 1.2 does not permit. Because no denominator in Table 3 Calabria, et al. Expires 13 February 2027 [Page 11] Internet-Draft AI Training Fabric Benchmarking August 2026 contains such a term, the normalization defined here does not reach inside the node, and this section and Section 1.2 are consistent by construction. 3.2.3. Comparability Set Results obtained on two different fabrics are compared directly for the KPIs in Table 3 when every parameter in Table 4 is equal and reported for both results. +===========================+===============================+ | Parameter | Why it must match | +===========================+===============================+ | Participating accelerator | Collective cost depends on N | | count N | through algo_factor and | | | through fabric path length | +---------------------------+-------------------------------+ | Accelerators per node, | Determines the fabric-visible | | and their placement and | fraction of collective | | rail mapping | traffic (Section 3.2.5) | +---------------------------+-------------------------------+ | B_acc, the per- | Denominator of Roofline_seq | | accelerator Ethernet line | and of BusBW efficiency | | rate at the DUT boundary | | +---------------------------+-------------------------------+ | Leaf oversubscription | Determines whether the fabric | | ratio | can satisfy the offered | | | pattern | +---------------------------+-------------------------------+ | Collective type, message | Definition of the offered | | size S, and algo_factor | workload | +---------------------------+-------------------------------+ | Transport (RoCEv2 or | Loss and retransmission | | UET), and for UET the | semantics differ by transport | | transport service | | +---------------------------+-------------------------------+ | Load balancing strategy | Comparisons are made strategy | | | by strategy | +---------------------------+-------------------------------+ Table 4: Comparability Set When any parameter in Table 4 differs, the results are reported side by side with the difference stated, and are not combined into a single comparative figure. Calabria, et al. Expires 13 February 2027 [Page 12] Internet-Draft AI Training Fabric Benchmarking August 2026 Topology class is a reported test condition, not a parameter that normalization removes. Two fabrics of different topology class that match on Table 4 are compared for the KPIs in Table 3, and the report states, for each result: the topology class; the switch tier count; the typical and worst-case hop count for the measured traffic pattern; the number of equal-cost paths available between a source- destination pair; and the bisection bandwidth. Such a comparison characterizes the topologies under the offered workload; it does not attribute the observed difference to any single fabric component. Topologies outside the reference set of Section 3.1, including mesh, torus, and direct-connect fabrics, are outside the scope stated in Section 1.2. Results obtained on such a fabric are reported as a deviation per Section 13, and the hop-count and equal-cost-path descriptors above are supplied only where they are well defined for that topology. 3.2.4. Comparing Fabrics of the Same Topology Class *Different scale.* Comparison is performed at equal N, and scale is characterized rather than normalized away. Section 10.1 requires JCT Ratio to be reported for each N in its parameter table and plotted against N; the shape of that curve is the scaling result, and no single scalar replaces it. Two fabrics whose maximum supported scale differs are compared over the range of N that both support; the report gives the JCT Ratio at the largest common N together with the maximum N each fabric sustains per Section 11.1. *Different switch capacity, radix, or buffer size.* Comparison is performed without adjustment, provided Table 4 matches. The difference in the measured KPIs is the result of the comparison. The differing DUT characteristics are reported per Section 3.3 and Appendix C to permit attribution, and are not used to scale the measured values. 3.2.5. Fabric-Visible Data Volume Collective placement determines how much of a collective's traffic crosses the DUT boundary. A hierarchical AllReduce that reduces within a node before reducing across the fabric presents less data to the fabric than a flat AllReduce across the same N accelerators, even though the application-level message size S is identical. Two results obtained under different placement therefore represent different offered fabric workloads at equal S. For every collective result reported under Section 9, Section 6.6, or Section 10, the report states: Calabria, et al. Expires 13 February 2027 [Page 13] Internet-Draft AI Training Fabric Benchmarking August 2026 * S, the application-level data size per participant; * S_fabric, the data volume per participant that crosses the Fabric DUT Boundary, together with the method used to obtain it. S_fabric is counted in application payload bytes, each byte counted once, using the byte-counting rule of the Fabric_Goodput definition in [TERMINOLOGY]. Retransmitted and duplicate bytes MUST NOT be included in S_fabric; they are reported separately as the RDMA Retransmission Rate and out-of-order delivery rate of Section 4. Measurement from NIC Ethernet port counters is preferred; derivation from the collective algorithm and the placement is acceptable when the derivation is stated. Under this byte-counting rule the two methods return the same value; when both are available and they disagree, the report gives both values and the difference; * the accelerator placement and, in rail-optimized topologies, the rail mapping; * the Intra-Node Transfer Overhead component defined in [TERMINOLOGY], reported separately. This component is never added to, subtracted from, or folded into a fabric KPI. Comparisons use the same S and the same placement on both sides. When placement cannot be matched, for example when comparing a rail- optimized fabric using rail-aware placement against a Clos fabric that has no rail structure, the report gives S_fabric for each result so that the difference in offered fabric work is visible, and the results are not presented as an equal-workload comparison. 3.3. Device Under Test (DUT) Identification +==============+===================+=========================+ | Parameter | Description | Example | +==============+===================+=========================+ | Switch | Vendor name, | Vendor Family Model | | Vendor/Model | product family, | | | | model number | | +--------------+-------------------+-------------------------+ | Switch ASIC | Silicon vendor, | Silicon Vendor ASIC | | | ASIC family, | Family Rev | | | revision | | +--------------+-------------------+-------------------------+ | NOS Version | Network operating | NOS Name Version | | | system name and | | | | version | | +--------------+-------------------+-------------------------+ | Port Speed | Per-port line | 400GbE, 800GbE | Calabria, et al. Expires 13 February 2027 [Page 14] Internet-Draft AI Training Fabric Benchmarking August 2026 | | rate | | +--------------+-------------------+-------------------------+ | Buffer | Shared/dedicated, | 32MB shared + 16MB VOQ | | Architecture | total buffer per | per port | | | ASIC/port | | +--------------+-------------------+-------------------------+ | Optics/ | Transceiver type, | Octal Small Form-factor | | Cables | cable type and | Pluggable (OSFP) 400G- | | | length | DR4, Direct Attach | | | | Copper (DAC) 3m cable | +--------------+-------------------+-------------------------+ | NIC Vendor/ | RDMA NIC vendor, | NIC Vendor Model Speed | | Model | model, firmware | | +--------------+-------------------+-------------------------+ | NIC Firmware | NIC firmware | Firmware Version | | | version | | +--------------+-------------------+-------------------------+ | Host Config | OS, CCL lib | OS Version, CCL | | | version, driver, | Version, OFED Version | | | BIOS settings | | +--------------+-------------------+-------------------------+ Table 5: DUT Identification Parameters 3.4. Traffic Generator Requirements 3.4.1. Mandatory Functional Capabilities The traffic generator supports: RoCEv2 transport emulation (QP establishment, RDMA Write/Read, ECN processing, DCQCN rate control); configurable QP scaling (1-256 QPs per source-destination pair); programmable collective communication patterns (AllReduce, AllToAll, AllGather with configurable message sizes); and nanosecond-precision timestamping. 3.4.2. Minimum Measurement Accuracy Requirements +==============+=================================================+ | Parameter | Minimum Requirement | +==============+=================================================+ | Timestamp | ≤ 100 nanoseconds | | accuracy | | +--------------+-------------------------------------------------+ | Frame rate | +/- 0.1% of specified rate | | accuracy | | +--------------+-------------------------------------------------+ | QP scaling | 1 to 256 QPs per src-dst pair | | range | | Calabria, et al. Expires 13 February 2027 [Page 15] Internet-Draft AI Training Fabric Benchmarking August 2026 +--------------+-------------------------------------------------+ | Message size | 64 B to 2 GB (single-message ceiling; see note) | | range | | +--------------+-------------------------------------------------+ | Flow counter | Per-flow byte and packet counts | | resolution | | +--------------+-------------------------------------------------+ | Loss | Exact per-packet loss counting | | measurement | | +--------------+-------------------------------------------------+ | Burst | Burst lengths at line rate sufficient to exceed | | generation | DUT buffering; configurable beyond 1000 frames | +--------------+-------------------------------------------------+ Table 6: Minimum Measurement Accuracy Requirements NOTE: A single RDMA message cannot exceed the 32-bit RETH DMA Length field, an absolute ceiling of 4 GB, and implementations commonly advertise a practical max_msg_sz of 1-2 GB. Transfers larger than the single-message ceiling are composed from multiple RDMA messages, and the message count and per-message size are reported alongside the aggregate transfer size. 3.4.3. Acceptable Implementations The platform used is identified in all test reports. *(a) Hardware Traffic Generator* – dedicated hardware capable of line-rate RDMA emulation meeting the Measurement Accuracy Requirements specified in this document. Suitable for point-to-point RDMA tests (Section 5 and Section 6). For collective tests (Section 9), the following limitations are documented: whether synchronization barriers are reproduced, whether flow patterns are schedule-driven or gradient-driven, and whether straggler behavior is modeled. *(b) Accelerator Cluster* – cluster running an actual collective communication library with RDMA tooling. Preferred for the collective benchmarks in Section 9. Host configuration (accelerator model, collective library name and version, PCIe topology, BIOS power management settings) is documented. Any non-fabric overhead in timing measurements is quantified and reported separately. When a hardware generator is used for collective benchmarks, results should be cross-validated against an accelerator cluster at one or more overlapping (message_size, N) configurations. Calabria, et al. Expires 13 February 2027 [Page 16] Internet-Draft AI Training Fabric Benchmarking August 2026 Discrepancies exceeding 10% in BusBW or JCT Ratio are investigated and reported. 4. KPI Framework and Metrics Taxonomy NOTE: Per BMWG charter, the definition of acceptance criteria or performance requirements is explicitly outside the scope of this Working Group. The KPI tables in this section define what is measured and how it is reported; they do not set pass/fail criteria. Indicative non-normative reference values reflecting current industry observations are provided in Appendix B; those values MUST NOT be used as pass/fail criteria in vendor evaluations. 4.1. Primary KPIs +============+===============+======================================+ | KPI | Unit | Definition | +============+===============+======================================+ | Job | seconds | Wall-clock time for | | Completion | | benchmark iteration | | Time (JCT) | | (compute + communication) | +------------+---------------+--------------------------------------+ | JCT Ratio | dimensionless | Measured JCT / Roofline JCT | +------------+---------------+--------------------------------------+ | Bus | Gbps/ | Effective per-accelerator | | Bandwidth | accelerator | throughput during | | (BusBW) | | collective. See the BusBW | | | | definition in [TERMINOLOGY] | +------------+---------------+--------------------------------------+ | Aggregate | Tbps | Total fabric goodput during | | Throughput | | collective phase | +------------+---------------+--------------------------------------+ | Packet | ppm | Frames lost end-to-end not | | Drop Rate | | retransmitted | +------------+---------------+--------------------------------------+ | Tail | us | 99th/99.9th percentile one- | | Latency | | way fabric latency | | (P99/ | | | | P99.9) | | | +------------+---------------+--------------------------------------+ Table 7: Primary KPIs Calabria, et al. Expires 13 February 2027 [Page 17] Internet-Draft AI Training Fabric Benchmarking August 2026 4.2. Secondary KPIs +================+===============+===========================+ | KPI | Unit | Definition | +================+===============+===========================+ | ECN Marking | % | Percentage of packets | | Ratio | | marked CE over | | | | measurement interval | +----------------+---------------+---------------------------+ | PFC Pause | events/sec | Rate of PFC PAUSE frames | | Count | | per priority per port | +----------------+---------------+---------------------------+ | PFC Pause | us | Cumulative time a port is | | Duration | | in PFC-paused state per | | | | interval | +----------------+---------------+---------------------------+ | RDMA | retx/sec | NIC-level retransmissions | | Retransmission | | due to timeouts or NAKs | | Rate | | | +----------------+---------------+---------------------------+ | ECMP Imbalance | dimensionless | Max-Mean Ratio of flow | | (MMR) | | counts across parallel | | | | uplinks | +----------------+---------------+---------------------------+ | Jain's | 1/N-1.0 | Fairness of traffic | | Fairness Index | | distribution; 1.0 = | | (JFI) | | perfect, 1/N = worst (N = | | | | number of parallel links) | +----------------+---------------+---------------------------+ | Queue Depth | bytes or | 95th percentile and | | (P95/Max) | cells | maximum egress queue | | | | occupancy per port | +----------------+---------------+---------------------------+ | Congestion | us | Time from congestion | | Control | | onset to DCQCN rate | | Convergence | | stabilization | +----------------+---------------+---------------------------+ | Out-of-Order | pkt/sec | Packets delivered out of | | Packet Rate | | sequence (relevant for | | | | packet spray) | +----------------+---------------+---------------------------+ Table 8: Secondary KPIs Calabria, et al. Expires 13 February 2027 [Page 18] Internet-Draft AI Training Fabric Benchmarking August 2026 4.3. Fabric Health Indicators +========================+=========+===============================+ | Indicator | Unit | Definition | +========================+=========+===============================+ | Switch CPU Utilization | % | Average and peak CPU usage on | | | | DUT control plane during test | +------------------------+---------+-------------------------------+ | Switch Memory | % | Average and peak memory | | Utilization | | usage, including FIB/MAC | | | | table occupancy | +------------------------+---------+-------------------------------+ | Forwarding Information | ms | Time to converge routing | | Base (FIB) / Route | | after topology change | | Convergence Time | | | +------------------------+---------+-------------------------------+ | Link Flap Count | events | Spurious link state changes | | | | during test period | +------------------------+---------+-------------------------------+ | CRC/FCS Error Rate | errors/ | Physical layer errors | | | sec | indicating cable or optics | | | | issues | +------------------------+---------+-------------------------------+ | Power Consumption | Watts | Per-switch and per-port power | | | | draw under test load | +------------------------+---------+-------------------------------+ Table 9: Fabric Health Indicators 5. Test Category 1: RDMA Transport Benchmarks These tests establish baseline fabric performance for RDMA traffic independent of collective communication patterns. They extend [RFC2544] and [RFC8239] methodology for RoCEv2 semantics. 5.1. Baseline Throughput *Objective:* Determine the maximum sustainable RDMA Write throughput through the DUT fabric at each tested message size. *Procedure:* * Configure N host pairs, each establishing Q Queue Pairs per pair * Initiate RDMA Write operations and measure aggregate goodput * Each test runs for at least 60 seconds at each rate Calabria, et al. Expires 13 February 2027 [Page 19] Internet-Draft AI Training Fabric Benchmarking August 2026 * Binary search per [RFC2544] Section 26.1 is used * Message sizes: 64B, 256B, 1KB, 4KB, 64KB, 256KB, 1MB, 4MB * QP counts: 1, 4, 16, 32 per src-dst pair * Test both unidirectional and bidirectional traffic *Reporting:* Report aggregate throughput (Tbps), per-port utilization (%), and throughput efficiency (measured/theoretical). Present as table indexed by message size × QP count, and as graph (message size on X-axis). 5.2. Latency Characterization *Objective:* Determine one-way and round-trip RDMA latency distribution at the throughput rate from Section 5.1. *Procedure:* * Inject tagged frames at 60s into a 120s stream (per [RFC2544] Section 26.2) * Nanosecond-precision timestamping * Reported statistics: min, mean, P50, P95, P99, P99.9, max * Each run MUST capture at least 10,000 latency samples to support a statistically meaningful P99.9 * Repeat at least 20 times; pool all samples across runs and compute percentiles from the pooled distribution, and report the per-run variability (e.g., min/max across runs) alongside the pooled statistics * Test under both zero-load (single QP) and loaded (full fabric utilization) conditions *Reporting:* Tabulate latency statistics per message size. Provide histogram and CDF plot. Report latency increase factor (loaded/ unloaded). 5.3. Back-to-Back Burst Absorption *Objective:* Characterize the DUT fabric's ability to absorb back-to- back RDMA bursts without loss. This test extends [RFC9004] methodology for RoCEv2. Calabria, et al. Expires 13 February 2027 [Page 20] Internet-Draft AI Training Fabric Benchmarking August 2026 *Procedure:* * Transmit bursts at line rate with minimum inter-frame gap * Increase burst length until first frame loss is detected * Test incast ratios: 2:1, 4:1, 8:1, 16:1, 32:1 * Repeat at least 50 times per burst length *Reporting:* Report burst absorption capacity (frames and bytes) for each message size and incast ratio. Plot burst capacity vs. incast ratio. 6. Test Category 2: UEC Transport Protocol Benchmarks The Ultra Ethernet Consortium (UEC) Specification 1.0 [UEC-1.0] defines UET, an RDMA transport positioned as an alternative to RoCEv2 for AI/HPC workloads. All UET tests use the libfabric API [LIBFABRIC] and run on UEC 1.0-compliant NICs. The UEC compliance profile (AI Base, AI Full, or HPC) used during testing is documented in the test report. 6.1. UET Throughput by Transport Service *Objective:* Determine maximum sustainable throughput under each UET transport service (ROD, RUD, RUDI, UUD) and compare to RoCEv2 Reliable Connected (RC) / Unreliable Connected (UC) on the same DUT fabric. *Procedure:* Use UEC 1.0-compliant NICs; establish PDCs; use libfabric fi_write. Apply binary search ([RFC2544] Section 26.1). Vary PDC counts: 1, 4, 16, 32. A parallel RoCEv2 test series is executed for comparison. Both unidirectional and bidirectional configurations are tested. *Reporting template:* Calabria, et al. Expires 13 February 2027 [Page 21] Internet-Draft AI Training Fabric Benchmarking August 2026 +==============+======+========+========+========+========+========+ | Metric |ROD | RUD | RUDI | UUD | RoCEv2 | RoCEv2 | | | | | | | RC | UC | +==============+======+========+========+========+========+========+ | Throughput @ |(meas)| (meas) | (meas) | (meas) | (meas) | (meas) | | 1MB (Gbps) | | | | | | | +--------------+------+--------+--------+--------+--------+--------+ | Throughput @ |(meas)| (meas) | (meas) | (meas) | (meas) | (meas) | | 4MB (Gbps) | | | | | | | +--------------+------+--------+--------+--------+--------+--------+ | Efficiency |(meas)| (meas) | (meas) | (meas) | (meas) | (meas) | | (% line | | | | | | | | rate) | | | | | | | +--------------+------+--------+--------+--------+--------+--------+ | Connection/ |(meas)| (meas) | (meas) | (meas) | (meas) | (meas) | | PDC | | | | | | | | Initiation | | | | | | | | Latency (us) | | | | | | | | (see note) | | | | | | | +--------------+------+--------+--------+--------+--------+--------+ | Max |(meas)| (meas) | (meas) | (meas) | (meas) | (meas) | | Sustained | | | | | | | | PDC/QP Count | | | | | | | +--------------+------+--------+--------+--------+--------+--------+ Table 10: UET Throughput by Transport Service NOTE: For RoCEv2 RC/UC, this measures QP connection setup time (handshake round-trip). For UET (ROD/RUD/RUDI/UUD), PDCs have no separate setup handshake; this measures the first-packet latency reflecting in-band PDC initiation instead. 6.2. UET Latency Characterization *Objective:* Measure latency distribution for UET transport services; quantify differential vs. RoCEv2, with particular attention to connectionless PDC establishment overhead. *Procedure:* Measure latency for: (a) steady-state PDC transfers; (b) first-packet latency (PDC + first data packet, measuring "data before handshake"); (c) zero-load baseline. Test ROD and RUD separately to isolate reordering-related latency. *Reporting:* Tabulate latency statistics per (transport_service, message_size, load_condition) tuple. Plot latency CDF for UET ROD, UET RUD, and RoCEv2 RC side-by-side. Calabria, et al. Expires 13 February 2027 [Page 22] Internet-Draft AI Training Fabric Benchmarking August 2026 6.3. Packet Spray Efficacy Under UET RUD *Objective:* Quantify the load balancing improvement achieved by UET's native per-packet spray with RUD, which eliminates the receiver reorder buffer constraint. *Procedure:* Test five configurations: * UET RUD + packet spray * UET ROD + packet spray * RoCEv2 RC + packet spray * RoCEv2 RC + standard ECMP (baseline) * UET RUD + DLB/Flowlet Measure MMR, JFI, out-of-order delivery rate, retransmission rate, and effective goodput. Vary ECMP paths: 4, 8, 16, 32. *Reporting template:* +=================+========+========+========+========+=============+ | Load Balancing | MMR | JFI | OOO | Retx | Effective | | Config | | | Rate | Rate | Goodput (%) | +=================+========+========+========+========+=============+ | UET RUD + | (meas) | (meas) | (meas) | (meas) | (meas) | | Packet Spray | | | | | | +-----------------+--------+--------+--------+--------+-------------+ | UET ROD + | (meas) | (meas) | (meas) | (meas) | (meas) | | Packet Spray | | | | | | +-----------------+--------+--------+--------+--------+-------------+ | RoCEv2 RC + | (meas) | (meas) | (meas) | (meas) | (meas) | | Packet Spray | | | | | | +-----------------+--------+--------+--------+--------+-------------+ | RoCEv2 RC + | (meas) | (meas) | (meas) | (meas) | (meas) | | ECMP | | | | | | | (baseline) | | | | | | +-----------------+--------+--------+--------+--------+-------------+ | UET RUD + DLB/ | (meas) | (meas) | (meas) | (meas) | (meas) | | Flowlet | | | | | | +-----------------+--------+--------+--------+--------+-------------+ Table 11: Packet Spray Efficacy Under UET RUD Calabria, et al. Expires 13 February 2027 [Page 23] Internet-Draft AI Training Fabric Benchmarking August 2026 This test evaluates whether UET RUD achieves zero host-visible reordering despite per-packet spray, since the transport layer is designed to tolerate unordered delivery. 6.4. UET Congestion Control Benchmarks *Objective:* Evaluate UET's dual-sided (sender + receiver) congestion control under N:1 incast conditions vs. RoCEv2 DCQCN. *Procedure:* Measure: (a) incast throughput at N = {2, 4, 8, 16, 32, 64}; (b) convergence time after doubling active senders (until all flows within 10% of fair share); (c) PFC avoidance with PFC disabled on the DUT; (d) receiver credit utilization. *Reporting:* Tabulate incast throughput, convergence time, peak queue depth, PFC event count, and packet drop rate for UET vs. DCQCN per incast ratio. *Key metric:* report whether UET achieves zero application-visible loss without PFC. 6.5. Link-Layer and Network-Layer Enhancement Benchmarks *Objective:* Measure performance impact of optional UEC enhancements: LLR and CBFC (link-layer) and Packet Trimming (PT, network-layer). *Procedure:* * *(a) LLR Retry Latency:* inject controlled bit errors; measure LLR retry latency (expected sub-microsecond per hop) vs. transport- layer retransmission (~10-100us RTT). Run with 80% background load. * *(b) Packet Trimming Effectiveness:* configure 2:1 oversubscription bottleneck; measure time from congestion onset to first retransmission request, bandwidth saved vs. full-packet drops. * *(c) CBFC vs. PFC:* identical N:1 (N=32) incast scenarios; measure head-of-line blocking duration (CBFC is per-destination, PFC is per-priority), pause propagation hops, and throughput of non- congested flows. *Reporting:* Before/after comparison table for each enhancement. Note which features are hardware-supported vs. software-emulated. Calabria, et al. Expires 13 February 2027 [Page 24] Internet-Draft AI Training Fabric Benchmarking August 2026 6.6. UET Collective Communication Performance *Objective:* Measure collective communication (AllReduce, AllToAll, AllGather) performance over UET and compare directly to RoCEv2, isolating the transport protocol contribution to collective efficiency. *Procedure:* Execute the collective benchmark suite from Section 9 over UET RUD transport using a UEC-compliant collective library. The same accelerator count (N), message sizes, and fabric topology are used for both UET and RoCEv2 runs to ensure a valid comparison. Run UET RUD + packet spray as the primary configuration and UET ROD + ECMP as the secondary baseline. For AllReduce, the UET TSS group-key encryption state (active or inactive) on the DUT NIC is documented as a required result field in the test report. When UET TSS group-key encryption is active during testing, report the observed BusBW computed from measured bytes transferred per the algo_factor formula defined in [TERMINOLOGY] (fixed per collective type); group-key encryption affects per-packet security processing overhead, not the transfer volume itself. The runtime algorithm in use is reported per message-size bucket. See [TERMINOLOGY] for the BusBW definition and algo_factor values. *Reporting:* Report the percentage improvement in BusBW and JCT attributable to UET native packet spray and congestion control. *Reporting template:* +============+======+========+=========+=========+========+========+ | Collective | Msg | N | UET RUD | UET ROD | RoCEv2 | Delta | | | Size | Accels | BusBW | BusBW | RC | UET/ | | | | | | | BusBW | RoCEv2 | +============+======+========+=========+=========+========+========+ | AllReduce | 1GB | 128 | (meas) | (meas) | (meas) | (meas) | +------------+------+--------+---------+---------+--------+--------+ | AllReduce | 1GB | 512 | (meas) | (meas) | (meas) | (meas) | +------------+------+--------+---------+---------+--------+--------+ | AllToAll | 1GB | 128 | (meas) | (meas) | (meas) | (meas) | +------------+------+--------+---------+---------+--------+--------+ | AllGather | 1GB | 128 | (meas) | (meas) | (meas) | (meas) | +------------+------+--------+---------+---------+--------+--------+ Table 12: UET Collective Communication Performance Calabria, et al. Expires 13 February 2027 [Page 25] Internet-Draft AI Training Fabric Benchmarking August 2026 6.7. UET PDC Scalability and Connection Setup Rate *Objective:* Measure PDC establishment rate and maximum concurrent PDC count vs. RoCEv2 QP-based connections. *Procedure:* (a) PDC establishment rate: initiate PDC creation to M = {100, 1000, 10000, 100000} remote endpoints. (b) Data-before- handshake: measure first-byte latency for UET vs. RoCEv2 RDMA Write. (c) Maximum concurrent PDC count: scale until per-PDC throughput drops below 90% of single-PDC rate. The UEC specification [UEC-1.0] targets millions of endpoints. NOTE: PDC and QP scaling limits are a host NIC capability, outside the DUT boundary per [TERMINOLOGY] and Section 1.2. They are reported as context because a NIC-side PDC/QP ceiling presents as fabric underperformance in the throughput and latency measurements above. 7. Test Category 3: Congestion Management AI training workloads generate repetitive micro-congestion during the back-propagation gradient synchronization phase. 7.1. ECN Marking Accuracy and Threshold *Objective:* Verify that the DUT marks packets with ECN CE at the configured threshold with correct granularity. *Procedure:* Configure threshold T on DUT egress queue. Verify: (a) no packets marked below T; (b) 100% marked above maximum threshold; (c) appropriate Weighted Random Early Detection (WRED) / Random Early Detection (RED) probability ramp between thresholds. Test thresholds: low (~100KB), medium (~1MB), high (~5MB). *Reporting:* Plot ECN marking probability vs. instantaneous queue depth. Report measured threshold accuracy (deviation from configured). 7.2. PFC Behavior Under Incast *Objective:* Characterize DUT's PFC generation behavior under N:1 incast conditions. *Procedure:* Generate N:1 incast at 100% line rate, N = {2, 4, 8, 16, 32, 64}. Measure PFC PAUSE frame count/sec per hop, PFC PAUSE duration per port, PFC storm onset, and end-to-end throughput. The test characterizes headroom sizing and PFC watchdog effectiveness. Calabria, et al. Expires 13 February 2027 [Page 26] Internet-Draft AI Training Fabric Benchmarking August 2026 7.3. DCQCN Convergence Time *Objective:* Measure time for DCQCN to converge to fair-share rate after congestion onset. *Procedure:* Establish M flows through a common bottleneck. At T0, inject additional M flows (creating 2:1 oversubscription). Measure time until all 2M flows achieve rates within 10% of fair share. Repeat for M = {4, 16, 64, 256}. Vary DCQCN parameters and report sensitivity. 7.4. PFC Storm and Deadlock Resilience *Objective:* Verify the DUT does not enter PFC deadlock or sustained PFC storm under adversarial traffic. *Procedure:* Generate cyclic traffic patterns known to cause PFC deadlocks. Run for 300 seconds. The test characterizes whether the DUT demonstrates resilience via PFC watchdog or architectural immunity (e.g., VOQ-based scheduling); the mechanism observed is reported. 8. Test Category 4: Load Balancing Efficacy Load balancing across parallel fabric paths is critical for AI training fabrics because the traffic consists of a small number of high-bandwidth, long-lived elephant flows. 8.1. ECMP Entropy and Polarization *Objective:* Quantify traffic polarization under standard ECMP hashing for AI training flow patterns. *Procedure:* Configure standard 5-tuple ECMP. Generate traffic with Q = {1, 4, 8, 16, 32} QPs per src-dst pair. Measure per-link utilization, MMR, and JFI. Test with and without ECMP hashing that includes the BTH destination QP field as a hash input (in addition to the standard 5-tuple). Repeat for fabric sizes of 8, 16, 32, and 64 leaf switches. 8.2. Dynamic Load Balancing (Flowlet) *Objective:* Evaluate DUT's flowlet-based DLB performance and compare to baseline ECMP. Calabria, et al. Expires 13 February 2027 [Page 27] Internet-Draft AI Training Fabric Benchmarking August 2026 *Procedure:* Configure vendor-specific DLB (document algorithm type). Generate traffic with Q=4 QPs. Measure MMR, JFI, per-link utilization, out-of-order rate. Vary flowlet gap timer and report sensitivity. 8.3. Packet Spraying *Objective:* Evaluate DUT's per-packet spraying performance and quantify the utilization vs. reordering tradeoff. *Procedure:* Configure per-packet load balancing. Measure MMR (expected ~1.0), JFI (expected ~1.0), out-of-order rate, and RDMA retransmission impact. If the DUT provides an in-fabric reorder buffer, document per Appendix C. 8.4. Jain's Fairness Index Measurement *Objective:* Single-number summary of load balancing quality comparable across all strategies. *Formula:* JFI = (Sum LinkTx_i)^2 / (N × Sum LinkTx_i^2) Figure 3: Jain's Fairness Index Formula where LinkTx_i = transmitted traffic on fabric link i, N = total parallel links. Range: 1/N (worst) to 1.0 (perfect). *Reporting:* Report JFI for each load balancing strategy. Provide bar chart comparing ECMP, DLB, and packet spray. 9. Test Category 5: Collective Communication Benchmarks These tests evaluate the fabric's performance under realistic collective communication patterns. Unlike synthetic RDMA tests in Section 5 and Section 6, these exercise the full stack including the collective communications library (CCL) in use (e.g., NCCL, RCCL, oneCCL). Because collective placement determines how much of a collective's traffic crosses the DUT boundary, every result in this section is reported together with the fabric-visible data volume and placement information required by Section 3.2.5. Calabria, et al. Expires 13 February 2027 [Page 28] Internet-Draft AI Training Fabric Benchmarking August 2026 9.1. AllReduce Benchmark *Objective:* Measure fabric performance during AllReduce operations, the dominant collective for gradient synchronization in data-parallel training. *Procedure:* Using N accelerators connected through the DUT fabric, execute AllReduce (sum) operations using a collective communications library benchmark suite (e.g., nccl-tests, rccl-tests, or equivalent). Test parameters: * Message sizes: 1 MB, 8 MB, 64 MB, 256 MB, 1 GB, 4 GB * Accelerator counts (N): 8, 16, 32, 64, 128, 256, 512, 1024 * Minimum iterations per (message_size, N) pair: 100 * Load balancing strategies: ECMP, DLB, packet spray For each (message_size, N) pair, record average, P50, P95, and P99 BusBW, ECN marking ratio, PFC pause count, and per-link utilization. BusBW is computed per the BusBW definition in [TERMINOLOGY]; algo_factor is fixed per collective type and does not vary with the algorithm the library selects at runtime. The runtime algorithm selected by the library for each message-size bucket is verified via library tracing and documented as part of the test conditions. *Reporting:* Tabulate BusBW for each (message_size, N, LB_strategy, Algorithm (verified)) combination. The "Algorithm (verified)" column is required; results without it are incomplete. Plot BusBW vs. N for each message size. Report BusBW efficiency = BusBW / NIC_line_rate. 9.2. AllToAll Benchmark *Objective:* Measure fabric performance during AllToAll operations, the dominant collective for Mixture-of-Experts (MoE) expert parallelism dispatch. *Procedure:* Using the same message sizes, accelerator counts, iteration count, and load balancing strategies as Section 9.1, execute AllToAll operations via the collective communication library. Calabria, et al. Expires 13 February 2027 [Page 29] Internet-Draft AI Training Fabric Benchmarking August 2026 AllToAll generates the worst-case fabric stress pattern: every accelerator simultaneously sends a unique payload to every other accelerator in the group, which creates maximum entropy and stresses every fabric link with many-to-many traffic overlap. This makes AllToAll JCT the most sensitive single indicator of fabric congestion management quality. BusBW is computed per the BusBW definition in [TERMINOLOGY]; algo_factor is fixed per collective type and does not depend on topology or library implementation. The runtime algorithm in use is verified via library tracing and documented as part of the test conditions. *Measurement:* Report BusBW (average, P50, P95, P99), JCT per iteration, ECN marking ratio, PFC pause count, and per-link utilization for each (message_size, N, LB_strategy) combination. *Reporting:* Same table format as Section 9.1, with the "Algorithm (verified)" column required. Additionally report JCT for each configuration; JCT degradation relative to the ECMP baseline is highlighted as the primary congestion sensitivity indicator. 9.3. AllGather Benchmark *Objective:* Measure fabric performance during AllGather operations, the dominant collective for weight and activation distribution in tensor-parallel training. *Procedure:* Using the same message sizes, accelerator counts, iteration count, and load balancing strategies as Section 9.1, execute AllGather operations via the collective communication library. AllGather consists of a gather phase only – each accelerator contributes a shard and receives the full concatenated tensor. There is no reduce phase, which produces lower peak fabric load than AllReduce at equivalent message size and N. This makes AllGather a useful baseline for isolating the gather-path fabric contribution from the combined send-and-reduce cost. BusBW is computed per the BusBW definition in [TERMINOLOGY]; algo_factor is fixed per collective type and does not depend on the library's algorithm selection. The runtime algorithm in use is verified via library tracing and documented as part of the test conditions. Calabria, et al. Expires 13 February 2027 [Page 30] Internet-Draft AI Training Fabric Benchmarking August 2026 *Measurement:* Report BusBW (average, P50, P95, P99), JCT per iteration, ECN marking ratio, PFC pause count, and per-link utilization for each (message_size, N, LB_strategy) combination. *Reporting:* Same table format as Section 9.1, with the "Algorithm (verified)" column required. Report BusBW efficiency = BusBW / NIC_line_rate. Where results are compared to AllReduce under identical parameters, the BusBW ratio (AllGather / AllReduce) quantifies the difference in fabric load between the two traffic patterns; AllReduce's additional network traffic reflects its ReduceScatter phase, not the reduction arithmetic itself, which is performed by the accelerators, not the fabric. 9.4. Collective Communication Library Bus Bandwidth Summary *Reporting template:* +============+======+========+============+===========+=============+ | Collective | Msg | N | ECMP BusBW | DLB BusBW | Spray | | | Size | Accels | (Gbps/ | (Gbps/ | BusBW | | | | | accel) | accel) | (Gbps/ | | | | | | | accel) | +============+======+========+============+===========+=============+ | AllReduce | 1GB | 128 | (meas) | (meas) | (meas) | +------------+------+--------+------------+-----------+-------------+ | AllReduce | 1GB | 512 | (meas) | (meas) | (meas) | +------------+------+--------+------------+-----------+-------------+ | AllToAll | 1GB | 128 | (meas) | (meas) | (meas) | +------------+------+--------+------------+-----------+-------------+ | AllToAll | 1GB | 512 | (meas) | (meas) | (meas) | +------------+------+--------+------------+-----------+-------------+ | AllGather | 1GB | 128 | (meas) | (meas) | (meas) | +------------+------+--------+------------+-----------+-------------+ | AllGather | 1GB | 512 | (meas) | (meas) | (meas) | +------------+------+--------+------------+-----------+-------------+ Table 13: Collective Communication Bus Bandwidth Summary 10. Test Category 6: Job Completion Time (JCT) Benchmarks JCT is the single most important user-facing KPI for AI training fabrics; it directly determines accelerator utilization and training cost. Calabria, et al. Expires 13 February 2027 [Page 31] Internet-Draft AI Training Fabric Benchmarking August 2026 10.1. Synthetic JCT Under Controlled Conditions *Objective:* Measure JCT for a defined synthetic workload with a known computation-to-communication ratio to isolate fabric-induced overhead. *Procedure:* Define a synthetic training iteration as a strictly sequential model: 1. Computation phase of C milliseconds (simulated sleep or GPU compute kernel) 2. Communication phase: AllReduce of S bytes across N accelerators +=====================+==============================+ | Parameter | Values | +=====================+==============================+ | Computation time C | 10 ms, 50 ms, 100 ms, 500 ms | +---------------------+------------------------------+ | Message size S | 256 MB, 1 GB, 4 GB | +---------------------+------------------------------+ | Accelerator count N | 64, 128, 256, 512, 1024 | +---------------------+------------------------------+ | Iterations | 1000 | +---------------------+------------------------------+ Table 14: Synthetic JCT Test Parameters Execute 1000 iterations and measure total wall-clock JCT. Calabria, et al. Expires 13 February 2027 [Page 32] Internet-Draft AI Training Fabric Benchmarking August 2026 Roofline_seq = Iterations × (C + (8 × S × algo_factor) / B_acc) JCT Ratio = Measured_JCT / Roofline_seq where: C = compute time per iteration, in seconds (convert the millisecond values in the parameter table to seconds) S = message size per iteration (bytes) algo_factor = fixed normalization constant per collective type; see the BusBW definition in the companion terminology document B_acc = aggregate per-accelerator NIC line rate (bits/second); sum across all NICs serving the accelerator (e.g., in rail-optimised topologies, the sum of all rail NIC speeds) Iterations = number of synthetic iterations executed The factor of 8 converts S from bytes to bits to match the units of B_acc. Figure 4: JCT Ratio Calculation This model assumes strictly sequential compute and communication phases and represents a conservative upper bound on communication overhead. Many frameworks overlap these phases via gradient bucketing or asynchronous collectives, which reduces the effective communication overhead visible in wall-clock JCT. Implementations using overlapped execution additionally report: Overlap_Fraction = 1 - (Measured_JCT - C_total) / Comm_time where: C_total = Iterations × C Comm_time = Iterations × (8 × S × algo_factor) / B_acc S, algo_factor, B_acc as defined for Roofline_seq above. Figure 5: Overlap Fraction Calculation An Overlap_Fraction of 0 indicates fully sequential execution; 1.0 indicates communication is perfectly hidden behind compute. When overlap is present, the residual fabric overhead is reported as: Effective_Comm_Overhead = Measured_JCT - C_total Calabria, et al. Expires 13 February 2027 [Page 33] Internet-Draft AI Training Fabric Benchmarking August 2026 The Overlap_Fraction and communication-library overlap configuration (e.g., bucket size, number of async streams) are documented as part of the test configuration when this optional measurement is reported. *Reporting:* Tabulate JCT Ratio for each (C, S, N, LB_strategy) combination. Plot JCT Ratio vs. N to characterize fabric scalability. NOTE: JCT Ratio values of 1.05 and 1.15 are cited elsewhere in this document (Appendix B) as illustrative reference points, not as pass/fail thresholds. Per the BMWG charter, the definition of acceptance criteria or performance requirements is explicitly outside the scope of this Working Group; deployment-specific thresholds are outside the scope of this document. 10.2. MLPerf-Aligned JCT *Objective:* Measure JCT using MLPerf Training benchmark workloads [MLPERF] to enable comparison with published industry results. *Procedure:* Execute the current MLPerf Training closed-division workloads per MLPerf submission rules ([MLPERF]); the specific workload set changes across MLPerf versions, so the test report MUST identify the MLPerf Training version and workload names used. Simultaneously capture all fabric KPIs from Section 4. Report time- to-train and/or tokens-per-second. 10.3. Multi-Tenant JCT Interference *Objective:* Quantify JCT impact when multiple training jobs share the same fabric. *Procedure:* Configure two or more independent training jobs. Jobs are configured to overlap in spine-layer link usage. Measure baseline JCT (isolated) and contention JCT (simultaneous). JCT Interference Factor = Contention_JCT / Baseline_JCT Figure 6: JCT Interference Factor Test with spine link overlap: 0%, 25%, 50%, 75%. 11. Test Category 7: Scale and Convergence 11.1. Fabric Scale Limits *Objective:* Determine the maximum fabric scale at which the DUT maintains acceptable KPI performance. Calabria, et al. Expires 13 February 2027 [Page 34] Internet-Draft AI Training Fabric Benchmarking August 2026 *Procedure:* Progressively increase active accelerator endpoints from N=64 to maximum topology support while running AllReduce (Section 9.1, S=1GB). At each scale point record JCT Ratio, BusBW, ECN ratio, PFC count, CPU and memory utilization. Also measure BGP/ routing convergence time after clearing all adjacencies (analogous to the convergence testing approach in [EVPN-BENCH]). 11.2. Link Failure Convergence *Objective:* Measure traffic disruption and JCT impact when a fabric link fails during active training. *Procedure:* With the fabric fully loaded (AllReduce, N=128, S=1GB), administratively fail a spine uplink. Measure: * Duration of packet loss * Packets lost * JCT overhead for the failure iteration vs. steady state * Time for load balancing mechanism to redistribute flows Repeat for: leaf uplink failure, spine switch failure, superspine link failure (if applicable). Test under each load balancing strategy. 11.3. Zero-Impact Failover Measurement *Objective:* Verify vendor claims of zero-impact or sub-microsecond failover. *Procedure:* Execute Section 11.2 with nanosecond-precision measurement. A failure is considered "zero-impact" if the measured JCT for the failure iteration is within the P99 JCT of steady-state iterations. 12. Test Category 8: Soak and Stability 12.1. 24-Hour Sustained Load *Objective:* Characterize DUT fabric stability under sustained AI training load over an extended period, following the soak-testing methodology pattern in [EVPN-BENCH]. *Procedure:* Configure DUT at maximum validated scale from Section 11.1. Generate bidirectional collective communication traffic (alternating AllReduce and AllToAll) at 80% of maximum Calabria, et al. Expires 13 February 2027 [Page 35] Internet-Draft AI Training Fabric Benchmarking August 2026 validated throughput, per the offered load fraction required by the Soak Test definition in [TERMINOLOGY]. Run continuously for 24 hours. Sample all KPIs from Section 4 every 60 seconds. The objective of the soak test is to monitor and document fabric behavior under extended load. The methodology does not establish pass/fail criteria for any reported metric. Any memory leaks, crashes, or other anomalies encountered during the test MUST be documented as an application log file or other dedicated file with their timestamps and durations. *Reporting:* Time-series plots of JCT Ratio, BusBW, ECN ratio, PFC count, CPU, and memory over the 24-hour period. Report standard deviation of JCT Ratio (stability metric). 12.2. Resource Leak Detection *Objective:* Detect memory leaks, handle exhaustion, or gradual performance degradation in DUT software. *Procedure:* Record per-process memory usage at T=0, T=1h, T=6h, T=12h, T=24h. Compute linear regression slope of memory usage over time. A slope exceeding *1 MB/hour* for any process indicates a potential memory leak and is reported; this slope is a reporting trigger for investigation, not a pass/fail criterion. Also monitor forwarding-plane counter wraparounds and hardware table occupancy trends. 13. Reporting Format Per the BMWG charter, the definition of acceptance criteria or performance requirements is explicitly outside the scope of this Working Group. This methodology defines what is measured and how it is reported; it does not set minimum acceptable values, certification, or pass/fail criteria. Any deployment-specific performance objectives are outside the scope of this document. Results from collective communication benchmarks (Section 9) MUST be reported per the reporting requirements stated in the BusBW definition of [TERMINOLOGY]. Test reports include the following sections: 1. *DUT Identification:* Complete parameters from Section 3.3 for all fabric components. 2. *Test Topology:* Diagram and description per Section 3.1, including physical cabling. Calabria, et al. Expires 13 February 2027 [Page 36] Internet-Draft AI Training Fabric Benchmarking August 2026 3. *Test Configuration:* All DUT configuration parameters: QoS policies (ECN thresholds, PFC headroom, DCQCN parameters), load balancing mode, buffer allocation, and vendor-specific tuning. 4. *Host Configuration:* Complete host stack description per Section 3.3 including NIC firmware, driver, collective library version, and any tuning. For UET tests, additionally report: UEC compliance profile, libfabric provider version, NIC UEC firmware version, and enabled optional features (LLR, Packet Trimming, Packet Rate Improvement (PRI), CBFC). 5. *Test Results:* For each test from Section 5 through Section 12, provide specified tables, graphs, and statistical summaries. For Section 6 tests, results include side-by-side UET vs. RoCEv2 comparison data on the identical DUT fabric. 6. *Anomalies:* Any deviations from specified procedures, test failures, or unexpected behaviors are documented. 7. *Repeatability Statement:* Report iteration count and coefficient of variation (std deviation / mean) for each test's primary metric. A CV of 5% is an illustrative reference point for typical run-to-run variation; per the charter disclaimer above, this document does not set a required or minimum threshold for test validity. 8. *Comparability Statement:* When a report compares two or more fabrics, tabulate the comparability set of Section 3.2 for each result and state explicitly which parameters differ. Reports comparing fabrics of different topology class additionally provide the structural descriptors listed in that section. Reports that present results from a single fabric do not require this section. 14. Security Considerations This document defines benchmarking methodology for controlled laboratory environments and does not specify any protocol mechanism. It therefore introduces no new protocol-level security considerations beyond those of the underlying technologies it references. The considerations below follow the BMWG convention established in [RFC8238] and align with the companion terminology document [TERMINOLOGY]. Benchmarking activities as described in this document are limited to technology characterization of AI training fabrics using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified herein. Calabria, et al. Expires 13 February 2027 [Page 37] Internet-Draft AI Training Fabric Benchmarking August 2026 The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network. This isolation requirement is particularly important for AI fabric benchmarking because the hop-by-hop flow- control mechanisms referenced in this document (PFC, CBFC) propagate backpressure toward traffic sources and can extend the blast radius of a misconfigured test beyond the immediate DUT; DCQCN reduces, but does not eliminate, reliance on these mechanisms. Benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the DUT as defined in [TERMINOLOGY]. Special capabilities SHOULD NOT exist in the DUT specifically for benchmarking purposes. Any implications for network security arising from the DUT SHOULD be identical in the lab and in production networks. In particular, RDMA memory-region permissions are properties of the deployed configuration, not of the benchmarking methodology, and SHOULD reflect production posture during testing. Per [RFC6815], the tests defined herein MUST NOT be performed on production networks. The use of dedicated test IP address ranges per [RFC2544] Appendix C (198.18.0.0/15 for IPv4; 2001:db8::/32 per [RFC3849] for IPv6) is RECOMMENDED to prevent accidental interaction with production infrastructure. The following considerations are specific to the methodology defined in this document: * *PFC leakage:* PFC PAUSE frames generated under incast or storm conditions (Section 7.2, Section 7.4) that escape the test environment can cause adjacent production switches sharing the same priority class to stop responding. Physical or VLAN-based isolation of the test fabric is required. * *Line-rate RDMA traffic generators:* the equipment specified in Section 3.4 is capable of saturating production links at line rate; such generators MUST be confined to the test fabric. * *PFC disabled in Section 6.4:* the UET PFC-free incast test deliberately disables PFC on the DUT. In this configuration, traffic leaking to adjacent infrastructure cannot be backpressured and will be dropped on the adjacent device's queues. Isolation is mandatory. Calabria, et al. Expires 13 February 2027 [Page 38] Internet-Draft AI Training Fabric Benchmarking August 2026 * *RDMA QP and PDC namespace isolation:* when RDMA/RoCEv2 traffic is used, the test environment SHOULD be isolated from production RDMA fabrics to prevent QP number space collisions or inadvertent PFC propagation. When UET traffic is used (Section 6), the test environment MUST ensure that UDP port 4793 traffic does not leak to production networks and that PDC identifier spaces are isolated. * *UET transport security sub-layer (TSS):* SHOULD NOT be enabled during performance benchmarking unless transport security overhead is explicitly being measured. 15. IANA Considerations This document has no IANA actions. 16. References 16.1. Normative References [RFC1242] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, . [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, . [RFC6815] Bradner, S., Dubray, K., McQuaid, J., and A. Morton, "Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful", RFC 6815, DOI 10.17487/RFC6815, November 2012, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8238] Avramov, L. and J. Rapp, "Data Center Benchmarking Terminology", RFC 8238, DOI 10.17487/RFC8238, August 2017, . Calabria, et al. Expires 13 February 2027 [Page 39] Internet-Draft AI Training Fabric Benchmarking August 2026 [RFC8239] Avramov, L. and J. Rapp, "Data Center Benchmarking Methodology", RFC 8239, DOI 10.17487/RFC8239, August 2017, . [RFC9004] Morton, A., "Updates for the Back-to-Back Frame Benchmark in RFC 2544", RFC 9004, DOI 10.17487/RFC9004, May 2021, . [TERMINOLOGY] Calabria, F., Pignataro, C., Wu, Q., Fioccola, G., and S. Reddy, "Benchmarking Terminology for AI Network Fabrics", Work in Progress, Internet-Draft, draft-calabria-bmwg-ai- fabric-terminology-04, 12 August 2026, . [UEC-1.0] Ultra Ethernet Consortium, "Ultra Ethernet Transport (UET) Specification 1.0", June 2025, . 16.2. Informative References [DCQCN-PAPER] Zhu, Y., Eran, H., Firestone, D., Guo, C., Lipshteyn, M., Liron, Y., Padhye, J., Raindel, S., Yahia, M. H., and M. Zhang, "Congestion Control for Large-Scale RDMA Deployments", ACM SIGCOMM pp. 523-536, DOI 10.1145/2785956.2787484, 2015, . [EVPN-BENCH] Jacob, S. and K. Tiruveedhula, "Benchmarking Methodology for EVPN and PBB-EVPN", Work in Progress, Internet-Draft, draft-ietf-bmwg-evpntest-11, August 2023, . [INFERENCE-BENCH] Calabria, F., Pignataro, C., Wu, Q., Fioccola, G., and S. Reddy, "Benchmarking Methodology for AI Inference Serving Network Fabrics", Work in Progress, Internet-Draft, draft- calabria-bmwg-ai-fabric-inference-bench-03, 6 July 2026, . Calabria, et al. Expires 13 February 2027 [Page 40] Internet-Draft AI Training Fabric Benchmarking August 2026 [LIBFABRIC] OpenFabrics Interfaces Working Group, "libfabric: Open Fabric Interfaces", n.d., . [LLM-BENCH] Gaikwad, et al, "Benchmarking Methodology for Large Language Model Serving", Work in Progress, Internet-Draft, draft-gaikwad-llm-benchmarking-methodology-00, January 2026, . [META-ROCE] Gangidi, A., Miao, R., Zheng, S., Bondu, S. J., Goes, G., Morsy, H., Puri, R., Riftadi, M., Shetty, A. J., Yang, J., Zhang, S., Fernandez, M. J., Gandham, S., and H. Zeng, "RDMA over Ethernet for Distributed Training at Meta Scale", ACM SIGCOMM '24 Sydney, NSW, Australia, DOI 10.1145/3651890.3672233, 2024, . [MLPERF] MLCommons, "MLPerf Training Benchmark Suite", n.d., . [RFC3849] Huston, G., Lord, A., and P. Smith, "IPv6 Address Prefix Reserved for Documentation", RFC 3849, DOI 10.17487/RFC3849, July 2004, . Appendix A. KPI-to-Test Mapping Summary +===============+=========+====================+====================+ | KPI | Test | Measurement Method | Reporting Unit | | | Section | | | +===============+=========+====================+====================+ | Throughput | Section | Binary search, | Tbps, % line | | Rate | 5.1 | zero-loss | rate | +---------------+---------+--------------------+--------------------+ | Latency (P99) | Section | Tagged frame, | us | | | 5.2 | loaded / unloaded | | +---------------+---------+--------------------+--------------------+ | Burst | Section | Max burst without | frames, bytes | | Absorption | 5.3 | loss | | +---------------+---------+--------------------+--------------------+ | ECN Accuracy | Section | Queue depth vs. | threshold | | | 7.1 | marking | deviation % | +---------------+---------+--------------------+--------------------+ | PFC Behavior | Section | Incast sweep | PAUSE events/ | Calabria, et al. Expires 13 February 2027 [Page 41] Internet-Draft AI Training Fabric Benchmarking August 2026 | | 7.2 | N=2..64 | sec, duration | +---------------+---------+--------------------+--------------------+ | DCQCN | Section | Rate stabilization | us | | Convergence | 7.3 | after onset | | +---------------+---------+--------------------+--------------------+ | PFC Deadlock | Section | Cyclic adversarial | observed/ | | | 7.4 | traffic | reported, | | | | | watchdog events | +---------------+---------+--------------------+--------------------+ | ECMP | Section | MMR, JFI per QP | dimensionless | | Imbalance | 8.1 | count | ratios | +---------------+---------+--------------------+--------------------+ | DLB Efficacy | Section | Throughput delta | %, out-of-order | | | 8.2 | vs. ECMP | rate | +---------------+---------+--------------------+--------------------+ | Spray | Section | JFI, | dimensionless, | | Efficacy | 8.3 | retransmission | retx/sec | | | | rate | | +---------------+---------+--------------------+--------------------+ | AllReduce | Section | CCL benchmark | Gbps per | | BusBW | 9.1 | | accelerator | +---------------+---------+--------------------+--------------------+ | AllToAll JCT | Section | CCL benchmark | seconds per | | | 9.2 | | iteration | +---------------+---------+--------------------+--------------------+ | AllGather | Section | CCL benchmark | Gbps per | | BusBW | 9.3 | | accelerator | +---------------+---------+--------------------+--------------------+ | Synthetic JCT | Section | Measured / | dimensionless | | Ratio | 10.1 | Roofline | | +---------------+---------+--------------------+--------------------+ | MLPerf JCT | Section | Time-to-train | minutes, | | | 10.2 | | tokens/sec | +---------------+---------+--------------------+--------------------+ | Multi-Tenant | Section | Contention / | interference | | Impact | 10.3 | Baseline JCT | factor | +---------------+---------+--------------------+--------------------+ | Scale Limit | Section | Max N with JCT | accelerator | | | 11.1 | Ratio | count | | | | characterized | | +---------------+---------+--------------------+--------------------+ | Failover Time | Section | Loss duration on | us | | | 11.2 | link fail | | +---------------+---------+--------------------+--------------------+ | 24h Stability | Section | JCT Ratio std | dimensionless | | | 12.1 | deviation | | +---------------+---------+--------------------+--------------------+ | UET | Section | Binary search per | Gbps, % line | Calabria, et al. Expires 13 February 2027 [Page 42] Internet-Draft AI Training Fabric Benchmarking August 2026 | Throughput | 6.1 | transport service | rate | | (RUD) | | | | +---------------+---------+--------------------+--------------------+ | UET First- | Section | PDC establish + | us | | Packet | 6.2 | first data | | | Latency | | | | +---------------+---------+--------------------+--------------------+ | UET Spray | Section | JFI/MMR under RUD | dimensionless, | | Efficacy | 6.3 | spray | OOO rate | +---------------+---------+--------------------+--------------------+ | UET PFC-Free | Section | Incast without PFC | %, retx | | Loss Rate | 6.4 | enabled | overhead | +---------------+---------+--------------------+--------------------+ | LLR Retry | Section | Per-hop error | nanoseconds | | Latency | 6.5 | recovery time | | +---------------+---------+--------------------+--------------------+ | Packet | Section | BW saved during | % bandwidth | | Trimming | 6.5 | congestion | | | Savings | | | | +---------------+---------+--------------------+--------------------+ | CBFC vs PFC | Section | Head-of-line | us | | HOL Blocking | 6.5 | blocking duration | | +---------------+---------+--------------------+--------------------+ | UET | Section | AllReduce/AllToAll | Gbps per | | Collective | 6.6 | over UET | accelerator | | BusBW | | | | +---------------+---------+--------------------+--------------------+ | PDC | Section | Sustained PDC | PDCs/second | | Establishment | 6.7 | creation rate | | | Rate | | | | +---------------+---------+--------------------+--------------------+ | Max | Section | Scale limit per | count | | Concurrent | 6.7 | NIC | | | PDCs | | | | +---------------+---------+--------------------+--------------------+ Table 15: KPI-to-Test Mapping Summary Appendix B. Indicative Reference Values (Non-Normative) This appendix provides indicative reference values for the KPIs defined in Section 4. The values reflect current industry observations for distributed AI training workloads as of 2025-2026. These values are NON-NORMATIVE and do not constitute benchmarking acceptance criteria or performance requirements. Per the BMWG charter, the definition of acceptance criteria or performance requirements is explicitly outside the scope of this Working Group. Implementers may use these values as contextual references when Calabria, et al. Expires 13 February 2027 [Page 43] Internet-Draft AI Training Fabric Benchmarking August 2026 interpreting results; they MUST NOT be used as pass/fail criteria in vendor evaluations. Deployment-specific targets will vary by topology, accelerator architecture, collective library, and operator requirements. +============+===================================================+ | KPI | Indicative Reference | +============+===================================================+ | JCT Ratio | ≤ 1.05 (≤ 1.15 acceptable) | +------------+---------------------------------------------------+ | BusBW | ≥ 90% of NIC line rate (intra-pod) | +------------+---------------------------------------------------+ | Aggregate | ≥ 95% of bisection BW | | Throughput | | +------------+---------------------------------------------------+ | Packet | 0 ppm wire-level loss (lossless RoCEv2 profiles); | | Drop Rate | 0 ppm application-visible loss (UET; see the Zero | | | Packet Loss definition in [TERMINOLOGY]) | +------------+---------------------------------------------------+ Table 16: Indicative Reference Values for Distributed AI Training Fabrics (Non- Normative) Appendix C. ASIC Feature Categories (Informational) This appendix identifies ASIC feature categories relevant to AI fabric performance. Implementers document which categories are present and enabled on the DUT. Specific vendor names are intentionally omitted. Calabria, et al. Expires 13 February 2027 [Page 44] Internet-Draft AI Training Fabric Benchmarking August 2026 +==============+================+===============+==================+ | Feature | Sub-types | Relevance to | What to Report | | Category | | AI Fabric | | +==============+================+===============+==================+ | Aggregate | ASIC-level | Cluster | Total Tbps; per- | | Switching BW | capacity | scale, | port speed | | | | bisection BW | (400/800GbE) | +--------------+----------------+---------------+------------------+ | Buffer | Shared, VOQ, | Microburst | Buffer type; | | Architecture | Cut-through | absorption, | total bytes; | | | | PFC behavior, | shared vs. | | | | lossless | dedicated split; | | | | operation | per-port/queue | | | | | allocation | +--------------+----------------+---------------+------------------+ | Packet | Per-flow, Per- | ECMP load | Supported | | Distribution | packet, | balancing | granularities; | | | Flowlet | quality and | in-fabric | | | | reordering | reorder buffer | | | | risk | (yes/no) | +--------------+----------------+---------------+------------------+ | Congestion | ECN marking, | DCQCN | ECN granularity | | Control | PFC, DCQCN | convergence | (port/queue/ | | | | and lossless | VOQ); PFC | | | | behavior | priorities; | | | | | DCQCN parameter | | | | | range | +--------------+----------------+---------------+------------------+ | Adaptive | Flowlet, ECMP, | Load | Algorithm type; | | Routing | Spray, | balancing | flowlet gap | | | Topology-aware | quality under | timer range; | | | | collective | topology-aware | | | | patterns | support | +--------------+----------------+---------------+------------------+ | Telemetry | Per-port, Per- | Required for | Monitoring | | | queue, Per- | KPI | granularity; | | | flow | measurement | streaming | | | | during | interval; INT | | | | benchmarking | support | +--------------+----------------+---------------+------------------+ | Cluster | 2-tier, 3-tier | Applicable | Max cluster size | | Scale | | topology | per topology; | | Support | | scales | ASIC count | +--------------+----------------+---------------+------------------+ Table 17: ASIC Feature Categories Calabria, et al. Expires 13 February 2027 [Page 45] Internet-Draft AI Training Fabric Benchmarking August 2026 All values are reported based on vendor documentation or measured capability. Additional DUT capabilities affecting benchmark results are also documented. Appendix D. RoCEv2 Test Frame Format +========+============+======+=====================================+ | Offset | Field | Size | Value / Description | +========+============+======+=====================================+ | 00 | Ethernet | 6B | DUT next-hop MAC | | | Dst MAC | | | +--------+------------+------+-------------------------------------+ | 06 | Ethernet | 6B | Test equipment MAC | | | Src MAC | | | +--------+------------+------+-------------------------------------+ | 12 | EtherType | 2B | 0x0800 (IPv4) when untagged; 0x8100 | | | / TPID | | (Tag Protocol Identifier — TPID) | | | | | when 802.1Q-tagged | +--------+------------+------+-------------------------------------+ | 14 | 802.1Q Tag | 4B | When tagged: Tag Control | | | (optional) | | Information (TCI: Priority Code | | | | | Point (PCP)=3 for RoCEv2 priority, | | | | | VLAN Identifier (VID)) followed by | | | | | inner EtherType 0x0800. Omit this | | | | | row entirely when untagged and | | | | | shift subsequent offsets back by 4B | +--------+------------+------+-------------------------------------+ | 18 | IPv4 | 20B | DSCP=26 (AF31, Assured Forwarding | | | Header | | class 3, drop precedence 1), | | | | | ECN=ECT(0) (ECN-Capable Transport), | | | | | Proto=17 (UDP) | +--------+------------+------+-------------------------------------+ | 38 | UDP Header | 8B | DstPort=4791 (RoCEv2), SrcPort=var | +--------+------------+------+-------------------------------------+ | 46 | BTH (Base | 12B | OpCode, DstQP, PSN, P_Key | | | Transport | | | | | Header) | | | +--------+------------+------+-------------------------------------+ | 58 | RDMA | 16B | Virtual Address (VA), R_Key, Direct | | | Extended | | Memory Access (DMA) Length | | | Transport | | | | | Header | | | | | (RETH; if | | | | | Write) | | | +--------+------------+------+-------------------------------------+ | 74 | Payload | var | Test data (incrementing octets) | +--------+------------+------+-------------------------------------+ | var | ICRC | 4B | Invariant CRC | Calabria, et al. Expires 13 February 2027 [Page 46] Internet-Draft AI Training Fabric Benchmarking August 2026 +--------+------------+------+-------------------------------------+ | var+4 | FCS | 4B | Ethernet Frame Check Sequence | +--------+------------+------+-------------------------------------+ Table 18: RoCEv2 Test Frame Format Appendix E. UET (Ultra Ethernet Transport) Frame Format UET runs over UDP/IP using UDP destination port 4793, IANA-assigned to Ultra Ethernet Transport. Unlike the RoCEv2 frame format above, this appendix does not specify a byte-accurate UET header layout. UET headers are layered and variable-length, and their wire formats are defined normatively by the UEC; test equipment implementations MUST follow [UEC-1.0] Section 4 for all wire-format details. The figure below is explicitly schematic: it shows only the layering and on-wire ordering of the protocol components that test equipment generates and parses. +-------------------------------------------------------------+ | Ethernet Header: Dst MAC, Src MAC, EtherType | | (optional 802.1Q tag: PCP=3 for UET priority class, VID) | +-------------------------------------------------------------+ | IPv4 Header: DSCP=26 (AF31), ECN=ECT(0), Proto=17 (UDP) | +-------------------------------------------------------------+ | UDP Header: DstPort=4793 (UET); | | SrcPort carries the Entropy Value, varied per packet | +-------------------------------------------------------------+ | PDS (Packet Delivery Sublayer) header(s): | | packet type, PDC identifiers, sequence number, | | ack/credit and congestion-control state | +-------------------------------------------------------------+ | SES (Semantic Sublayer) header(s): | | operation (Write/Read/Send/Atomic), addressing, | | message identification | +-------------------------------------------------------------+ | Payload | +-------------------------------------------------------------+ | ICRC (4B) | FCS (4B) | +-------------------------------------------------------------+ Figure 7: UET Frame Layering (Schematic Only; Byte Layouts Per UEC 1.0 Section 4) Layering notes: Calabria, et al. Expires 13 February 2027 [Page 47] Internet-Draft AI Training Fabric Benchmarking August 2026 * The Entropy Value is carried in the UDP source port field, not in a separate UET header field; see the Entropy Value definition in [TERMINOLOGY]. Test equipment varies the UDP source port per packet to exercise packet spraying. * PDS headers precede SES headers on the wire. * CMS (Congestion Management Sublayer) state is carried in PDS congestion-control fields and control packets rather than as a separate wire header. TSS (Transport Security Sublayer), when enabled, adds security headers and authentication data per [UEC-1.0] Section 4. E.1. Key Differences from RoCEv2 +==========+=================+====================+=================+ |Aspect | RoCEv2 | UET | Notes | +==========+=================+====================+=================+ |UDP Dst | 4791 (IANA- | 4793 (IANA- | Distinct | |Port | assigned) | assigned) | transports, | | | | | both over UDP/ | | | | | IP | +----------+-----------------+--------------------+-----------------+ |Transport | QP Number (24b) | PDC ID | PDC state is | |Endpoint | | | established | | | | | in-band with | | | | | the first | | | | | packet | +----------+-----------------+--------------------+-----------------+ |Entropy | UDP src port, | UDP src port, | Per-packet | |Source | typically fixed | varied per packet | spraying uses | | | per QP/ | | existing ECMP | | | connection | | hashing | +----------+-----------------+--------------------+-----------------+ |Congestion| ECN bits; CNP- | ECN plus UET | Sender- and | |Signalling| based feedback | congestion-control | receiver-based | | | (e.g., DCQCN) | state carried in | CC per | | | | PDS | [UEC-1.0] | +----------+-----------------+--------------------+-----------------+ |Ordering | Always in-order | Per-service | RUD/RUDI allow | |Guarantee | (RC) | (ROD/RUD/RUDI/UUD) | out-of-order | | | | | delivery | +----------+-----------------+--------------------+-----------------+ |Header | Fixed BTH plus | Layered, variable- | Per [UEC-1.0] | |Structure | per-opcode | length (PDS, SES) | Section 4 | | | extension | | | | | headers | | | +----------+-----------------+--------------------+-----------------+ Calabria, et al. Expires 13 February 2027 [Page 48] Internet-Draft AI Training Fabric Benchmarking August 2026 Table 19: RoCEv2 vs. UET Comparison 1. *UDP Destination Port:* UET uses port 4793 vs. RoCEv2 port 4791. 2. *Entropy Value:* Carried in the UDP source port field and varied per packet for ECMP path selection. Test equipment varies the source port to achieve uniform path distribution. 3. *Transport Service Indicator:* Header encodes transport service (ROD/RUD/RUDI/UUD). Tests set this to match the service being benchmarked. 4. *PDC Identifier:* In-band-established PDC ID replaces RoCEv2's Destination QP. Test equipment tracks PDC lifecycle for accurate measurement. 5. *Layered Sub-Headers:* UET uses four sub-layers (SES, PDS, CMS, TSS) with variable-length headers. Implementations MUST follow [UEC-1.0] Section 4 for wire format details. 6. *Optional Feature Headers:* When the LLR or PRI link-layer features, or the network-layer Packet Trimming feature, are enabled, additional or modified framing may be present. Test equipment is configured to recognize and parse these. Acknowledgments This work has benefited from the discussions that occurred during the joint IPPM and BMWG meeting and on the BMWG mailing list. Thanks to Carsten Rossenhoevel and Mohamed Boucadair for valuable review and comments. Thanks to Andrew Yourtchenko for a thorough review of the document set. Thanks to Niangen Ye for the review comments on Fabric-Visible Data Volume provenance and on forwarding-work accounting, which prompted the byte-counting rule stated in Section 3.2.5. Authors' Addresses Fernando Calabria Cisco United States Email: fcalabri@cisco.com Carlos Pignataro Blue Fern Consulting United States Email: carlos@bluefern.consulting Calabria, et al. Expires 13 February 2027 [Page 49] Internet-Draft AI Training Fabric Benchmarking August 2026 Qin Wu Huawei China Email: bill.wu@huawei.com Giuseppe Fioccola Huawei Italy Email: giuseppe.fioccola@huawei.com Sowjanya Reddy Apple United States Email: sowjredd@gmail.com Calabria, et al. Expires 13 February 2027 [Page 50]