| Internet-Draft | FARE in SUN | June 2026 |
| Xu, et al. | Expires 12 December 2026 | [Page] |
FARE‑BGP enables weighted ECMP load balancing using a path‑bandwidth extended community. FARE‑in‑SUN extends this mechanism from switches to GPUs for scale‑up networks, which are typically multi‑plane. Large AI training clusters increasingly adopt multi‑plane scale-out network topologies. This document further extends FARE‑BGP from switches to RoCE NICs (RNICs) for such multi‑plane scale‑out networks. The document also presents two techniques to address route scalability concerns caused by the injection of numerous host routes.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 12 December 2026.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Large AI training clusters (beyond 100,000 GPUs) increasingly use multi‑plane scale‑out network topologies (see below) to reduce the total number of switches and links. In such a topology, a high‑speed RNIC is split into multiple lower‑speed lanes, each connected to an independent CLOS fabric (a “plane”). Because there are no links between planes, the RNIC itself must decide which plane to use for each packet or flow. In other words, the RNIC must know the reachability of each plane and then perform global load balancing across planes.¶
=========================================
# +----+ +----+ +----+ +----+ #
# | S1 | | S2 | | S3 | | S4 | (Spine) #
# +----+ +----+ +----+ +----+ #
# Plane-1 #
# +----+ +----+ +----+ +----+ #
# | L1 | | L2 | | L3 | | L4 | (Leaf) #
# +----+ +----+ +----+ +----+ #
=========================================
=================================== ===================================
# +-----+ +-----+ +-----+ +-----+ # # +-----+ +-----+ +-----+ +-----+ #
# |RNIC1| |RNIC2| |RNIC3| |RNIC4| # # |RNIC1| |RNIC2| |RNIC3| |RNIC4| #
# +-----+ +-----+ +-----+ +-----+ # # +-----+ +-----+ +-----+ +-----+ #
# Server-1 # # Server-n #
#================================== ... ===================================
=========================================
# +----+ +----+ +----+ +----+ #
# | L1 | | L2 | | L3 | | L4 | (Leaf) #
# +----+ +----+ +----+ +----+ #
# Plane-2 #
# +----+ +----+ +----+ +----+ #
# | S1 | | S2 | | S3 | | S4 | (Spine) #
# +----+ +----+ +----+ +----+ #
=========================================
Figure 1
¶
(For simplicity, the diagram above omits the connections between RNICs and leaf switches. In practice, each RNIC is multi‑homed to one leaf switch in every plane.)¶
FARE‑in‑SUN [I-D.xu-rtgwg-fare-in-sun] describes how to extend the FARE‑BGP protocol [I-D.xu-idr-fare] from switches to GPUs for scale‑up networks. Because scale‑up shares the same multi‑plane architectural pattern as multi-plane scale-out networks, the adaptive routing approach defined in FARE‑in‑SUN can be applied directly to multi‑plane scale‑out networks.¶
The solution described in this document is almost identical to FARE‑in‑SUN, with the following two essential differences. First, FARE‑BGP is extended from switches to RNICs rather than to GPUs. Second, in a scale‑up network, the number of route entries is small (typically a few hundred) and can be installed directly on GPUs. In an isolated multi‑plane scale‑out network with 100,000 GPUs and four planes, each plane may propagate up to 100,000 host routes – a total of 400,000 routes. Storing all these routes on an RNIC is impractical. Therefore, the RNIC must suppress the routing table using the techniques described in Section 4.¶
This document describes how to extend the Fully Adaptive Routing Ethernet (FARE) using BGP (FARE-BGP in short) as described in , which was originally designed for scale-out netowrks, to scale-up networks.¶
In an isolated multi‑plane scale‑out network, an RNIC connects to each plane and is configured as a stub BGP speaker per plane. It establishes separate BGP sessions with the attached leaf switches of each plane. The BGP neighbor discovery [I-D.xu-idr-neighbor-autodiscovery] can be used to simplify configuration.¶
Through these sessions, the RNIC learns routes to remote GPUs together with the path‑bandwidth extended community. Because the RNIC participates in BGP with each plane independently, it aggregates per‑plane path‑bandwidth information and performs weighted load balancing across planes. The RNIC thus performs the same Weighted Equal‑Cost Multi‑Path (WECMP) functions as a FARE‑capable switch, distributing traffic in proportion to the path bandwidth of each ECMP route.¶
Two modes of WECMP are supported:¶
Per‑flow WECMP (for RNICs that cannot handle disordered packet delivery): The RNIC establishes at least one QP per plane. The number of QPs allocated to a plane is proportional to the plane’s weight. All packets of a given flow go through the same plane, preserving order.¶
Per‑packet WECMP (for RNICs that support out‑of‑order packet delivery): A single QP per (source, destination) RNIC pair suffices. The RNIC sprays each packet of that QP across all available planes according to the weights.¶
In an isolated multi‑plane scale‑out network with 100,000 GPUs and four planes, each plane may propagate up to 100,000 host routes – a total of 400,000 routes. Storing all these routes on an RNIC is impractical. Two complementary approaches can reduce the number of routes the RNIC must store.¶
It's straightfoward to resort to route aggregation mechanism, i.e., aggregating host routes when advertising them from leaf to spine. However, naive aggregation can cause route blackholes: if a specific host within an aggregate becomes unreachable, the aggregated route still points to that plane. Consequently, traffic destined for that host will still be forwarded according to the aggregated route and then dropped.¶
To address this issue, the switches MUST explicitly advertises unreachable host routes for a given RNIC to the other RNICs. When a RNIC becomes unreachable via a particular plane, the leaf switch advertises this unreachability to the RNIC using one of two methods:¶
Path bandwidth value of 0: The leaf switch advertises the host route (NLRI) with the BGP path‑bandwidth extended community set to 0. The RNIC interprets this as “unreachable” and excludes that plane from the next‑hop set for that destination.¶
Specific BGP unreachability advertisement: The leaf switch sends a dedicated BGP unreachability message. This is distinct from a standard BGP route withdrawal. It explicitly marks the host as unreachable via that plane while keeping the aggregated route intact.¶
Upon receiving such an advertisement, the RNIC updates its forwarding table as follows:¶
It locates the longest‑matching aggregated route that covers the unreachable host (e.g., a default route or a supernet prefix).¶
From that aggregated route’s set of next‑hops (which originally included multiple planes), it removes the next‑hop corresponding to the plane where the host is unreachable.¶
It then installs a host‑specific route for the unreachable destination, with the remaining next‑hops from the aggregated route.¶
Example: Suppose an RNIC has a default route (0.0.0.0/0) with next‑hops pointing to planes A, B, C, and D. Host X (a specific /32) becomes unreachable via plane A. The RNIC learns an unreachable advertisement for X. It then creates a host route for X with next‑hops set to {B, C, D} – i.e., the original aggregated next‑hops minus the next‑hop associated with plane A. Traffic to X will never be sent to plane A, avoiding blackholes.¶
This technique dramatically reduces BGP table size on the RNIC: the RNIC only needs to store aggregated routes (e.g., a handful of default routes per plane) plus explicit unreachable host routes for the small number of hosts that are actually unreachable. The majority of reachable hosts are covered by aggregates and require no per‑host state. The approach is especially effective when unreachability is rare, which is typical in well‑managed clusters.¶
Switches within each plane does not need to install the unreachable host route into their FIB tables.¶
Since a given RNIC communicates only with a limited subset of GPUs (due to AI training parallelism patterns), it’s possible for the RNIC to filter routes to retain only those it actually needs.¶
The RNIC sends Address Prefix ORF entries to its BGP peer (leaf switch) per plane. These entries indicate the host routes for remote RNICs the local RNIC is interested in. The peer filters outbound route updates accordingly, sending only the requested routes. In this way, the RNIC stores only a limited number of routes.¶
For switches, there is no need install host routes for remote RNICs. Therefore, the FIB-suppression mechanism as described in Virtual Aggregation Auto-configuration [I-D.ietf-grow-va-auto] could be reused.¶
TBD.¶
TBD.¶