NoLimit architecture update
As the foundation speeds toward its 1.0 architecture, many details have solidified. We are now ready to share more about what the initial release will include.
NoLimit is more than a single feature or function. The NoLimit Transport (NLT) is central to the work, but the architecture tunes many layers to serve both AI and HPC workloads. This post reviews the advances in software, transport, congestion control, in-network collectives, security, and the link layer.
Software APIs
The software for NLT is based on libfabric v2.0, extended to support the new transport. To make the transition from specification to product smooth, the foundation is developing NLT reference code that runs over standard NICs. The code confirms correct network operation and demonstrates that NLT fits the libfabric APIs well.
The familiar Send, Write, and Read operations are supported, along with rendezvous variants where sender and receiver exchange buffer information first so buffers are guaranteed to be available before transmission. Both tagged and untagged operations are supported. Untagged operations consume buffers sequentially. Tagged operations let partners name a buffer with a tag and skip the address pre-exchange typical of read and write semantics.
NLT also introduces Deferrable Send, an optimistic rendezvous that assumes a buffer is available at the destination and avoids the round trip of checking. If no buffer is available, the receiver resumes the send when one is, reducing reliance on sender timers.
The transport
NLT uses RDMA for its performance, placing data directly from the network into host or accelerator memory and bypassing the kernel for the lowest latency. It takes a clean-slate approach to the RDMA transport protocol to meet the needs of evolving AI and HPC workloads.
NLT removes connection establishment latency and minimizes persistent connection state. State is not required for every possible peer; a pool of connection resources is shared among active endpoints. Connections are ephemeral: no handshake is required before transmission, state is established while communicating, and it is discarded when the transaction ends. That matters for three reasons. Scalability, because storing state in the fast path caps cluster size. Latency, because a handshake before every transmission is costly for inference and HPC. And cost, because large state tables are expensive in silicon area and power.
NLT is designed to work well over best-effort networks that may drop packets. It also runs well over lossless networks using Priority Flow Control. To avoid deadlocks in lossless fabrics and to prioritize latency-sensitive control traffic, NLT uses two traffic classes and maps requests and responses carefully across them.
Three profiles allow optimized implementations. AI Base targets core AI use cases, including distributed inference and training. AI Full adds Deferrable Send, exact-matching tagged sends, and extended atomics. HPC includes advanced tagging semantics, more atomics, and full rendezvous support for workloads beyond AI.
Congestion control
NLT adds a load-balancing mechanism that overcomes ECMP's limits. Senders spray packets across many paths, loading links far more evenly and avoiding flow collisions. They then use lightweight, real-time congestion information to steer the spraying, shrink queues, and keep utilization high. If a path is congested, packets shift immediately to one that is not. Signals come from the whole network: endpoints, transit nodes, and the last hop, which incast hits hardest during collectives and bulk storage traffic.
Two newly designed congestion-control approaches are included. The first is a sender-based scheme built on several existing ideas and tuned for AI and HPC. It is an adaptive window-based scheme in which senders adjust their window based on measured round-trip time, ECN marks, and loss. The second, optional, is receiver-based: a sender asks permission and the receiver grants credits so transmission never overwhelms it. This is especially effective against incast and allows optimistic transmission with a small pre-allocated credit.
Both provide the properties that set NLT apart: ramping from zero to wire rate instantly and backing off fast; responsive behavior in short-RTT, very high-speed environments where a transfer may last only a handful of round trips; efficient multipath management that avoids congested paths; support for optional packet trimming in switches, which gives sender and receiver an early, explicit congestion signal even when spraying reorders packets; and precise flow control within the host, where congestion can occur in the memory subsystem or on the PCIe link between accelerator and NIC.
In-network collectives
The architecture supports in-network collectives, sometimes called switch offload, where collective operations are offloaded to or accelerated in a switch. This is the first such standard over Ethernet. Switch vendors, accelerator designers, system vendors, and hyperscale operators designed it jointly as a lightweight protocol suitable for hardware implementation, mindful of the constraints of a switch moving hundreds of terabits per second across hundreds of interfaces.
Security
Security has been integrated from the start, with authentication, authorization, and confidentiality inside NLT. It borrows from IPsec and PSP, including AES-GCM, key derivation, and replay protection, and differs in two ways: efficient group keying so many endpoints in one security domain (typical of a single job) can communicate securely with minimal state, and independent client-server keys for efficient scaling across trust domains. All of it preserves the ephemeral connection model described above.
Link layer
At the link layer, the architecture standardizes Link Layer Retry. AI servers use extremely high bandwidth density over many links, and in a parallel job any underperforming link slows the whole group. LLR improves the reliability of marginal links affected by transient physical disruption, intermittent components, or faulty wiring, protecting tail latency across tens or hundreds of thousands of accelerators. It is negotiated hop by hop via LLDP extensions; each packet is held at the sender until the receiver acknowledges it.
Watch this space
Drawing on more than 90 member companies, the foundation is working aggressively to finalize the initial architecture and carry Ethernet forward as the premier network for AI and HPC. Further work in storage, performance and debugging, compliance, and management is underway.