NoLimit progresses toward its 1.0 architecture
The foundation continues to make progress on its 1.0 set of specifications. Together they give implementers a recipe for deploying the latest data center network innovations in service of AI and HPC workloads. Because we preserve existing AI frameworks and HPC library APIs, we expect existing workloads to move onto NoLimit networks with no changes required.
The core idea is to modernize RDMA over Ethernet. The NoLimit Transport (NLT) does this through a set of innovations that raise network utilization and lower tail latency, both of which are critical to shorter AI and HPC job completion times. As models and workloads grow, clusters grow with them, and the network increasingly becomes the limiting factor while its share of power consumption and total cost creeps upward. A modest investment in this networking technology pays back quickly.
The stack
NoLimit follows the familiar layered approach: broadly deployed AI frameworks and HPC libraries, through libfabric and the collective communication libraries, to NLT, to Ethernet. Every networking engineer knows this shape, and it preserves all of Ethernet's software and tooling. The APIs promote interoperability while leaving room for vendor differentiation. Choosing libfabric as the northbound API means the stack drops into frameworks where libfabric is already common. Two points are worth noting: RDMA operations are optimized to match workload expectations, minimize hardware complexity, and reduce state, and NLT provides multiple transport services that augment RDMA hardware.
Key features
- Multipath packet spraying
- Flexible ordering
- State-of-the-art, easily configured congestion control
- End-to-end telemetry
- Multiple transport delivery services
- Switch offload, also known as in-network collectives
- Security as a first-class citizen, co-designed with the transport
- Optional Ethernet link and physical layer enhancements
Multipath packet spraying. Every flow may use all paths to its destination simultaneously. Rather than relying on a simple ECMP hash to pick links, imbalances are avoided by a coordinated choice of paths between endpoints and switches, guided by real-time congestion information. The result is better utilization and lower tail latency.
Flexible ordering. The libfabric API lets an application state its ordering requirements for messages and packets, which steers NLT's choice of transport service. A large AI collective can be treated as a bulk transfer where the application only cares when the last piece lands. Multipathing causes out-of-order arrival, and the reorder buffers other technologies use to compensate add cost, latency, complexity, and power. NLT couples multipathing, flexible ordering, and congestion control so the receiver does not need a reorder buffer. Per-packet direct data placement, or zero copy, is one of the mechanisms that makes this efficient.
Congestion handling. Large AI clusters and HPC networks create traffic patterns that traditional data center congestion handling was not built for, especially as accelerator bandwidth climbs. Microsecond round trips, parallel applications, and systolic, coordinated multiflow traffic call for a sender-based approach. For collectives like AllReduce and All-to-All, coordinated congestion control across paths is what guides spraying. All-to-All can also produce incast at the last switch before a receiver, so NLT defines an optional receiver-based scheme that grants credits to senders to augment the base mechanism. NLT also performs well when some traffic is sprayed and some is not, which happens when single-path and multipath services mix.
Telemetry. Explicit Congestion Notification is broadly supported by data center switches and is now assisted by NLT's endpoint mitigation algorithms. NLT adds optional switch-based advanced telemetry that shortens signaling time so short congestion events can be sensed and answered quickly. One example is packet trimming, where a compliant switch truncates a congested packet rather than dropping it and forwards the header plus congestion information to the receiver. Paired with selective acknowledgments instead of heavyweight go-back-N, the response to congestion and loss is faster and far more bandwidth efficient.
Transport delivery services. Reliable Ordered Delivery for applications that need message order. Reliable Unordered Delivery for operations that must arrive exactly once but can tolerate reordering, enabling spraying without a reorder buffer. Reliable Unordered Delivery for Idempotent operations for the very largest scales, where a packet may be delivered more than once and the receiver keeps minimal state.
Switch offload. The architecture defines how collective operations such as AllReduce can be offloaded to the network, improving bandwidth utilization and shortening the collectives that gate the next compute phase of many jobs. It is optional to implement, and when 1.0 lands it will be the first time this has been standardized over Ethernet.
Security. Security is co-designed with the transport to cover integrity, confidentiality, and replay prevention efficiently. At the scale and speed we expect, key management is the hard part, so particular attention goes to overhead and to efficient group keying. With hundreds of thousands of endpoints and 1.6 Tb/s links imminent, that is a requirement, not a nicety.
Profiles. AI and HPC are converging but still differ. Most AI workloads produce large, bandwidth-sensitive messages. Many HPC workloads depend on ordering, produce short messages, and are latency-sensitive. Profiles bundle transport services, libfabric APIs, and required features to fit each, allowing product flexibility and vendor differentiation while keeping interoperability.
Link layer and lossless networks. The architecture is compliant and interoperable with existing Ethernet switches, with optional extensions such as Link Level Reliability for fast hardware reaction when a marginal link would otherwise slow an entire parallel job. NLT performs well on lossy networks and is also architected for lossless ones, making them easier to tune and avoiding the congestion spreading that has long been a concern.
What makes a NoLimit device
Consistent with the trend of the last two decades, the key innovations sit at the endpoints. A compliant network must have endpoints that support NLT under one of the profiles. That brings together simplified RDMA, multipathing with zero copy, higher utilization, and lower tail latency, all of which add up to the lowest AI and HPC job completion times. NLT runs over existing switches and performs even better where advanced telemetry and the enhanced link and software layers are present.
The growing community
As of this writing the foundation has 55 member companies and more than 750 individuals participating in eight working groups: Transport, Software, Link Layer, Physical Layer, Storage, Management, Performance and Debug, and Compliance. We are open to new members who want to contribute. Watch this site for news on the 1.0 architecture later this year.