Spectrum-X: It's Bigger Than Software

There's been a lot of confusion around Spectrum-X, some of which NVIDIA seems to have created intentionally. The company's branding is part of the issue, as it seems to conflate Spectrum-X with the Spectrum line of Ethernet switch chips. In fact, Spectrum-X is simply a software license that enables new features across a set of existing hardware products.

The reality that Spectrum-X is a set of software, however, devalues what NVIDIA has actually delivered. Working on top of the company's end-to-end Ethernet hardware, the software creates the first merchant congestion-managed Ethernet fabric. Minimizing tail latency is critical to AI-training workloads, as detailed in our recent white paper. We use the merchant qualifier because some hyperscalers have developed their own congestion-management schemes that work with standard Ethernet-switch hardware. One example is Amazon, which developed the scalable reliable datagram (SDP) protocol for use with its internally-developed Nitro smart-NIC along with merchant switch silicon.

Spectrum-X (Source: NVIDIA)

Spectrum-X comprises software for BlueField-3 DPUs, residing in servers, and Ethernet switches based on Spectrum-4. The Spectrum-X announcement included the news that the Spectrum-4 51.2Tbps switch chip is now in production and available in new SN5000 Series switch systems. That follows BlueField-3 production availability, which NVIDIA revealed at GTC in March. At its core, Spectrum-X implements ROCEv2 adaptive routing and congestion management to create a lossless Ethernet fabric with predictable and bounded latency. As is the case with Amazon's SDP, adaptive routing enables optimal path selection on a per-packet basis, also known as packet spraying. This can result in out-of-order packets at the destination, which the DPU in the endpoint must handle. On the transmit side, the DPU can also limit the packet-injection rate in response to congestion notifications. Spectrum-4 provides realtime telemetry data (branded NetQ), such as queue depths.

NVIDIA's full-stack solution provides an Ethernet-based alternative to InfiniBand for AI training. The company positions Spectrum-X for multitenant clouds where AI isn't the only workload, whereas InfiniBand provides ultimate scaling for "AI factories." In other words, you won't be using Spectrum-X to train the next ChatGPT, which is the domain of the DGX GH200. Spectrum-X is also NVIDIA's answer to Ethernet scheduled fabrics available from Broadcom and Cisco. In Cisco's parlance, Spectrum-X is a telemetry-assisted Ethernet solution, which also describes Amazon's implementation. Fully-scheduled fabrics instead handle congestion management in the fabric and present standard Ethernet ports at ingress/egress. This approach enables servers to use standard NICs, such as NVIDIA's ConnectX-7, rather than smart-NICs (or DPUs). Although the scheduled fabric is vendor-specific (or proprietary), it decouples the switch fabric from the NIC.

We might think Spectrum-X is simply a defensive response to Ethernet competitors, aimed at protecting NVIDIA's sizable InfiniBand business. But the company is building what it says will be Israel's most powerful supercomputer, dubbed Israel-1, around the Spectrum-X fabric. Hyperscalers including Amazon and Meta are already using Ethernet fabrics in AI systems, but Spectrum-X will serve the next tier of cloud providers. In the race to build AI infrastructure, there's no one-size-fits-all network architecture, meaning neither NVIDIA's InfiniBand nor Broadcom's Tomahawk will be the dominant solution.

Comments

Post a Comment

Popular posts from this blog

AMD Looks to Infinity for AI Interconnects

Image
With the formal launch of the MI300 GPU, AMD revealed new plans for scaling the multi-GPU interconnects vital to AI-training performance. The company's approach relies on a partner ecosystem, which stands in stark contrast with NVIDIA's end-to-end solutions. The plans revolve around AMD's proprietary Infinity Fabric and its underlying XGMI interconnect. Infinity Fabric Adopts Switching As with its prior generation, AMD uses XGMI to connect multiple MI300 GPUs in what it calls a hive. The hive shares a homogeneous memory space formed by the HBM attached to each GPU. In current designs, the GPUs connect directly using XGMI in a mesh or ring topology. Each MI300X GPU has up to seven Infinity Fabric links, each with 16 lanes. The 4th-gen Infinity Fabric supports up to 32Gbps per lane, yielding 128GB/s of bidirectional bandwidth per link. At the MI300 launch, Broadcom announced that its next-generation PCI Express (PCIe) switch chip will add support for XGMI. At last October'
Read more

NVIDIA Networks NVLink

Image
I attended several sessions at last week's Hot Chips, and NVIDIA's NVSwitch talk was a standout. Ashraf Eassa did a great job of covering the talk's contents in an NVIDIA blog, so I will focus on analysis here.  SuperPOD Bids Adieu to InfiniBand From a system-architecture perspective, the biggest change is extending NVLink beyond a single chassis. NVLink Network is a new protocol built on the NVLink4 link layer. It reuses 400G Ethernet cabling to enable passive-copper (DAC), active-copper (AEC), and optical links. The build its DGX H100 SuperPOD, NVIDIA designed a 1U switch system around a pair of NVSwitch chips. The pod ("scalable unit") includes a central rack with 18 NVLink Switch systems, connecting 32 DGX H100 nodes in a two-level fat-tree topology. This pod interconnect yields 460.8Tbps of bisectional bandwidth. NVLink Network replaces InfiniBand (IB) as the first level of interconnect in DGX SuperPODs. The A100 generation pod uses an IB HDR leaf/spine inte
Read more

NVIDIA Reveals DGX GH200 System Architecture

Image
We industry analysts sometimes get out over our skis when trying to project details of new products. Following NVIDIA's DGX GH200 announcement at Computex, we noticed industry press making the same mistake. Rather than correct our previous NVLink Network post, we'll explain here what we've since learned. Our revelation came when NVIDIA published a white paper titled NVIDIA Grace Hopper Superchip Architecture . The figure below from that paper shows the interconnects at the HGX-module level. The GPU (Hopper) side uses NVLink as a coherent interconnect, whereas the CPU (Grace) side uses InfiniBand, in this case connected through a Bluefield-3 DPU. From a networking perspective, the NVLink and InfiniBand domains are independent, that is, there is no bridging between the two protocols. HGX Grace Hopper Superchip System With NVLink Switch (Source: NVIDIA) The new DGX GH200 builds a SuperPOD based on this underlying module-level architecture. You've probably seen the headlin
Read more