AI Unsurprisingly Dominates Hot Chips 2024

This year's edition of the annual Hot Chips conference represented the peak in the generative-AI hype cycle. Consistent with the theme, OpenAI's Trevor Cai made the bull case for AI compute in his keynote. At a conference known for technical disclosures, however, the presentations from merchant chip vendors were disappointing; despite a great lineup of talks, few new details emerged. Nvidia's Blackwell presentation mostly rehashed previously disclosed information. In a picture-is-worth-a-thousand-words moment, however, one slide included the photo of the GB200 NVL36 rack shown below.

GB200 NVL36 rack (Source: Nvidia)

Many customers prefer the NVL36 over the power-hungry NVL72 configuration, which requires a massive 120kW per rack. The key difference for our readers is that the NVLink switch trays shown in the middle of the rack have front-panel cages, whereas the "non-scalable" NVLink switch tray used in the NVL72 has only back-panel connectors for the NVLink spine/backplane. Although Nvidia omitted cabling details, external NVLink cables could include passive (DAC) and active copper (ACC/AEC) as well as pluggable optics. The NVL36 creates a new market segment for cabling, albeit primarily for copper in the near term.

Intel announced its Gaudi 3 AI accelerator back in April, but it's worth reiterating its networking specifications. The new chip integrates 24x200G Ethernet ports using 48x112G SerDes. Like Microsoft's Maia, Gaudi 3 chips connect directly for scale up and through switches for cluster scale out. The integrated NICs support the RoCE protocol for low-latency RDMA data transfers. In a separate talk, Intel once again discussed the 4Tbps Optical Compute Interconnect (OCI) chiplet that we've written about previously. The speaker emphasized the value of laser integration and stressed the 0.1FIT reliability seen with more than 30 million lasers shipped to date.

Broadcom presented its copackaged optics (CPO) technology for AI-compute ASICs with optical attach. Whereas the first part of the presentation mostly covered its Bailly CPO switch, the latter part included some new details and concepts for CPO in XPUs. The company introduced the concept of package "oceanfront," which differs from silicon beachfront. Attaching the optical engines to the CoWoS package substrate provides more linear space for CPO attach than the silicon interposer. The talk also introduced the idea of using bidirectional (bidi) signaling in future optical engines to cut the number of required fibers in half.

Maybe it's appropriate that the merchant chip vendors took a back seat to data-center operators this year. Afterall, it's the latter group driving massive investments in AI infrastructure. Diversity of AI architectures should be good for the supply chain, opening opportunities outside of Nvidia's bespoke full-stack solutions.

LightCounting subscribers can access additional details, figures, and analysis in a full-length research note: www.lightcounting.com/login

Comments

Post a Comment

Popular posts from this blog

AMD Looks to Infinity for AI Interconnects

Image
With the formal launch of the MI300 GPU, AMD revealed new plans for scaling the multi-GPU interconnects vital to AI-training performance. The company's approach relies on a partner ecosystem, which stands in stark contrast with NVIDIA's end-to-end solutions. The plans revolve around AMD's proprietary Infinity Fabric and its underlying XGMI interconnect. Infinity Fabric Adopts Switching As with its prior generation, AMD uses XGMI to connect multiple MI300 GPUs in what it calls a hive. The hive shares a homogeneous memory space formed by the HBM attached to each GPU. In current designs, the GPUs connect directly using XGMI in a mesh or ring topology. Each MI300X GPU has up to seven Infinity Fabric links, each with 16 lanes. The 4th-gen Infinity Fabric supports up to 32Gbps per lane, yielding 128GB/s of bidirectional bandwidth per link. At the MI300 launch, Broadcom announced that its next-generation PCI Express (PCIe) switch chip will add support for XGMI. At last October...
Read more

NVIDIA Networks NVLink

Image
I attended several sessions at last week's Hot Chips, and NVIDIA's NVSwitch talk was a standout. Ashraf Eassa did a great job of covering the talk's contents in an NVIDIA blog, so I will focus on analysis here.  SuperPOD Bids Adieu to InfiniBand From a system-architecture perspective, the biggest change is extending NVLink beyond a single chassis. NVLink Network is a new protocol built on the NVLink4 link layer. It reuses 400G Ethernet cabling to enable passive-copper (DAC), active-copper (AEC), and optical links. The build its DGX H100 SuperPOD, NVIDIA designed a 1U switch system around a pair of NVSwitch chips. The pod ("scalable unit") includes a central rack with 18 NVLink Switch systems, connecting 32 DGX H100 nodes in a two-level fat-tree topology. This pod interconnect yields 460.8Tbps of bisectional bandwidth. NVLink Network replaces InfiniBand (IB) as the first level of interconnect in DGX SuperPODs. The A100 generation pod uses an IB HDR leaf/spine inte...
Read more

Ultra Ethernet Promises New RDMA Protocol

Image
This week saw the formal launch of the Ultra Ethernet Consortium (UEC), which aims to reinvent Ethernet fabrics for massive-scale AI and HPC deployments. An impressive list of founding members back this ambitious effort: hyperscalers Meta and Microsoft; chip vendors AMD, Broadcom, and Intel; OEMs Arista, Atos, and HPE; and Cisco, which straddles the chip and OEM camps. Absent this backing, we could easily write off this consortium as doomed to failure. Our skepticism is rooted not in the obvious need the UEC looks to serve but rather in the challenges of standardizing and implementing a full-stack approach. The effort plans to replace existing transport protocols as well as user-space APIs. Specifically, the Ultra Ethernet Transport (UET) protocol will be a new RDMA protocol to replace ROCE, and new APIs will replace the Verbs API from the InfiniBand heritage. UET will provide an alternative to RoCEv2 and Amazon’s SRD , both of which are deployed in hyperscale data centers. (Source: Ul...
Read more