Cisco Joins 51.2T Switch-Chip Crowd

Four chip vendors may not sound like a crowd, but in the leading-edge data-center switch segment, it's likely unsustainable over the long term. The problem is the small number of customers for these devices, that is, the hyperscalers. Despite stiff competition, however, Cisco continues to invest in its Silicon One product line, which now includes 5nm switch chips. The new top-end device is the Silicon One G200, a 51.2Tbps chip built around an internally developed 112Gbps serdes. As a refresher, the company announced production of its 7nm 25.6Tbps G100 chip last October along with design wins in new Cisco platforms.

As the figure below shows, Cisco makes some bold claims regarding the G200. The most startling is the statement that the G200 is twice as power efficient as its predecessor. In other words, the G200 dissipates the same power as the G100 at twice the throughput. The company's new serdes design must improve power efficiency, as the move from 7nm to 5nm should account for a 30-40% reduction. Cisco says it began internal serdes development seven years ago and that the G200 is its second-generation design. It claims the DSP-based 112Gbps serdes can drive a 4m DAC cable, linear-drive optics, and backplane channels, matching the features of Broadcom's serdes integrated in Tomahawk5.

Silicon One G200 Features (Source: Cisco)

Cisco also says the G200 halves latency compared with the G100, but it withheld the latter's specifications. The new design preserves P4 software compatibility while apparently doubling the packet engine's speed. Although the headline switch density is 64x800GbE ports, the G200 also handles a full 512x100GbE ports for maximum radix. The company also highlighted the G200's lookup rate of 435 billion lookups per second as enabling SRv6 micro segments (uSID), which represent segments using only two bytes.

Coincidentally, the G200 analyst briefing happened the same day the Ultra Ethernet Consortium (UEC) launched, and Cisco carved out a new "enhanced Ethernet" middle ground between plain-vanilla Ethernet and fully-scheduled fabrics. Specifically, it placed congestion-aware load balancing techniques in this middle category, which at present uses proprietary protocols to communicate congestion data between switch nodes. In a simulated 2,048-GPU cluster running 128 AI jobs, the enhanced-Ethernet load balancing delivered a job-completion time only 29% greater than that of a fully-scheduled fabric. The UEC plans to standardize congestion management, and Cisco is a founding member.

Reviewing the landscape, the G200 joins three previously announced 51.2T switch chips. Broadcom was first to production with Tomahawk5, followed by NVIDIA with Spectrum-4. Cisco is sampling the G200, which should reach production by mid-2024. Marvell preannounced Teralynx 10 and has not updated its status, but we expect that chip to reach production about the same time as the G200. We note that all four 51.2T designs rely on serdes designed by the respective vendors, removing dependency on third-party serdes IP that torpedoed multiple 25.6T designs.

400/800G Ethernet Switch Chips

Aside from time to market, business models differentiate these vendors. Broadcom and Marvell remain solely chip vendors, with ODMs and OEMs building systems based on their chips. NVIDIA favors vertical integration, selling full-stack solutions including its Spectrum-X suite. Although it has a few chip-level design wins, it primarily ships branded system-level products. When Cisco launched its Silicon One line in late 2019, it loudly announced newfound business-model flexibility. The company would ship a chip, a white box, or a fully-integrated system, whatever the customer desired. Now, Cisco says it has wins at five out of six "webscale" companies, including wins using each of the three business models. With its webscale business achieving a multibillion-dollar run rate, it can justify investing in leading-edge silicon like the G200.

Comments

Post a Comment

Popular posts from this blog

AMD Looks to Infinity for AI Interconnects

Image
With the formal launch of the MI300 GPU, AMD revealed new plans for scaling the multi-GPU interconnects vital to AI-training performance. The company's approach relies on a partner ecosystem, which stands in stark contrast with NVIDIA's end-to-end solutions. The plans revolve around AMD's proprietary Infinity Fabric and its underlying XGMI interconnect. Infinity Fabric Adopts Switching As with its prior generation, AMD uses XGMI to connect multiple MI300 GPUs in what it calls a hive. The hive shares a homogeneous memory space formed by the HBM attached to each GPU. In current designs, the GPUs connect directly using XGMI in a mesh or ring topology. Each MI300X GPU has up to seven Infinity Fabric links, each with 16 lanes. The 4th-gen Infinity Fabric supports up to 32Gbps per lane, yielding 128GB/s of bidirectional bandwidth per link. At the MI300 launch, Broadcom announced that its next-generation PCI Express (PCIe) switch chip will add support for XGMI. At last October'
Read more

NVIDIA Networks NVLink

Image
I attended several sessions at last week's Hot Chips, and NVIDIA's NVSwitch talk was a standout. Ashraf Eassa did a great job of covering the talk's contents in an NVIDIA blog, so I will focus on analysis here.  SuperPOD Bids Adieu to InfiniBand From a system-architecture perspective, the biggest change is extending NVLink beyond a single chassis. NVLink Network is a new protocol built on the NVLink4 link layer. It reuses 400G Ethernet cabling to enable passive-copper (DAC), active-copper (AEC), and optical links. The build its DGX H100 SuperPOD, NVIDIA designed a 1U switch system around a pair of NVSwitch chips. The pod ("scalable unit") includes a central rack with 18 NVLink Switch systems, connecting 32 DGX H100 nodes in a two-level fat-tree topology. This pod interconnect yields 460.8Tbps of bisectional bandwidth. NVLink Network replaces InfiniBand (IB) as the first level of interconnect in DGX SuperPODs. The A100 generation pod uses an IB HDR leaf/spine inte
Read more

NVIDIA Reveals DGX GH200 System Architecture

Image
We industry analysts sometimes get out over our skis when trying to project details of new products. Following NVIDIA's DGX GH200 announcement at Computex, we noticed industry press making the same mistake. Rather than correct our previous NVLink Network post, we'll explain here what we've since learned. Our revelation came when NVIDIA published a white paper titled NVIDIA Grace Hopper Superchip Architecture . The figure below from that paper shows the interconnects at the HGX-module level. The GPU (Hopper) side uses NVLink as a coherent interconnect, whereas the CPU (Grace) side uses InfiniBand, in this case connected through a Bluefield-3 DPU. From a networking perspective, the NVLink and InfiniBand domains are independent, that is, there is no bridging between the two protocols. HGX Grace Hopper Superchip System With NVLink Switch (Source: NVIDIA) The new DGX GH200 builds a SuperPOD based on this underlying module-level architecture. You've probably seen the headlin
Read more