Overview

Today’s mainstream AI models contain billions of parameters, demanding tremendous compute power, memory, and system connectivity for training and deployment. Moore’s Law no longer keeps pace with this exponential growth. To address this challenge, enterprises adopt AI the NVIDIA way: scaling up and out diverse AI and data analytics workloads using NVIDIA® GPU computing. Central to every NVIDIA® DGX™ and NVIDIA® HGX™ system are GPUs connected via NVIDIA NVLink™. This high-speed interconnect enables GPUs to access each other’s memory at NVLink speeds, delivering exceptional bandwidth and low latency.

Over time, NVLink has become the de facto high-speed, direct GPU-to-GPU interconnect for single-node, multi-GPU systems. Many of these systems are further connected through high-speed networks like InfiniBand to build large-scale supercomputers. With an NVLink Network (also known as NVLink Multi-Node), these supercomputers are interconnected using NVLink, effectively forming a massive accelerator with shared memory.

Definitions, Acronyms, and Abbreviations

Abbreviations	Definitions
Node or Compute Node	An OS instance with at least one GPU.
NVLink Domain or cluster	A set of nodes that can communicate over NVLink.
FM	Fabric Manager. NVLink Network control plane service is provided by the FM service
NVLSM	NVLink Subnet Manager A service that originates from NVIDIA InfiniBand switches and has the necessary modifications to effectively manage NVSwitches.
IMEX Domain	A set of compute nodes connected by NVLink on which the nvidia-imex service has been installed and configured to communicate with each other via the nodes_config.cfg

NVIDIA NVLink Network Servers

This section provides information about the NVLink network servers.

NVIDIA GB200 NVL72

The NVIDIA GB200 NVL72 rack scale architecture is an Open Compute Project (OCP) Open Rack v3 (ORv3) rack with two configurations that support up 72-GPUs . The NVIDIA reference design provides options for 36-GPUs in a single rack, 72-GPUs in two racks, or 72-GPUs in a single rack.

A 36-GPU rack contains the following:

• 9x 2RU compute trays, each with 2 Grace CPUs and 4 Blackwell GPUs.

• 9x 1RU NVLink switch trays with 18 OSFP front panel interfaces.

• TOR Switches for management.

• Power shelves for supplying power to all trays and switches.

Figure 1‑2 shows the configuration of a single rack 36-GPU configuration and a dual rack 72-GPU configuration

Figure 1‑2. GB200 NVLink Rack Configuration

A 72-GPU rack contains the following:

• 18x 1RU compute trays, each with two Grace CPUs and four Blackwell GPUs.

• 9x 1RU NVLink switch trays (with no OSFP front-panel interfaces).

• TOR Switches for management.

• Power shelves to supply power to all trays and switches.

Figure 1‑4 shows the configuration of a single-rack, 72-GPU configuration.

Figure 1‑4. A Single-Rack, 72-GPU Configuration

The NVIDIA Import/Export Service for Internode Memory Sharing

The IMEX service supports GPU memory export and import (NVLink P2P) and shared memory operations across OS domains in an NVLink multi-node deployment.

Multi-Node Memory Sharing Model

This section provides information about the multi-node memory sharing model.

Figure ‑. Multi-Node Memory Sharing Model

At a high level, the job entails coordinating CUDA processes that are running on each compute node in its own OS domain. On one node, the CUDA process allocates GPU memory and obtains the corresponding sharable memory handle. This allocation and handle creation trigger the establishment of a Virtual Address (VA) to Physical Address (PA) to Fabric Address (FA) mapping on the exporting node. Here is an overview of the process:

1. For importing, the process shares the memory handle with the coordinating process on other nodes using MPI/NCCL.

2. On these nodes, the CUDA processes prompt the GPU driver to import the relevant memory using the received handle.

3. The GPU driver on the importing node creates the requisite memory objects and establishes the VA-to-FA mapping.

4. When the importing process accesses the VA address space, the GPU memory system on the importing nodes recognizes the corresponding physical page tables by leveraging a foreign GPU and generates NVLink packets to access the pertinent memory.

To facilitate the importing node’s establishment of the VA-to-FA mapping, a privileged entity capable of communicating across OS/Node domains is essential to retrieve the necessary memory mapping information from the exporting node. Operating at a higher level, the IMEX service serves this function by acting as an orchestrator for memory export and import across compute nodes.

Here are some key features of the IMEX service:

• Facilitates memory sharing between compute nodes.

• Manages the life cycle of the shared memory.

• Registers for memory import/unimport events with the GPU Driver.

• Does not directly communicate with CUDA or user applications.

• Communicates across nodes using the compute node’s network by using TCP/IP and gRPC connections.

• Runs exclusively on compute nodes.

This guide provides an overview of various IMEX features and is intended for multi-node system administrators.

Note

Before launching jobs in an NVLink multi-node cluster, start the IMEX service. If the IMEX service is not properly initialized, NVLink multi-node jobs will fail.