GPUDirect RDMA (Remote Direct Memory Access) is a technology that enables a direct path for data exchange between the GPU and a third-party peer device using standard features of PCI Express.

The NVIDIA GPU driver package provides a kernel module, nvidia-peermem.ko, which provides Mellanox InfiniBand based HCAs (Host Channel Adapters) direct peer-to-peer read and write access to the NVIDIA GPU's video memory. It allows GPUDirect RDMA-based applications to use GPU computing power with the RDMA interconnect without needing to copy data to host memory.

This capability is supported with Mellanox ConnectX-3 VPI or newer adapters. It works with both InfiniBand and RoCE (RDMA over Converged Ethernet) technologies.

Mellanox OFED (Open Fabrics Enterprise Distribution) or MOFED, introduces an API between the InfiniBand Core and peer memory clients such as NVIDIA GPUs, called PeerDirect, see https://community.mellanox.com/s/article/howto-implement-peerdirect-client-using-mlnx-ofed.

The nvidia-peermem.ko module registers the NVIDIA GPU with the InfiniBand subsystem by using peer-to-peer APIs provided by the NVIDIA GPU driver.

This module, originally maintained by Mellanox on GitHub, is now included with the NVIDIA Linux GPU driver. The original GitHub project at https://github.com/Mellanox/nv_peer_memory should be considered deprecated and only critical bugs will be addressed for existing installations.

The kernel must have the required support for RDMA peer memory either through additional patches to the kernel or via Mellanox OFED package (MOFED) as a prerequisite for loading and using nvidia-peermem.ko.

It is possible that the nv_peer_mem module from the GitHub project may be installed and loaded on the system. Installation of nvidia-peermem.ko will not affect the functionality of the existing nv_peer_mem module. But, to load and use nvidia-peermem.ko, users must disable the nv_peer_mem service. Additionally, it is encouraged to uninstall the nv_peer_mem package to avoid any conflict with nvidia-peermem.ko since only one module can be loaded at any time.

Stop the nv_peer_mem service:

    # service nv_peer_mem stop

Check if nv_peer_mem.ko is still loaded after stopping the service:

    # lsmod | grep nv_peer_mem

If nv_peer_mem.ko is still loaded, unload it with:

    # rmmod nv_peer_mem

Uninstall nv_peer_mem package:

For DEB based OS:

    # dpkg -P nvidia-peer-memory

    # dpkg -P nvidia-peer-memory-dkms

For RPM based OS:

    # rpm -e nvidia_peer_memory

After ensuring kernel support and installing the GPU driver, nvidia-peermem.ko can be loaded with the following command with root privileges in a terminal window:

    # modprobe nvidia-peermem

Note: If the NVIDIA GPU driver is installed before MOFED, the GPU driver must be uninstalled and installed again to make sure nvidia-peermem.ko is compiled with the RDMA APIs that are provided by MOFED.

peerdirect_support: this parameter takes the following integer values:

As a reference, in the legacy PeerDirect APIs, the peer_memory_client structure declared in peer_mem.h has the two extra function pointers shown below:

  void* (*get_context_private_data)(u64 peer_id);
  void  (*put_context_private_data)(void *context);

Note that MOFED LTS as well as MOFED 5.0 and previous releases ship with legacy PeerDirect APIs. So for example, when using MOFED LTS, GPUDirect RDMA support for the Mellanox HCAs will not work correctly unless peerdirect_support is set to one.

Instead for MOFED 5.1 or newer, the default value of zero is appropriate, so no special actions are needed.