Skip to content

Instantly share code, notes, and snippets.

@1duo

1duo/openmpi.use.gpudirect.rdma.md

Last active May 13, 2022 08:33
Show Gist options
  • Save 1duo/be25fdabfd120bef79bac915b0688548 to your computer and use it in GitHub Desktop.
Save 1duo/be25fdabfd120bef79bac915b0688548 to your computer and use it in GitHub Desktop.
Download ZIP
Use OpenMPI + GPUDirect RDMA for better performance.
Raw
openmpi.use.gpudirect.rdma.md

  • Install OpenMPI, make sure enable CUDA support by adding --with-cuda flag.

  • Verify GPU Direct RDMA Information

$ ompi_info --all | grep btl_openib_have_cuda_gdr
   MCA btl: informational "btl_openib_have_cuda_gdr" (current value: "true", data source: default, level: 4 tuner/basic, type: bool)
  • Add flag to mpirun
$ mpirun --mca btl_openib_want_cuda_gdr 1 ...

Reference: https://www.open-mpi.org/faq/?category=runcuda

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment