InfiniBand and High-Performance Networking

OneSlurm can configure IPoIB on the workers and set Slurm for MPI jobs over InfiniBand. It is disabled by default.

Before You Start

Ensure that:

  • Each worker VM has an InfiniBand HCA, attached by PCI passthrough in the worker VM template. See PCI Passthrough.
  • The InfiniBand fabric already has a subnet manager. OneSlurm does not configure it.

What the Images Include

ImagePackages
Workerrdma-core, infiniband-diags, UCX from the Canonical DOCA PPA, Open MPI runtime
ControllerOpen MPI development packages, to compile MPI programs

Both images set the memlock limit to unlimited, which RDMA needs.

Enable InfiniBand

ParameterDefaultDescription
ONEAPP_SLURM_INFINIBAND_ENABLENOYES configures IPoIB on the workers and MPI settings in slurm.conf
ONEAPP_SLURM_IPOIB_SUBNETemptyIPv4 subnet for IPoIB, for example 10.20.0.0/24. Without a prefix, /24 is used. Required when InfiniBand is enabled

With InfiniBand enabled, the controller adds two lines to slurm.conf.

SettingEffect
MpiDefault=pmixsrun and sbatch use PMIx without --mpi=pmix
PropagateResourceLimitsExcept=MEMLOCKJobs keep the memlock limit of the worker

IPoIB Addresses

Each worker builds its IPoIB address from the subnet and its Ethernet address. It copies the last octets of the Ethernet address, as many as the prefix leaves free.

IPoIB subnetWorker Ethernet IPWorker IPoIB address
10.20.0.0/24172.16.0.1110.20.0.11/24
10.20.0.0/16172.16.5.4210.20.5.42/16
10.0.0.0/8172.16.5.4210.16.5.42/8
  • Only subnets /8, /16 and /24 are valid.
  • The copied octets must be different on each worker. Otherwise two workers get the same IPoIB address.
  • The copied octets cannot be all 0 or all 255.
  • The worker writes the address to /etc/netplan/90-oneslurm-ipoib.yaml.

Slurm itself stays on the Ethernet network. Controller discovery, Munge, LDAP and slurmd do not use IPoIB. MPI and RDMA traffic can use InfiniBand.

Check InfiniBand

On a worker:

ibstat
ibv_devinfo
ip addr show ib0
ucx_info -d | grep mlx5

On the controller:

mpicc --version
scontrol show config | grep -E 'MpiDefault|PropagateResourceLimitsExcept'
srun -N2 hostname

Run an MPI Job

Compile on the controller and run with srun:

mpicc mpi_program.c -o mpi_program
srun -N2 ./mpi_program