Worker Profiles and Accelerators

CPU, memory, disk and GPUs are settings of the worker VM template, not service inputs. Change the template before you create the service or before you add workers.

onetemplate update <WORKER_TEMPLATE_ID>

Each worker reports its CPUs, memory and GPUs to Slurm when it joins, so Slurm always uses the real size of the VM.

CPU and Memory

The Marketplace worker template has 1 CPU and 1 GB of memory. Set values for CPU, VCPU and MEMORY in the template appropriately for your jobs. For CPU pinning and NUMA, refer to the documentation about NUMA and CPU Pinning.

NVIDIA GPUs

The worker image includes the NVIDIA driver packages (nvidia-driver-595-server-open). The worker template has no attached GPU by default.

Add a GPU to the worker VM template, as a PCI device or a PCI profile of your Host. See PCI Passthrough. This example requests an NVIDIA H100 PCIe:
```
PCI = [
  VENDOR = "10de",
  DEVICE = "2331",
  CLASS  = "0302" ]
```
The device ID and the PCI profiles depend on your Hosts. You can find the relevant parameters by running lspci -nn | grep -i nvidia on the Host’s command line.
Create the service or add workers. Each worker counts its GPUs with nvidia-smi and starts slurmd with Gres=gpu:<count>.

Check from the controller:

scontrol show nodes | grep Gres
srun -N1 -n1 --gres=gpu:1 nvidia-smi -L

InfiniBand

Workers for MPI jobs need an InfiniBand HCA in the VM template. Refer to InfiniBand and High-Performance Networking.

GIVE FEEDBACK

Was this resource helpful?

Glad to hear it

Sorry to hear that