Service Architecture

Components

ComponentWhereRole
OneFlow service OneSlurmOpenNebula Front-endCreates the roles in order and scales the workers
controller role1 VMslurmctld, Munge, optional OpenLDAP, reconciler
worker role1 or more VMs, child of controllerslurmd, Munge, self-drain hook
OneGateOpenNebula Front-endShares data between the VMs of the service
Service networkVirtual NetworkConnects all the VMs. Chosen when the service is created

The service uses deployment: straight and ready_status_gate: true. OneFlow creates the controller first and the workers only after the controller reports READY=YES.

Data in OneGate

AttributePublished byRead byContent
READYControllerOneFlow, workersYES when the controller is configured
SLURM_MUNGE_KEYControllerWorkersMunge key in base64
LDAP_URLControllerWorkersLDAP URL. Empty when LDAP is disabled
LDAP_DOMAINControllerWorkersLDAP domain or base DN
LDAP_ADMIN_USERControllerLDAP admin user, when set
LDAP_BIND_USERControllerWorkersLDAP bind user, when set
LDAP_BIND_PASSWORDControllerWorkersLDAP bind password, when set
SLURM_NODENAMEEach workerController reconcilerName of the Slurm node of this VM

With local LDAP, LDAP_URL is ldap://<CONTROLLER_IP>.

Network Ports

The following table shows the default ports of each service.

PortServiceFromTo
6817/TCPslurmctld and configless configurationWorkersController
389/TCPLDAPAll VMsLocal LDAP on the controller, or the external LDAP
2049/TCPNFSv4All VMsNFS server, when NFS is used
22/TCPSSHUsersController

Files and Units

ItemVMPurpose
/etc/slurm/slurm.conf, gres.conf, cgroup.confControllerSlurm configuration, written at each boot
/etc/munge/munge.keyAllMunge key
/etc/sssd/sssd.confAll, with LDAPSSSD client for LDAP
oneslurm-reconcile.timerControllerRuns the reconciler 2 minutes after boot, then every 60 seconds
/usr/local/sbin/oneslurm-reconcile-nodesControllerReconciler script
oneslurm-self-drain.serviceWorkersRemoves the node from Slurm at shutdown
/etc/systemd/system/slurmd.serviceWorkersslurmd in configless mode with dynamic registration
/etc/netplan/90-oneslurm-ipoib.yamlWorkers, with InfiniBandIPoIB address

Boot Sequence

In every VM, the NFS exports are mounted first, while the network is configured. Then the appliance configures the role.

Controller

  1. Sets the hostname slurm-one-controller.
  2. Writes slurm.conf, gres.conf and cgroup.conf.
  3. Creates the Munge key at the first boot and starts slurmctld.
  4. Configures LDAP, local or external, and publishes the LDAP data in OneGate.
  5. Publishes SLURM_MUNGE_KEY and READY=YES in OneGate.
  6. Starts the reconciler timer.

Worker

  1. Waits until the controller publishes READY=YES, then reads its IP, the Munge key and the LDAP data.
  2. Sets its hostname. The Marketplace template uses the VM name. Without SET_HOSTNAME, the hostname is slurm-one-worker-<VM_ID>.
  3. Checks that port 6817 of the controller answers.
  4. Maps slurm-one-controller to the controller IP in /etc/hosts.
  5. Configures IPoIB, when InfiniBand is enabled.
  6. Installs the Munge key and starts slurmd.
  7. Publishes SLURM_NODENAME in OneGate.
  8. Installs the self-drain hook.
  9. Configures the SSSD client, when the controller published LDAP data.