Slurm Cluster Lifecycle Management

A OneSlurm Cluster is a OneFlow service. You create, scale and delete it with the oneflow commands.

TaskCommand
Create a Clusteroneflow-template instantiate 'Service OneSlurm'
List Clustersoneflow list
Show the VMs of a Clusteroneflow show <SERVICE_ID>
Change the number of workersoneflow scale <SERVICE_ID> worker <NUMBER>
Delete a Clusteroneflow delete <SERVICE_ID>

The Quick Start Guide demonstrates how to create a Cluster.

Add Workers 

oneflow scale <SERVICE_ID> worker 4

OneFlow creates the new worker VMs. Each new worker reads the Cluster data from OneGate and joins Slurm as a dynamic node, as described in Core Concepts. Check the new nodes from the controller.

sinfo
LimitValue
Recommended minimum number of workers1
Maximum number of nodes in Slurm100 (MaxNodeCount in slurm.conf)
Cooldown after a scale operation30 seconds. During the cooldown you cannot scale or delete the service

Remove Workers 

oneflow scale <SERVICE_ID> worker 1

OneFlow deletes the extra worker VMs. Two mechanisms then remove the nodes from Slurm.

MechanismWhere it runsWhenWhat it does
Self-drainEach worker, oneslurm-self-drain.serviceWhen the VM shuts down normallySets the node DOWN and deletes it from Slurm
ReconcilerController, oneslurm-reconcile.timer2 minutes after boot, then every 60 secondsRemoves nodes that have no VM anymore

The reconciler is a safety net for VMs that stop without a normal shutdown. It acts on a node only when all three following conditions are true:

  1. The node is registered in Slurm.
  2. No worker VM of the service publishes this node name in OneGate.
  3. Slurm sees the node as DOWN or NOT_RESPONDING.

Then it deletes the node. If the node still has running jobs, it drains the node instead, with the reason removed from OneFlow service.

If OneGate does not answer, or no worker has published its name yet, the reconciler changes nothing. Keep at least one worker in the service.

Delete the Cluster 

oneflow delete <SERVICE_ID>

This deletes all VMs of the Cluster. Data stored inside the VMs is lost, including the users of the local LDAP on the controller. Data on external NFS exports persists.