I recently re-worked our SLURM configuration to extrapolate the number of logical CPUs based on Sockets, Cores, and Threads. Therefore, a machine with four eight-core Xeons and HyperThreading would have 64 logical CPUs available to SLURM; 4 * 8 * 2 = 64. I thought I was being clever, but it turned out to bite me in the ass a bit…

I started seeing messages like this in /var/log/slurm/slurmd.log during job submission:

[2013-04-24T10:50:44+03:00] scaling CPU count by factor of 2

Even if a job only requested one CPU, you’d find that two were being allocated, and therefore nodes with 64 slots fill up with only 32 jobs! Very frustrating…

RTFM

It turns out that this unexpected behavior is slightly expected (or at least known to give unexpected results!). From ThreadsPerCore in man slurm.conf:

... If your system is configured with more than one thread per core, execution of a different job on each thread is not supported unless you configure SelectTypeParameters=CR_CPU plus CPUs; do not configure Sockets, CoresPerSocket or ThreadsPerCore... Note also if you are running with more than 1 thread per core and running the select/cons_res plugin you will want to set the SelectTypeParameters variable to something other than CR_CPU to avoid unexpected results.  The default value is 1.

As I’m not worried so much about CPU affinity / locality, I changed my nodes back to using CPUs=x so I can just run our jobs with a more practical CPU accounting. We’re not rocket scientists here… just regular dudes and dudettes trying to share a few compute nodes!