User Tools

Site Tools


hpc_user_guide:queuespes

Queues and parallel environments

Queues

Rosalind queue summary:

Queue nameRam (GB)runtime limitAccepts
Batch
jobs
Accepts
Interactive
Jobs
Available
parallel
environmnets
Slots(cores)Max slots
per user
HighMemLongterm.q 380 no limit y y mpi-slots, smp 240 undefined
HighMemShortterm.q 380 192 hours y y mpi-slots, smp 400 undefined
LowMemLongterm.q 190 no limit y y mpi-slots, smp 480 undefined
LowMemShortterm.q 190 192 hours y y mpi-slots, smp 760 undefined
InterLowMem.q 190 72 hours n y none 40 2
LowCores.q 32 192 hours y y mpi-16 1088 undefined
SMP.q 64, 128 192 hours y y smp-16 320 undefined
SMPHighSpec.q 256,1024 192 hours y y smpHS 80 undefined
GPU.q various 120 hours y y gpu-mpi, gpu-mpi-k40 18 undefined

Compute nodes in the “HighMem*” queues have 380GB accessible by the users and the “LowMem*” queues have 190GB available - (see this section in the HPC user guide for details on reserving memory.)

Queues named “*Longterm.q” have no hard time limit at the queue level however when submitting jobs you still need to specify your run time ( see here for details). Queues named “*Shortterm.q” have a hard runtime limit of 8 days. Even if you specify a longer runtime at submission time, jobs on these queues cannot run for longer than this limit.

All of the queues accept interactive jobs however the queue InterLowMem.q acceptes interactive jobs only and will not accept batch jobs. This queue has a run time limit of 24 hours. See here for instructions on how to launch interactive jobs on this queue.

You can inspect the queue configurations in more detail with the command

qconf -sq <queue name>

Parallel Environments

All queues except the interactive only queue, are configured with parallel environments as follows:

  • smp:
    • Symmetric Multiprocessing - this environment is suited to multi-threaded jobs which require all of the processes to execute on the same node. Most of the nodes on the Rosalind cluster have 20 cores, so jobs requesting more cores will likely never be scheduled. Usually in SMP jobs you need to synchronized the requested slots with the threads that your program runs SO you need to put in your script this lines:
 export MKL_NUM_THREADS=$NSLOTS
 export NUMEXPR_NUM_THREADS=$NSLOTS
 export OMP_NUM_THREADS=$NSLOTS
  • mpislots:
    • this environment is suited to MPI (Message Passing Interface) jobs. Individual processes can be scheduled on different nodes and it should be request in increments of 20s. So your process can run in 20, 40, 60, 80 cores etc.

You can inspect the parallel environment configurations in more detail with the command:

qconf -sp <parallel environment name>

The interactive queue can use the smp parallel environment but not the mpi parallel environment.

hpc_user_guide/queuespes.txt · Last modified: 2017/10/16 14:21 by admin