User Tools

Site Tools


hpc_user_guide:queuespes

Queues and parallel environments

Queues

Rosalind queue summary:

Queue nameRam (GB)runtime limitAccepts
Batch
jobs
Accepts
Interactive
Jobs
Available
parallel
environmnets
Slots(cores)Max slots
per user
HighMemLongterm.q 380 no limit y y mpislots, smp 240 undefined
HighMemShortterm.q 380 192 hours y y mpislots, smp 400 undefined
LowMemLongterm.q 190 no limit y y mpislots, smp 480 undefined
LowMemShortterm.q 190 192 hours y y mpislots, smp 760 undefined
InterLowMem.q 190 72 hours n y none 40 2
LowCores.q 32 192 hours y y mpi-16 1088 undefined
SMP.q 64, 128 192 hours y y smp-16 320 undefined
SMPHighSpec.q 256,1024 192 hours y y smpHS 80 undefined
GPU.q various 120 hours y y gpu-mpi, gpu-mpi-k40 18 undefined

Compute nodes in the “HighMem*” queues have 380GB accessible by the users and the “LowMem*” queues have 190GB available - (see this section in the HPC user guide for details on reserving memory.)

Queues named “*Longterm.q” have no hard time limit at the queue level however when submitting jobs you still need to specify your run time ( see here for details). Queues named “*Shortterm.q” have a hard runtime limit of 8 days. Even if you specify a longer runtime at submission time, jobs on these queues cannot run for longer than this limit.

All of the queues accept interactive jobs however the queue InterLowMem.q acceptes interactive jobs only and will not accept batch jobs. This queue has a run time limit of 24 hours. See here for instructions on how to launch interactive jobs on this queue.

You can inspect the queue configurations in more detail with the command

qconf -sq <queue name>

Parallel Environments

All queues except the interactive only queue, are configured with parallel environments as follows:

  • smp: Symmetric Multiprocessing - this environment is suited to multi-threaded jobs which require all of the processes to execute on the same node. Most of the nodes on the Rosalind cluster have 20 cores, so jobs requesting more cores will likely never be scheduled. Usually in SMP jobs you need to synchronized the requested slots with the threads that your program runs SO you need to put in your script this lines:
 export MKL_NUM_THREADS=$NSLOTS
 export NUMEXPR_NUM_THREADS=$NSLOTS
 export OMP_NUM_THREADS=$NSLOTS
  • mpislots: This environment is suited to MPI (Message Passing Interface) jobs. Individual processes can be scheduled on different nodes and it should be request in increments of 20s. So your process can run in 20, 40, 60, 80 cores etc.
  • mpi-16: Similarly to mpi this environment is suited to MPI (Message Passing Interface) jobs but the increments should be of 16s. So your process can run in 16, 32, 48, 64 cores etc. Your job will run in the LowCores nodes which have previous generation CPUs.
  • smp-16: Similarly to smp but maximum cores are 16. Furthermore, the nodes used by this pe are not having infiniband connection, so writing/reading a lot of data during the running would be slower.
  • smpHS: Similarly to smp but maximum cores are 32 and 48 as in the respective queue there only two nodes. These nodes have infiniband connection, so can be used for writing/reading a lot of data during running;
  • gpu-mpi,gpu-mpi-k40: Thisis for running in GPU nodes consisting of nvidia k20 and k40 Tesla GPUs, respectively.These nodes have infiniband connection, so can be used for writing/reading a lot of data during running and for runnign multinode gpu jobs;

You can inspect the parallel environment configurations in more detail with the command:

qconf -sp <parallel environment name>

The interactive queue can use the smp parallel environment but not the mpi parallel environment.

hpc_user_guide/queuespes.txt · Last modified: 2018/02/23 14:34 by admin