Information about TIFR-CAM cluster

The nodes are put into two groups, see the file /opt/torque/server_priv/nodes

PBS is a batch handling system to manage parallel applications submitted by users. On the cluster, PBS uses Maui as the scheduler. Jobs are submitted to PBS using a script; examples are given below under the openmpi and mpich2 sections. If the script is called famosa.pbs, you can submit the job to PBS using

$ qsub famosa.pbs

Here are some parameters that can be given in a PBS script file:

• -N jobname (name the job “jobname”)
• -q @nic-cluster.cc.umr.edu (The cluster address to send the job to)
• -e errfile (redirect standard error to a file named errfile)
• -o outfile (redirect standard output to a file named outfile)
• -j oe (combine standard output and standard error)
• -l walltime=N (request a walltime of N in the form hh:mm:ss)
• -l cput=N (request N sec of CPU time; or in the form hh:mm:ss)
• -l mem=N[KMG][BW] (request total N kilo| mega| giga} {bytes|words} of memory on all requested processors together)
• -l nodes=N:ppn=M (request N nodes with M processors per node)
• -m abe (mail the user when the job aborts/began running/ended)
• -S shell (use shell instead of your login shell to interpret the batch script; must include a complete path)
• -V (job inherits the full environment of the current shell, including $DISPLAY)

Once a job is submitted, you can check its status using qstat

[praveen@master piaggio_pso]$ qstat
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
25.master                 20080919_3       roms            22:58:50 R default        
28.master                 FAMOSA           praveen         00:11:11 R default 

To get more detailed information, use qstat -f or qstat -f <jobid>

To delete a running job, use

$ qdel <jobid>

If the job is not killed by the above command, then force it using

$ qdel -p <jobid>

Note: qpeek was not working when torque was installed from Rocks 5. It would give an error that there is no file in /opt/torque/spool. After commenting line 142 in /opt/torque/bin/qpeek, it works.

Openmpi was compiled with the following configure options

./configure --with-tm=/opt/torque --prefix=/opt/openmpi-1.2.7 \
                      --enable-prefix-by-default --enable-static

After compiling, check that all required features are enabled using ompi_info. In particular, to verify that torque support is built in, do

[praveen@master ]$ /opt/openmpi-1.2.7/bin/ompi_info |grep tm
              MCA memory: ptmalloc2 (MCA v1.0, API v1.0, Component v1.2.7)
                 MCA ras: tm (MCA v1.0, API v1.3, Component v1.2.7)
                 MCA pls: tm (MCA v1.0, API v1.3, Component v1.2.7)

The following is an example PBS script for use with openmpi.

#PBS -N "rae2822"
#PBS -l "nodes=5:hardy:ppn=6"
#PBS -l "walltime=48:00:00"
#PBS -j oe
#PBS -o famosa.log
#PBS -m e

export OPENMPI=/opt/openmpi-1.2.7
export PATH=$OPENMPI/bin:$PATH
export LD_LIBRARY_PATH=$OPENMPI/lib
 
cd $PBS_O_WORKDIR
 
mpirun $HOME/src/famosa/build/bin/Famosa_mpi

Mpich2 is installed using Rocks in /opt/mpich2/gnu and uses gfortran as the fortran compiler.

Use mpiexec in /opt/mpiexec to launch mpich2 programs together with PBS. An example script is given below

#PBS -N "rae2822"
#PBS -l "nodes=5:hardy:ppn=6"
#PBS -l "walltime=48:00:00"
#PBS -j oe
#PBS -o famosa.log
#PBS -m e

cd $PBS_O_WORKDIR
 
/opt/mpiexec/bin/mpiexec --comm=pmi $HOME/src/famosa/build/bin/Famosa_mpi

This command can be use to execute something on all nodes. For example to see the list of processes for user praveen, do

cluster-fork ps -U praveen

To run some command only on a particular set of nodes, use

cluster-fork -n "c0-0 c0-1 c0-2 c0-3 c0-4" ps -U praveen

Another was is to use

cluster-fork --nodes="c0-%d:5-14" ps -U praveen

This command gives some information about a submitted job

checkjob -v <JOBID>

where JOBID is given by qstat.

showq gives a concise summary of all jobs running or in the queue.

showscript will return the contents of the PBS script that you have submitted. The only argument is the job’s PBS jobid.

You can use this to suspend or resume a PBS job. See the help

[praveen@master ]$ mjobctl --help
Usage: mjobctl [FLAGS]
  --about
  --configfile=<FILENAME>
  --format=<FORMAT>
  --help
  --host=<SERVERHOSTNAME>
  --keyfile=<FILENAME>
  --loglevel=<LOGLEVEL>
  --port=<SERVERPORT>
  --version

  -c <JOBID> // CANCEL
  -C <JOBID> // CHECKPOINT
  -h <JOBID> // HOLD
  -r <JOBID> // RESUME
  -R <JOBID> // REQUEUE
  -s <JOBID> // SUSPEND
  -S <JOBID> // SUBMIT
  -x <JOBID> // EXECUTE