Editing QUEUES for FELIU MASERAS group

go back to [[Main_Page]], [[Computational Resources]], [[Clusters]], [[Local Clusters]], [[Kimik2]]



== QUEUES SYSTEM MANAGEMENT==  


kimik2 manages automatically calculations sent from your computer:

1- The maximum number of calculations that each user can run at a time is 25, if you send more than that they will wait in the queue until one of your spaces is free.

2- In order to keep the sharing equality of the cluster we decided that the users cannot have more than a number of process running at the same time. 

Each member will have a number of credits "quota" that might change depending on the group circumstances: number of users and how active they are. If you notice that queues are too full or too empty for a period larger than two weeks ask Feliu's permision to encrease/decrease the quota. 

Today (Oct 2018) each member has 1080 credits of quota. The value (price in credits) of each calculation depends on the queue where it is submitted.

The prices of the queues are: 

----
  '''queue   --- nodes x credits --- credits per calc.'''
  c36m192 ---  36x3   credits --- total  96
  c28m128 ---  28x4   credits --- total 112
  c24m128 ---  24x5   credits --- total 120
  c20m128 ---  20x6   credits --- total 120
  c12m24  ---  12x5   credits --- total  60
  c8m24   ---   8x0   credits --- total   0
  c4m8    ---   4x0   credits --- total   0
  cq4m4   ---   4x0   credits --- total   0

  c12m128gpu8 --- 12x0 credits --- total 0
----

Your calculations will enter into empty queues as far as you have free credits. I.e. if you have a quota of 1080, you can have 9 calculation in the c24m128 or 18 to the c12m24. Or anything in between: 2 in c28m128 + 4 in c20m48 + 6 in c12m24.
 
3.- If you have not left quota, your calculations will wait even if there are empty machines. 

4.-If you can put calculations in hold (qhold JOBID to hold & qalter -h U to unhold), if you need others to be done before. 

5.- The small queues can be used freely up to complete the 25 slot allowance.

6.- Please use the c12m128gpu8 queue for calculations that need the gpus

== QUEUES==

Currently there are four queues where calculations can be sent:

c4m8.q (18 NODES)
MACHINE TYPE
                1 x Xeon X3360 2,83Ghz 2x6Mb cache (4 cores)
                4x2GB RAM DDR2 800Mhz (8GB mem)
   PE
                c4m8_mpi
                c4m8_smp
cq4m4.q (26 NODES) 
MACHINE TYPE
                1 x  Core2 Quad Q6600 2,4GHz  8M cache (4 cores)
                4x1GB DDR2 667MHz (4GB mem)
    PE
                cq4m4_mpi
                cq4m4_smp


c8m24.q (27 NODES) 
MACHINE TYPE
                2 x Xeon E5530 2.4GHz 8M cache (8 cores)
                6x4GB RAM DDR3 1333MHz ECC Registered (24GB mem)
    PE
                c8m24_mpi
                c8m24_smp

c12m24.q (18 NODES)
MACHINE TYPE
                2 X Intel(R) Xeon(R) CPU E5645 2.40GHz 12M cache (12 cores)
                6x4GB RAM DDR3 1333MHz ECC Registered (24GB mem)
    PE
                c12m24_mpi
                c12m24_smp


c20m48.1 (4 NODES)
MACHINE TYPE
                ???? (20 cores)
                ??? (48 GB mem)
    PE
                c20m48_mpi
                c20m48_smp

c24m128.1 (23 NODES)
MACHINE TYPE
                ???? (24 cores)
                ??? (128 GB mem)
    PE
                c24m128_mpi
                c24m128_smp

c28m128.1 (4 NODES)
MACHINE TYPE
                ???? (28 cores)
                ??? (128 GB mem)
    PE
                c28m128_mpi
                c28m128_smp

c36m192.1 (3 NODES)
MACHINE TYPE
                ???? (36 cores)
                ??? (192 GB mem)
    PE
                c36m192_mpi
                c36m192_smp

== GPU's NODES==

* c12m128gpu8.1 

There are some gpu nodes that can be used. Terachem, orca and other type of calculations can be achieved with them.



== SENDING CALCULATIONS ==

* ''' Gaussian jobs'''   
 qs TYPE_OF_CALCULATION inputfile.in

where TYPE_OF_CALCULATION can be:


----
for g16:
 g16.c36m192 (36 cores smp pe)
 g16.c28m128 (28 cores smp pe)
 g16.c24m128 (24 cores smp pe)
 g16.c20m48 (20 cores smp pe)
 g16.c12m24 (12 cores smp pe)
 g16.cq4m4 (4 cores smp pe)
----
for g09:
 g09.c36m192 (36 cores smp pe)
 g09.c28m128 (28 cores smp pe)
 g09.c24m128 (24 cores smp pe)
 g09.c20m48 (20 cores smp pe)
 g09.c12m24 (12 cores smp pe)
 g09.c8m24 (8 cores smp pe)
 g09.cq4m4 (4 cores smp pe)       
 g09.c4m8 (4 cores smp pe) 

----
for g03
 g03.cq4m4 (4 cores smp pe)
 g03.c4m8 (4 cores smp pe)
----


'''NOTE:''' remember to add the correct parameters in your inputfile.in according to the TYPE_OF_CALCULATION. '''%nproc=...''' and '''%mem=...'''

* '''Orca jobs'''

 qs orcaVERSION.QUEUE NMACHINES INPUTNAME

where:
 VERSION could be 27 (for version 2.7), 28 (for version 2.8) or 40 (for version 4.0)
 QUEUE For versions previous to 4.0, could be cq4m4 or c4m8 or c8m24 or c12m24. 
       For version 4.0 the queues available are: cq4m4.q, c20m48.q, c24m128.q, c28m128 and c12m128gpu8.q 
 NMACHINES is the number of the machines to use (this parameter have directly relation with ! PALX of the input file, see the [[ORCA]] page)
 INPUTNAME is the name of your input

If when using Orca 4 you don't want too many files copied into your home you could use instead of "qs orcaVERSION.QUEUE NMACHINES INPUTNAME" the submiting script [[submit]]

* ''' GRRM jobs'''

Currently GRRM calculations can only be submitted to the c20m48.q

 qs grrm14.c20m48 infile.in

* '''Other jobs'''

Other type of jobs might need a customized script, like MECP searches (via Harvey or via easyMECP) and others. To submit them use:

 qsub scriptfile

----
The available queues for XTB 6.2.3 are:
 c36m192 
 c28m128
 c24m128
 c20m48
 c12m24
 cq4m4
----

== SENDING THE CALCULATIONS TO SPECIFIC NODES ==

To send calculations to a specific group of nodes of a certain queue we just need to set the hostname in the .in.sub files. We will use some generic g09 calculation in c12m24 as example.

* Generate the .in.sub without running the calculation. (This step '''has to''' be done in an ssh connection to kimik2, it can't be done from your computer)

For a single file
 qs g09.c12m24 0 infile01.in 

For all files in the directory
 for File in infile*.in; do qs g09.c12m24 0 ${File}; done 

Ensure that in the line where '''"#$ -pe"''' the number at the end is the same as the number of processors of the node. (c12m24 -> 12, c20m48 ->20 , c24m128 -> 24 ...) 
Check the line number in which the '''"#$ -l credits"''' appears. 
* Lets assume that its line 13th, therefore the next line is the line 14th.
Ensure that in that line the number of credits is correct (should be 5 for the c12m24).
Add a line below to set the valid hostnames. 
* Let's imagine that we want to use the nodes: kimik2136, kimik2137 and kimik2138. We can indicate it as: 
# hostname=kimik2136|kimik2137|kimik2138
# hostname=kimik213[6-8]
* If we wanted to also include the nodes kimik2139 and kimik2140 we can do it as: 
# hostname=kimik2136|kimik2137|kimik2138|kimik2139|kimik2140
# hostname=kimik213[6-9]|kimik2140

Check that it works properly for one file.
 sed '14i#$ -l hostname=kimik2136|kimik2137|kimik2138' infile01.in.sub
''where the '''14i''' translates as  '''i'''nsert the following text as line '''14'''th,''

If it does work properly, apply the change to all the files in the directory
 sed -i '14i#$ -l hostname=kimik2136|kimik2137|kimik2138' infile*.in.sub

Submit the calculations (This step '''has to''' be done in an ssh connection to kimik2, it can't be done from your computer)

* For a single file
 qsub infile.in.sub

* For all the files in the directory
 for File in infile*.in.sub; do qsub ${File}; done