Editing Submitting a Quantum Espresso job

Go back to [[Scripts_for_QE]]

To submit to CSUC collserola, usually for small jobs (max 24 cores per node), shorter waiting queue for small jobs. 
<pre>
#!/bin/bash
# specifies the name of your job
# BSUB -J jobname
#
# send standard output here
# BSUB -o quantum_espresso.log  
#
# send standard error here
# BSUB -e quantum_espresso.err 
#
# pick a queue
# BSUB -q parallel8
#
# pick the machine
# BSUB -R collserola
#
# Indicate the number of cores
# BSUB -n 8
# All process should be executed in the same node
# BSUB -R span[hosts=1] 
#
# Send an email notice once the job is finished
# BSUB -N -u MYEMAIL@iciq.es

export ESPRESSO_PSEUDO=$HOME/espresso_pseudo


INDIR=$ESPRESSO_TMPDIR # directory with your input and the .UPF files
INPUT=INPUT.in
OUTPUT=OUTPUT.out

date

# Set up the environment
. /opt/modules/default/init/bash
module load quantumespresso/6.1

export ESPRESSO_PSEUDO=$HOME/espresso_pseudo

unset OMP_NUM_THREADS

EXE="/prod/QuantumESPRESSO/6.1/bin/pw.x -environ"

# Change to the work directory and copy all the necessary files to this folder

cd $TMPDIR
cp $INDIR/$INPUT $INDIR/*.UPF $TMPDIR
cp $INDIR/environ_water.in $TMPDIR/environ.in

	
# Run the job

mpijob $EXE < $INPUT > $OUTPUT

# Create a new directory in the $SCRATCH directory, each new directory for each job specifying the JobID, and copy the output to this directory

mkdir -p $SCRATCH/output_$LSB_JOBID
cp -r * $SCRATCH/output_$LSB_JOBID

# move in HOME and delete in scratch
mv $SCRATCH/output_$LSB_JOBID $HOME/.

date
</pre>

To submit to CSUC pirineus, usually for very big jobs that requires a lot of nodes and therefore longer waiting queue
<pre>
#!/bin/ksh
# specifies the name of your job
# BSUB -J jobname
#
# send standard output here
# BSUB -o quantum_espresso.log  
#
# send standard error here
# BSUB -e quantum_espresso.err 
#
# pick a queue
# BSUB -q parallel32 
#
# pick the machine
# BSUB -R pirineus
#
# Indicate the number of cores
# BSUB -n 32
# All process should be executed in the same node
# BSUB -R span[hosts=1] 
#
# Send an email notice once the job is finished
# BSUB -N -u MYEMAIL@iciq.es

INDIR=$HOME/WORKDIRECTORY # directory with your input and the .UPF files
INPUT=INPUT.in
OUTPUT=OUTPUT.out

date

# Set up the environment
. /opt/modules/default/init/bash
module load quantumespresso/6.1

export ESPRESSO_PSEUDO=$HOME/espresso_pseudo

unset OMP_NUM_THREADS

EXE="/prod/QuantumESPRESSO/6.1/bin/pw.x -environ"

# Change to the work directory and copy all the necessary files to this folder

cd $TMPDIR
cp $INDIR/$INPUT $INDIR/*.UPF $TMPDIR
cp $INDIR/environ_water.in $TMPDIR/environ.in

	
# Run the job

mpijob $EXE < $INPUT > $OUTPUT

# Create a new directory in the $SCRATCH directory, each new directory for each job specifying the JobID, and copy the output to this directory

mkdir -p $SCRATCH/output_$LSB_JOBID
cp -r * $SCRATCH/output_$LSB_JOBID

# move in HOME and delete in scratch
mv $SCRATCH/output_$LSB_JOBID $HOME/.

date
</pre>
Jobs are sent to CSUC queue with the command
<pre>
bsub< jobname.lsf
</pre>
Do not forget the "<" or you will see the message that your job was submitted to the default "short" queue but in reality nothing happened.

To submit a job to CESVIMA pmagerit
<pre>
#!/bin/bash
#----------------------- Start job description -----------------------
#@ total_tasks = 32
#@ wall_clock_limit = 48:00:00
#@ output = out-%j.log
#@ error = err-%j.log
#@ initialdir = /home/iciq23/iciq23336/espresso_tmp/

#------------------------ End-job description ------------------------
    
#-------------------------- Start execution --------------------------
    
# Run our program
module load gcc/4.7
srun /sw/openmpi/Quantum-ESPRESSO/6.1/bin/pw.x < INPUT.in > OUTPUT.out
   
#--------------------------- End execution ---------------------------
</pre>

Jobs are sent CESVIMA queue with
<pre>
jobsubmit jobname.sh
</pre>


To submit an environ job to MareNostrum4 (MN4)

<pre>
#!/bin/bash
#SBATCH --ntasks=48
#SBATCH --tasks-per-node=48
#SBATCH --time=48:00:00 ## hour
#SBATCH --cpus-per-task=1
#SBATCH --constraint=highmem
#SBATCH --error=qe%J.err
#SBATCH --job-name="hello_test"

module load quantumespresso/6.2-environ-1.0

PREFIXfolder=FeNi2O4
INPUT=${PREFIXfolder}.in2
OUTPUT=${PREFIXfolder}.out2

export ESPRESSO_PSEUDO=$HOME/espresso_pseudo
export ESPRESSO_TMPDIR=$SCRATCH/espresso_tmp/vacuum/${PREFIXfolder}  #all temp files goes there
export WORKDIR=$HOME/espresso_tmp/vacuum/spinel  #your work directory
mkdir $ESPRESSO_TMPDIR



#cp environ_vacuum.in environ.in

time srun  pw.x -environ < $INPUT > $OUTPUT
</pre>
To submit an environ job to MareNostrum4
<pre>
sbatch script.lsf
</pre>

WARNING: The previous script does not use the scratch directory of MareNostrum4. You risk to reach your disk quota quickly. The following script use the scratch but the variable SCRATCH must be defined in your .bashrc first! (export SCRATCH=/gpfs/scratch/iciq72/iciq72520 in my case)

Writing wavefunctions in QE needs a very high amount of memory; often calculations crash due to exceeding memory.
It is generally a good idea to add the line:

SBATCH --constraint=highmem

It runs calculations in a high memory node. 


<pre>
#!/bin/bash
#SBATCH --ntasks=48
#SBATCH --tasks-per-node=48
#SBATCH --time=1:00:00 ## hour
#SBATCH --cpus-per-task=1
#SBATCH --error=qe%J.err
#SBATCH --job-name="hello_test"

module load quantumespresso/6.2-environ-1.0

PREFIXfolder=h2o_water
INPUT=${PREFIXfolder}.in
OUTPUT=${PREFIXfolder}.out1

export ESPRESSO_PSEUDO=$HOME/espresso_pseudo
export ESPRESSO_TMPDIR=$SCRATCH/espresso_tmp/water  #all temp files goes there
export WORKDIR=$HOME/espresso_tmp/water
cp environ_water.in environ.in

# Run the job in SCRATCH
time srun  pw.x -environ < $INPUT > $OUTPUT

# copy back in HOME the pot.save folder
cp -r $ESPRESSO_TMPDIR/$PREFIXfolder.save $WORKDIR/.
</pre>

Here is a python script that make the lsf submission script for you. You just need to replace the variables name_prefix and medium.
<pre>
#!/usr/bin/python2
#This script write the MN4 lsf script for job submission

name_prefix="sym5"

#output=name_prefix+".in" #output filename for QE input run
output_lsf=name_prefix+".lsf96" #
LDOS=name_prefix+"_LDOS.in" #filename for LDOS calculation3
LDOS_lsf=name_prefix+"_LDOS.lsf48" #filename for LDOS calculation

medium="vacuum"

outlsf=open(output_lsf,"w")
print >> outlsf, "#!/bin/bash"
print >> outlsf, "#SBATCH --ntasks=96"
print >> outlsf, "#SBATCH --tasks-per-node=48"
print >> outlsf, "#SBATCH --time=48:00:00 ## hour"
print >> outlsf, "#SBATCH --cpus-per-task=1"
print >> outlsf, "#SBATCH --error=qe%J.err"
print >> outlsf, ""
print >> outlsf, "module load quantumespresso/6.2-environ-1.0"
print >> outlsf, "INPUT="+name_prefix+".in"
print >> outlsf, "OUTPUT="+name_prefix+".out1"
print >> outlsf, ""
print >> outlsf, "export ESPRESSO_PSEUDO=$HOME/espresso_pseudo"
print >> outlsf, "export ESPRESSO_TMPDIR=$SCRATCH/espresso_tmp/"+medium
print >> outlsf, "export WORKDIR=$HOME/espresso_tmp/"+medium
print >> outlsf, "cp environ_"+medium+".in environ.in"
print >> outlsf, ""
print >> outlsf, "time srun  pw.x -environ < $INPUT > $OUTPUT"
print >> outlsf, ""
print >> outlsf, "# copy back in HOME the pot.save folder"
print >> outlsf, "cp -r $ESPRESSO_TMPDIR/$PREFIXfolder.save $WORKDIR/."
outlsf.close

#writing QE LDOS file
ldos=open(LDOS,"w")
print >> ldos, " &PROJWFC"
print >> ldos, "                      prefix =","'"+name_prefix+"'", ","
print >> ldos, "                      !outdir = '/media/hnguyen/WAREHOUSE/tekla/espresso_tmp/SurfaceUSPP/001' ,"
print >> ldos, "                      DeltaE = 0.01 ,"
print >> ldos, "                        Emin = -150.0 ,"
print >> ldos, "                        Emax = 30.0 ,"
print >> ldos, "                      ngauss = 0," 
print >> ldos, "                     degauss = 0.001469972, ! =0.02 eV"
print >> ldos, "                     pawproj = .false. ,"
print >> ldos, " /"
ldos.close

#writing LDOS lsf script
ldoslsf=open(LDOS_lsf,"w")
print >> ldoslsf, "#!/bin/bash"
print >> ldoslsf, "#SBATCH --ntasks=48"
print >> ldoslsf, "#SBATCH --tasks-per-node=48"
print >> ldoslsf, "#SBATCH --time=10:00:00 ## hour"
print >> ldoslsf, "#SBATCH --cpus-per-task=1"
print >> ldoslsf, "#SBATCH --error=qe%J.err"
print >> ldoslsf, ""
print >> ldoslsf, "module load quantumespresso/6.2-environ-1.0"
print >> ldoslsf, "PREFIXfolder="+name_prefix
print >> ldoslsf, "INPUT=${PREFIXfolder}_LDOS.in"
print >> ldoslsf, "OUTPUT=${PREFIXfolder}_LDOS.out" 
print >> ldoslsf, ""
print >> ldoslsf, "export ESPRESSO_PSEUDO=$HOME/espresso_pseudo"
print >> ldoslsf, "export ESPRESSO_TMPDIR=$SCRATCH/espresso_tmp/"+medium
print >> ldoslsf, "export WORKDIR=$HOME/espresso_tmp/"+medium
print >> ldoslsf, ""
print >> ldoslsf, "time srun  projwfc.x < $INPUT > $OUTPUT"
print >> ldoslsf, "mkdir PDOS"
print >> ldoslsf, "mkdir PDOS/$PREFIXfolder"
print >> ldoslsf, "mv *pdos* PDOS/$PREFIXfolder/."
print >> ldoslsf, "grep 'polarization' $OUTPUT > ${OUTPUT}.polarization"
print >> ldoslsf, "grep 'charge' $OUTPUT > ${OUTPUT}.charge"
print >> ldoslsf, ""
print >> ldoslsf, "# copy back in HOME the pot.save folder"
print >> ldoslsf, "cp -r $ESPRESSO_TMPDIR/${PREFIXfolder}.save  $WORKDIR/."
print >> ldoslsf, "cp -r ${OUTPUT}.polarization ${OUTPUT}.charge $WORKDIR/."
print >> ldoslsf, "cp -r PDOS/$PREFIXfolder $WORKDIR/PDOS/."
ldoslsf.close
</pre>


A simple script to submit to tekla
<pre>
#!/bin/bash
#   - Dra. Nuria's Lopez Group  -
##########################################
# SGE Parameters
##########################################
#$ -S /bin/bash
#$ -N Jobname
#$ -cwd
#$ -masterq c24m128ib.q
#$ -pe c24m128ib_mpi  24
#$ -m ae
#$ -M youremail@iciq.es
#$ -o $JOB_NAME.o$JOB_ID
#$ -e $JOB_NAME.e$JOB_ID
cat $TMP/machines.$JOB_ID >> $JOB_NAME.MACHINES.$JOB_ID


# Set up the environment
. /etc/profile.d/modules.sh
module load quantum-espresso/6.1

##########################################
# Running Job
##########################################

export OMP_NUM_THREADS=1

echo $PWD >> $JOB_NAME.o$JOB_ID
echo $TMP >> $JOB_NAME.o$JOB_ID


PREFIXfolder=Jobname
INPUT=${PREFIXfolder}.in
OUTPUT=${PREFIXfolder}.out1

export ESPRESSO_PSEUDO=$HOME/espresso_pseudo  #your pseudopotential directory
export ESPRESSO_TMPDIR=$HOME/espresso_tmp/vacuum/${PREFIXfolder}  #all temp files goes there
export WORKDIR=$HOME/espresso_tmp/vacuum/spinel  #your work directory
mkdir $ESPRESSO_TMPDIR

time mpirun -np $NSLOTS $BIN_DIR/pw.x -input  $INPUT >  $OUTPUT
</pre>
The folders for ESPRESSO_PSEUDO, ESPRESSO_TMPDIR must exist for the script to work correctly.