James: Text replacement - "[pP][bB][sS]" to "SLURM"

2022-10-27T19:33:39Z

Text replacement - "[pP][bB][sS]" to "SLURM"

← Older revision		Revision as of 19:33, 27 October 2022
Line 46:		Line 46:
	For meaning and proper choice of values in <font face="courier"><font color= red><b><chunks></b></font></font color> and <font face="courier"><font color=red><b><cpus></b></font></font color> fields please read the section [[Submitting_Jobs\|“Writing a job submit script.”]]		For meaning and proper choice of values in <font face="courier"><font color= red><b><chunks></b></font></font color> and <font face="courier"><font color=red><b><cpus></b></font></font color> fields please read the section [[Submitting_Jobs\|“Writing a job submit script.”]]

	<font color=red><b>Example:</b></font color> The following example ~~PBS~~ scripts are described below: (1) General Script Template (2) Penzias (a) 4 cpus and 1 gpu, (b) 4 cpus and 2 gpus (c) 2 mpi threads per gpu and a submit script for (3) Andy.		<font color=red><b>Example:</b></font color> The following example SLURM scripts are described below: (1) General Script Template (2) Penzias (a) 4 cpus and 1 gpu, (b) 4 cpus and 2 gpus (c) 2 mpi threads per gpu and a submit script for (3) Andy.
	==General submit script template==		==General submit script template==

Line 68:		Line 68:
	==PENZIAS==		==PENZIAS==
	=====A. Example Submit Script: 4 CPUs and 1 GPU=====		=====A. Example Submit Script: 4 CPUs and 1 GPU=====
	In the script below, the ~~PBS~~ scheduler is asked to schedule 4 CPUs and 1 GPU for a job.		In the script below, the SLURM scheduler is asked to schedule 4 CPUs and 1 GPU for a job.

	On PENZIAS, there are 2 versions of GROMACS which are GPU enabled: GROMACS 5.X and GROMACS 4.X. The 5.x uses more advanced communication protocol between GPU and has improved CPU performance. The following ~~PBS~~ script can be used as a model on how to run a MPI parallel GROMACS 5.X job with GPUs on PENZIAS. First, load the module for GROMACS with command:		On PENZIAS, there are 2 versions of GROMACS which are GPU enabled: GROMACS 5.X and GROMACS 4.X. The 5.x uses more advanced communication protocol between GPU and has improved CPU performance. The following SLURM script can be used as a model on how to run a MPI parallel GROMACS 5.X job with GPUs on PENZIAS. First, load the module for GROMACS with command:
	module load gromacs/5.0.4		module load gromacs/5.0.4

Line 121:		Line 121:

	==APPEL==		==APPEL==
	The following ~~PBS~~ script can be used as a model how to run MPI only simulations with GROMACS 4.X on ANDY. First load the module for the GROMACS with the command:		The following SLURM script can be used as a model how to run MPI only simulations with GROMACS 4.X on ANDY. First load the module for the GROMACS with the command:
	module load gromacs		module load gromacs

	The ~~PBS~~ script is as follows:		The SLURM script is as follows:

	#!/bin/bash		#!/bin/bash

James: Created page with "

GROMACS

__TOC__
Description: GROMACS is a highly optimized, fast, Molecular Dynamics (MD) simulation package primarily used for research on proteins, lipids, and polymers. The package can also be applied to a wide variety of chemical and biological research questions. Additional Notes: GROMACS 5.X represents the newest version of GROMACS and ha..."

2022-10-17T17:43:25Z

Created page with "<h1>GROMACS</h1> __TOC__ Description: GROMACS is a highly optimized, fast, Molecular Dynamics (MD) simulation package primarily used for research on proteins, lipids, and polymers. The package can also be applied to a wide variety of chemical and biological research questions. Additional Notes: GROMACS 5.X represents the newest version of GROMACS and ha..."

New page

<h1>GROMACS</h1>
 
__TOC__
 
Description: GROMACS is a highly optimized, fast, Molecular Dynamics (MD) simulation package primarily used for research on proteins, lipids, and polymers. The package can also be applied to a wide variety of chemical and biological research questions.

Additional Notes: GROMACS 5.X represents the newest version of GROMACS and has been optimized for both parallel MPI and GPU performance. This version is installed on PENZIAS and is recommended. GROMACS 4.X is also installed. There are significant differences between versions of GROMACS, which may affect the simulation process.

Availability: PENZIAS and APPEL
::MPI: Yes
::SMP: No
::GPU: Yes on PENZIAS, No on APPEL

Module file: gromacs

Citation: Include in published paper the following citation regarding use of GROMACS:
1. Berendsen, et al. (1995) Comp. Phys. Comm. 91: 43-56, Lindahl, et al. (2001) J. Mol. Model. 7: 306-317, Van der Spoel, et al. (2005) J. Comput. Chem. 26: 1701-1718, Hess, et al. (2008) J. Chem. Theory Comput. 4: 435-447, Pronk, et al. (2013) Bioinformatics 29 845-854.

Documentation: http://www.gromacs.org/Documentation
 Tutorials: http://www.gromacs.org/Documentation/Tutorials

Related Packages:
::LAMMPS
::NWChem
::TINKER

General notes of use: 
The GROMACS package comes with a set of tools, which can be used to prepare the molecular system for simulation. genbox tool is no longer available in GROMACS 5.X. Also, the tools in GROMACS 5.X cannot be used directly, (as they were with GROMACS 4.X), but via wrapper called gmx (i.e. gmx pdb2gmx). Users should consult the tutorial(s)/documentation specific for the particular version. Also there are significant changes in algorithms used in GROMACS 5.X vs. those in GROMACS 4.X.

GROMACS parallelization schemes: 
Version 5.X on PENZIAS utilizes MPI parallelism, OpenMP parallelism and GPU-direct mechanism (P2P) to run simulations across several GPUs from different nodes. GROMACS 4.X installed on ANDY utilizes OpenMP (in node parallelism, SMP) and MPI parallelism. The GPU enabled 4.X version on PENZIAS utilizes MPI and OpenMP parallelism as well, but also uses GPUs as accelerator.

Use of GPUs: 
In the GPU enabled version of GROMACS, the Particle Mesh Ewald (PME) calculations are always done on CPU, while particle-particle (PP) interactions are done on GPU. As such, the optimal ratio between number of CPUs and number of GPUs depends on type of molecular system simulated. For X86_64 architecture the recommended number of atoms per CPU is typically about 100 to 150. As such, and as a rule of thumb, for a computer with K20 GPUs the ratio between the number of CPUs and GPUs should be at least 4:1, but the optimal number depends on molecular system simulated.

Use: 


PENZIAS:
module load gromacs/x.x.x

ANDY:
module load gromacs


For meaning and proper choice of values in <chunks> and <cpus> fields please read the section [[Submitting_Jobs|“Writing a job submit script.”]]

Example: The following example PBS scripts are described below: (1) General Script Template (2) Penzias (a) 4 cpus and 1 gpu, (b) 4 cpus and 2 gpus (c) 2 mpi threads per gpu and a submit script for (3) Andy.
==General submit script template==

The sample submit script should be modified based upon sysem (PENZIAS or ANDY) and desired CPU/GPU usage.

#!/bin/bash
#SBATCH --job-name water_job
#SBATCH --partition production
#SBATCH --nodes=<chunks>
#SBATCH --ntasks=<cpus>
#SBATCH --gres=gpu=<gpus>
#SBATCH --mem=<memory_mb>

# You must explicitly change to your working directory in SLURM

cd $SLURM_SUBMIT_DIR

mpirun -np <chunks*cpus> gmx mdrun –s topol.tpr –o md_para.trr –c md_para.gro –e md_para.edr –g md_para.log –gpu_id 0 > water.out 2>&1

==PENZIAS==
=====A. Example Submit Script: 4 CPUs and 1 GPU=====
In the script below, the PBS scheduler is asked to schedule 4 CPUs and 1 GPU for a job.

On PENZIAS, there are 2 versions of GROMACS which are GPU enabled: GROMACS 5.X and GROMACS 4.X. The 5.x uses more advanced communication protocol between GPU and has improved CPU performance. The following PBS script can be used as a model on how to run a MPI parallel GROMACS 5.X job with GPUs on PENZIAS. First, load the module for GROMACS with command:
module load gromacs/5.0.4

#!/bin/bash
#SBATCH --job-name water_job
#SBATCH --partition production
#SBATCH --nodes=<chunks>
#SBATCH --ntasks=<cpus>
#SBATCH --gres=gpu:1
#SBATCH --mem=3660mb

# You must explicitly change to your working directory in SLURM

cd $SLURM_SUBMIT_DIR

mpirun -np 4 gmx mdrun –s topol.tpr –o md_para.trr –c md_para.gro –e md_para.edr –g md_para.log –gpu_id 0 > water.out 2>&1

=====B. Example Submit Script: 4 CPUs and 2 GPUs=====
On PENZIAS, each virtual node has 2 GPUs. The following script shows how to use 2 GPUs:

#!/bin/bash
#SBATCH --job-name water_job
#SBATCH --partition production
#SBATCH --nodes=<chunks>
#SBATCH --ntasks=<cpus>
#SBATCH --gres=gpu:1
#SBATCH --mem=3660

cd $SLURM_SUBMIT_DIR

mpirun -np 32 gmx mdrun –s topol.tpr –o md_para.trr –c md_para.gro –e md_para.edr –g md_para.log –gpu_id 01 > water.out 2>&1

The above script will allocate 32 CPUs and 8 GPUs across several nodes.

=====C. Example Submit Script: 2 MPI threads per GPU=====
For some molecular systems is beneficial to submit two MPI threads per GPU. On PENZIAS that can be achieved with small change in above script in the field of –gpu_id (i.e., adding 0011):

#!/bin/bash
#SBATCH --job-name water_job
#SBATCH --partition production
#SBATCH --nodes=<chunks>
#SBATCH --ntasks=<cpus>
#SBATCH --gres=gpu:1
#SBATCH --mem=3660

cd $SLURM_SUBMIT_DIR

mpirun -np 32 gmx mdrun –s topol.tpr –o md_para.trr –c md_para.gro –e md_para.edr –g md_para.log –gpu_id 0011 > water.out 2>&1

==APPEL==
The following PBS script can be used as a model how to run MPI only simulations with GROMACS 4.X on ANDY. First load the module for the GROMACS with the command:
module load gromacs

The PBS script is as follows:

#!/bin/bash
#SBATCH --job-name GRMX_32bit
#SBATCH --partition production_qdr
#SBATCH --nodes=16
#SBATCH --ntasks
#SBATCH --mem=2880

# You must explicitly change to your working directory in SLURM

cd $SLURM_SUBMIT_DIR

mpirun -np 16 mdrun_mpi -px -pf -s md_para.tpr -o md_para.trr -c md_para.gro -e md_para.edr -g md_para.log > GRMX_32bit.out 2>&1

In the above example, the topology file (.tpr), trajectory file (.trr) coordinates file (.gro) etc. are all prepared with GROMACS tools (not shown here).

Applications Environment/gromacs - Revision history

James: Text replacement - "[pP][bB][sS]" to "SLURM"

James: Created page with "

GROMACS