Applications Environment/gromacs

GROMACS

Description: GROMACS is a highly optimized, fast, Molecular Dynamics (MD) simulation package primarily used for research on proteins, lipids, and polymers. The package can also be applied to a wide variety of chemical and biological research questions.

Additional Notes: GROMACS 5.X represents the newest version of GROMACS and has been optimized for both parallel MPI and GPU performance. This version is installed on PENZIAS and is recommended. GROMACS 4.X is also installed. There are significant differences between versions of GROMACS, which may affect the simulation process.

Availability: PENZIAS and APPEL

MPI: Yes

SMP: No

GPU: Yes on PENZIAS, No on APPEL

Module file: gromacs

Citation: Include in published paper the following citation regarding use of GROMACS: 1. Berendsen, et al. (1995) Comp. Phys. Comm. 91: 43-56, Lindahl, et al. (2001) J. Mol. Model. 7: 306-317, Van der Spoel, et al. (2005) J. Comput. Chem. 26: 1701-1718, Hess, et al. (2008) J. Chem. Theory Comput. 4: 435-447, Pronk, et al. (2013) Bioinformatics 29 845-854.

Documentation: http://www.gromacs.org/Documentation
Tutorials: http://www.gromacs.org/Documentation/Tutorials

Related Packages:

LAMMPS

NWChem

TINKER

General notes of use: The GROMACS package comes with a set of tools, which can be used to prepare the molecular system for simulation. genbox tool is no longer available in GROMACS 5.X. Also, the tools in GROMACS 5.X cannot be used directly, (as they were with GROMACS 4.X), but via wrapper called gmx (i.e. gmx pdb2gmx). Users should consult the tutorial(s)/documentation specific for the particular version. Also there are significant changes in algorithms used in GROMACS 5.X vs. those in GROMACS 4.X.

GROMACS parallelization schemes: Version 5.X on PENZIAS utilizes MPI parallelism, OpenMP parallelism and GPU-direct mechanism (P2P) to run simulations across several GPUs from different nodes. GROMACS 4.X installed on ANDY utilizes OpenMP (in node parallelism, SMP) and MPI parallelism. The GPU enabled 4.X version on PENZIAS utilizes MPI and OpenMP parallelism as well, but also uses GPUs as accelerator.

Use of GPUs: In the GPU enabled version of GROMACS, the Particle Mesh Ewald (PME) calculations are always done on CPU, while particle-particle (PP) interactions are done on GPU. As such, the optimal ratio between number of CPUs and number of GPUs depends on type of molecular system simulated. For X86_64 architecture the recommended number of atoms per CPU is typically about 100 to 150. As such, and as a rule of thumb, for a computer with K20 GPUs the ratio between the number of CPUs and GPUs should be at least 4:1, but the optimal number depends on molecular system simulated.

Use:

PENZIAS:

module load gromacs/x.x.x

ANDY:

module load gromacs

For meaning and proper choice of values in <chunks> and <cpus> fields please read the section “Writing a job submit script.”

Example: The following example SLURM scripts are described below: (1) General Script Template (2) Penzias (a) 4 cpus and 1 gpu, (b) 4 cpus and 2 gpus (c) 2 mpi threads per gpu and a submit script for (3) Andy.

General submit script template

The sample submit script should be modified based upon sysem (PENZIAS or ANDY) and desired CPU/GPU usage.

#!/bin/bash 
#SBATCH --job-name water_job 
#SBATCH --partition production 
#SBATCH --nodes=<chunks>
#SBATCH --ntasks=<cpus>
#SBATCH --gres=gpu=<gpus>
#SBATCH --mem=<memory_mb> 
   

# You must explicitly change to your working directory in SLURM 

cd $SLURM_SUBMIT_DIR  

mpirun -np <chunks*cpus> gmx mdrun –s topol.tpr –o md_para.trr –c md_para.gro –e md_para.edr –g md_para.log –gpu_id 0 > water.out 2>&1

PENZIAS

A. Example Submit Script: 4 CPUs and 1 GPU

In the script below, the SLURM scheduler is asked to schedule 4 CPUs and 1 GPU for a job.

On PENZIAS, there are 2 versions of GROMACS which are GPU enabled: GROMACS 5.X and GROMACS 4.X. The 5.x uses more advanced communication protocol between GPU and has improved CPU performance. The following SLURM script can be used as a model on how to run a MPI parallel GROMACS 5.X job with GPUs on PENZIAS. First, load the module for GROMACS with command:

module load gromacs/5.0.4

#!/bin/bash 
#SBATCH --job-name water_job 
#SBATCH --partition production 
#SBATCH --nodes=<chunks>
#SBATCH --ntasks=<cpus>
#SBATCH --gres=gpu:1
#SBATCH --mem=3660mb   

# You must explicitly change to your working directory in SLURM 

cd $SLURM_SUBMIT_DIR  

mpirun -np 4 gmx mdrun –s topol.tpr –o md_para.trr –c md_para.gro –e md_para.edr –g md_para.log –gpu_id 0 > water.out 2>&1

B. Example Submit Script: 4 CPUs and 2 GPUs

On PENZIAS, each virtual node has 2 GPUs. The following script shows how to use 2 GPUs:

#!/bin/bash 
#SBATCH --job-name water_job 
#SBATCH --partition production 
#SBATCH --nodes=<chunks>
#SBATCH --ntasks=<cpus>
#SBATCH --gres=gpu:1
#SBATCH --mem=3660 

cd $SLURM_SUBMIT_DIR   

mpirun -np 32 gmx mdrun –s topol.tpr –o md_para.trr –c md_para.gro –e md_para.edr –g md_para.log –gpu_id 01 > water.out 2>&1 

The above script will allocate 32 CPUs and 8 GPUs across several nodes.

C. Example Submit Script: 2 MPI threads per GPU

For some molecular systems is beneficial to submit two MPI threads per GPU. On PENZIAS that can be achieved with small change in above script in the field of –gpu_id (i.e., adding 0011):

#!/bin/bash 
#SBATCH --job-name water_job 
#SBATCH --partition production 
#SBATCH --nodes=<chunks>
#SBATCH --ntasks=<cpus>
#SBATCH --gres=gpu:1
#SBATCH --mem=3660 

cd $SLURM_SUBMIT_DIR   

mpirun -np 32 gmx mdrun –s topol.tpr –o md_para.trr –c md_para.gro –e md_para.edr –g md_para.log –gpu_id 0011 > water.out 2>&1

APPEL

The following SLURM script can be used as a model how to run MPI only simulations with GROMACS 4.X on ANDY. First load the module for the GROMACS with the command:

module load gromacs 

The SLURM script is as follows:

#!/bin/bash 
#SBATCH --job-name GRMX_32bit 
#SBATCH --partition production_qdr 
#SBATCH --nodes=16
#SBATCH --ntasks
#SBATCH --mem=2880 
  

# You must explicitly change to your working directory in SLURM 

cd $SLURM_SUBMIT_DIR   
 

mpirun -np 16 mdrun_mpi -px -pf -s md_para.tpr -o md_para.trr -c md_para.gro -e md_para.edr  -g md_para.log > GRMX_32bit.out 2>&1 

In the above example, the topology file (.tpr), trajectory file (.trr) coordinates file (.gro) etc. are all prepared with GROMACS tools (not shown here).