James: Created page with "The CUNY HPC Center has installed version 50.79 on SALK (the Cray) and ANDY (the SGI) for general academic use. ADCIRC can be run in serial or MPI-parallel mode on either system. ADCIRC has demonstrated good scaling properties up to 512 cores on SALK and 64 cores on ANDY. A step-by-step walk through of running an ADCIRC test case in both serial and parallel mode follows. ==== Serial Execution ==== Create a directory where all the files needed to run the serial ADCIR..."

2022-10-17T17:23:15Z

Created page with "The CUNY HPC Center has installed version 50.79 on SALK (the Cray) and ANDY (the SGI) for general academic use. ADCIRC can be run in serial or MPI-parallel mode on either system. ADCIRC has demonstrated good scaling properties up to 512 cores on SALK and 64 cores on ANDY. A step-by-step walk through of running an ADCIRC test case in both serial and parallel mode follows. ==== Serial Execution ==== Create a directory where all the files needed to run the serial ADCIR..."

New page

The CUNY HPC Center has installed version 50.79 on SALK (the Cray) and ANDY (the SGI) for general academic use. ADCIRC
can be run in serial or MPI-parallel mode on either system. ADCIRC has demonstrated good scaling properties up to 512
cores on SALK and 64 cores on ANDY. A step-by-step walk through of running an ADCIRC test case in both serial and parallel
mode follows.

==== Serial Execution ====

Create a directory where all the files needed to run the serial ADCIRC job will be kept.

<pre>
salk$ mkdir test_sadcirc
salk$ cd test_sadcirc
</pre>

Copy the Shinnecok Inlet example from ADCIRC installation tree and unzip it.

<pre>
salk$ cp /share/apps/adcirc/default/testcase/serial_shinnecock_inlet.zip ./
salk$ unzip ./serial_shinnecock_inlet.zip
Archive: ./serial_shinnecock_inlet.zip
inflating: serial_shinnecock_inlet/fort.14
inflating: serial_shinnecock_inlet/fort.15
inflating: serial_shinnecock_inlet/fort.16
inflating: serial_shinnecock_inlet/fort.63
inflating: serial_shinnecock_inlet/fort.64
</pre>

Change into the unpacked subdirectory.

<pre>
salk$ cd serial_shinnecock_inlet/
</pre>

There you should find the following files:

<pre>
salk$ ls
fort.14 fort.15 fort.16 fort.63 fort.64
</pre>

Next, create a SLURM script with the following lines in it to be used to submit the serial
ADCIRC job to the Cray (SALK) SLURM queues. Note that on SALK running a serial job
requires allocating (and wasting most of) 16 processors because fractional compute
nodes cannot be allocated on SALK.

<pre>
#!/bin/bash
#SBATCH --partition production
#SBATCH --job-name SADCIRC.test
#SBATCH --nodes=16
#SBATCH --ntasks=1
#SBATCH --mem=2048
#SBATCH --o sadcirc.out

# Find out name of master execution host (compute node)
echo ""
echo -n ">>>> SBATCH Master compute node is: "
hostname

# You must explicitly change to the working directory in SBATCH
cd $SLURM_SUBMIT_DIR

echo ">>>> Begin ADCRIC Serial Run ..."
aprun -n 1 /share/apps/adcirc/default/bin/adcirc
echo ">>>> End ADCRIC Serial Run ..."
</pre>

And finally to submit the serial job to the SLURM queue enter:

<pre>
salk$ qsub sadcirc.job
</pre>

==== Parallel Execution ====

The steps required to run ADCIRC in parallel include some additional mesh partitioning
and decomposition steps based on the number processors planned for the job. As before,
create a directory where all the files needed for the job will be kept:

<pre>
salk$ mkdir test_padcirc
salk$ cd test_padcirc
</pre>

Again, copy the Shinnecok Inlet example from ADCIRC installation tree and unzip it. The
starting point for the serial and parallel tests is the same, but for the parallel case the serial
data set used above is partitioned and decomposed for the parallel run.

<pre>
salk$ cp /share/apps/adcirc/default/testcase/serial_shinnecock_inlet.zip ./
salk$ unzip ./serial_shinnecock_inlet.zip
Archive: ./serial_shinnecock_inlet.zip
inflating: serial_shinnecock_inlet/fort.14
inflating: serial_shinnecock_inlet/fort.15
inflating: serial_shinnecock_inlet/fort.16
inflating: serial_shinnecock_inlet/fort.63
inflating: serial_shinnecock_inlet/fort.64
</pre>

Rename and change into directory you just unpacked:

<pre>
salk$ mv serial_shinnecock_inlet parallel_shinnecock_inlet
salk$ cd parallel_shinnecock_inlet/
</pre>

Now we need to run the ADCIRC preparation program 'adcprep' to partition the serial domain
and decompose problem:

<pre>
salk$ /share/apps/adcirc/default/bin/adcprep
</pre>

When prompted, enter 8 for number of processors to be used in our parallel example here:

<pre>

*****************************************
ADCPREP Fortran90 Version 2.3 10/18/2006
Serial version of ADCIRC Pre-processor
*****************************************

Input number of processors for parallel ADCIRC run:
8

</pre>

Next, enter 1 to complete partitioning the domain for 8 processors using METIS:

<pre>

#-------------------------------------------------------
Preparing input files for subdomains.
Select number or action:
1. partmesh
- partition mesh using metis ( perform this first)

2. prepall
- Full pre-process using default names (i.e., fort.14)

...

#-------------------------------------------------------

calling: prepinput

use_default = F
partition = T
prep_all = F
prep_15 = F
prep_13 = F
hot_local = F
hot_global = F

</pre>

Next, provide that name of the unpartitioned file unzipped from the serial test case, fort.14:

<pre>
Enter the name of the ADCIRC UNIT 14 (Grid) file:
fort.14
</pre>

This will generate some additional output to your terminal and complete the mesh partition step.

You must then run 'adcprep' again to decompose the problem. When prompted enter 8, number
of processors as before, but this time followed by a 2 to decompose the problem. When this preparation
step completes you will find the following files and directories in your working directory:

<pre>
salk$ ls
fort.15 fort.63 fort.80 partmesh.txt PE0001 PE0003 PE0005 PE0007
fort.14 fort.16 fort.64 metis_graph.txt PE0000 PE0002 PE0004 PE0006
</pre>

The 8 subdirectories created in the second 'adcprep' run contain the partitioned and decomposed
problem that each MPI processor (8 in this case) will work on.

Copy the parallel ADCIRC binary to the working directory.

<pre>
# cp /share/apps/adcirc/default/bin/padcirc ./
</pre>

At this point you'll have all the files needed to run the parallel job. The
files and directories created and required for this 8 core parallel run are
shown here:

<pre>
# ls
adc fort.14 fort.15 fort.16 fort.80 metis_graph.txt partmesh.txt
PE0000/ PE0001/ PE0002/ PE0003/ PE0004/ PE0005/ PE0006/ PE0007/
</pre>

Create a SLURM script with the following lines in it to be used to submit the parallel
ADCIRC job to the Cray (SALK) SLURM queues:

<pre>
#!/bin/bash
#SBATCH --partition production
#SBATCH --job-name PADCIRC.test
#SBATCH --nodes=16
#SBATCH --ntasks=1
#SBATCH --mem=2048
#SBATCH --o padcirc.out

# Find out name of master execution host (compute node)
echo ""
echo -n ">>>> SLURM Master compute node is: "
hostname

# Change to working directory
cd $SLURM_SUBMIT_DIR

echo ">>>> Begin PADCRIC MPI Parallel Run ..."
aprun -n 8 /share/apps/adcirc/default/bin/padcirc
echo ">>>> End PADCRIC MPI Parallel Run ..."
</pre>

And finally to submit the parallel job to the SLURM queue enter:

<pre>
salk$ qsub padcirc.job
</pre>

The CUNY HPC Center has also built and provided a parallel-coupled version
of ADCIRC and SWAN to include surface wave affects in the simulation. This
executable is called 'padcswan' and can be run with largely the same preparation
steps and the same SLURM script shown above for 'padcirc'. Details on the minor
differences and additional input files required are available at the SWAN websites
given in the introduction.

ADCIRC - Revision history