James: Text replacement - "[pP][bB][sS]" to "SLURM"

2022-10-27T19:46:03Z

Text replacement - "[pP][bB][sS]" to "SLURM"

← Older revision		Revision as of 19:46, 27 October 2022
Line 51:		Line 51:
	</pre>		</pre>

	Next, create a ~~PBS~~ script with the following lines in it to be used to submit the serial		Next, create a SLURM script with the following lines in it to be used to submit the serial
	ADCIRC job to the Cray (SALK) ~~PBS~~ queues. Note that on SALK running a serial job		ADCIRC job to the Cray (SALK) SLURM queues. Note that on SALK running a serial job
	requires allocating (and wasting most of) 16 processors because fractional compute		requires allocating (and wasting most of) 16 processors because fractional compute
	nodes cannot be allocated on SALK.		nodes cannot be allocated on SALK.
Line 58:		Line 58:
	<pre>		<pre>
	#!/bin/bash		#!/bin/bash
	#~~PBS~~ -q production		#SLURM -q production
	#~~PBS~~ -N SADCIRC.test		#SLURM -N SADCIRC.test
	#~~PBS~~ -l select=16:ncpus=1:mem=2048mb		#SLURM -l select=16:ncpus=1:mem=2048mb
	#~~PBS~~ -l place=free		#SLURM -l place=free
	#~~PBS~~ -j oe		#SLURM -j oe
	#~~PBS~~ -o sadcirc.out		#SLURM -o sadcirc.out
	#~~PBS~~ -V		#SLURM -V

	# Find out name of master execution host (compute node)		# Find out name of master execution host (compute node)
	echo ""		echo ""
	echo -n ">>>> ~~PBS~~ Master compute node is: "		echo -n ">>>> SLURM Master compute node is: "
	hostname		hostname

	# You must explicitly change to the working directory in ~~PBS~~		# You must explicitly change to the working directory in SLURM
	cd $~~PBS_O_WORKDIR~~		cd $SLURM_O_WORKDIR

	echo ">>>> Begin ADCRIC Serial Run ..."		echo ">>>> Begin ADCRIC Serial Run ..."
Line 79:		Line 79:
	</pre>		</pre>

	And finally to submit the serial job to the ~~PBS~~ queue enter:		And finally to submit the serial job to the SLURM queue enter:

	<pre>		<pre>
Line 206:		Line 206:
	</pre>		</pre>

	Create a ~~PBS~~ script with the following lines in it to be used to submit the parallel		Create a SLURM script with the following lines in it to be used to submit the parallel
	ADCIRC job to the Cray (SALK) ~~PBS~~ queues:		ADCIRC job to the Cray (SALK) SLURM queues:

	<pre>		<pre>
	#!/bin/bash		#!/bin/bash
	#~~PBS~~ -q production		#SLURM -q production
	#~~PBS~~ -N PADCIRC.test		#SLURM -N PADCIRC.test
	#~~PBS~~ -l select=16:ncpus=1:mem=2048mb		#SLURM -l select=16:ncpus=1:mem=2048mb
	#~~PBS~~ -l place=free		#SLURM -l place=free
	#~~PBS~~ -j oe		#SLURM -j oe
	#~~PBS~~ -o padcirc.out		#SLURM -o padcirc.out
	#~~PBS~~ -V		#SLURM -V

	# Find out name of master execution host (compute node)		# Find out name of master execution host (compute node)
	echo ""		echo ""
	echo -n ">>>> ~~PBS~~ Master compute node is: "		echo -n ">>>> SLURM Master compute node is: "
	hostname		hostname

	# Change to working directory		# Change to working directory
	cd $~~PBS_O_WORKDIR~~		cd $SLURM_O_WORKDIR

	echo ">>>> Begin PADCRIC MPI Parallel Run ..."		echo ">>>> Begin PADCRIC MPI Parallel Run ..."
Line 232:		Line 232:
	</pre>		</pre>

	And finally to submit the parallel job to the ~~PBS~~ queue enter:		And finally to submit the parallel job to the SLURM queue enter:

	<pre>		<pre>
Line 241:		Line 241:
	of ADCIRC and SWAN to include surface wave affects in the simulation. This		of ADCIRC and SWAN to include surface wave affects in the simulation. This
	executable is called 'padcswan' and can be run with largely the same preparation		executable is called 'padcswan' and can be run with largely the same preparation
	steps and the same ~~PBS~~ script shown above for 'padcirc'. Details on the minor		steps and the same SLURM script shown above for 'padcirc'. Details on the minor
	differences and additional input files required are available at the SWAN websites		differences and additional input files required are available at the SWAN websites
	given above.		given above.

James: Created page with "= ADCIRC = ADCIRC is a system of programs for solving time-dependent, free-surface, circulation and transport problems in two and three dimensions. These programs utilize the finite element method in space allowing the use of highly flexible, unstructured grids. The ADCIRC distribution includes and integrates the METIS tool for unstructured grid generation. In addition, ADCIRC includes a distribution of SWAN to which it can be coupled to add a shore wave simulation mode..."

2022-10-17T17:23:59Z

Created page with "= ADCIRC = ADCIRC is a system of programs for solving time-dependent, free-surface, circulation and transport problems in two and three dimensions. These programs utilize the finite element method in space allowing the use of highly flexible, unstructured grids. The ADCIRC distribution includes and integrates the METIS tool for unstructured grid generation. In addition, ADCIRC includes a distribution of SWAN to which it can be coupled to add a shore wave simulation mode..."

New page

= ADCIRC =
ADCIRC is a system of programs for solving time-dependent, free-surface, circulation and transport problems in
two and three dimensions. These programs utilize the finite element method in space allowing the use of highly flexible,
unstructured grids. The ADCIRC distribution includes and integrates the METIS tool for unstructured grid generation.
In addition, ADCIRC includes a distribution of SWAN to which it can be coupled to add a shore wave simulation model.

Typical ADCIRC applications have included: (i) modeling tides and wind driven circulation, (ii) analysis of hurricane storm surge
and flooding, (iii) dredging feasibility and material disposal studies, (iv) larval transport studies, (v) near shore marine
operations. For more detail on using ADCIRC, please visit the ADCIRC website here [http://adcirc.org/index.html] and read
the ADCIRC manual [http://adcirc.org/documentv49/ADCIRC_title_page.html]. Details on using SWAN with ADCIRC
can be found here [http://www.caseydietrich.com/swanadcirc] and at the SWAN web site [http://swanmodel.sourceforge.net].

The CUNY HPC Center has installed version 50.79 on SALK (the Cray) and ANDY (the SGI) for general academic use. ADCIRC
can be run in serial or MPI-parallel mode on either system. ADCIRC has demonstrated good scaling properties up to 512
cores on SALK and 64 cores on ANDY. A step-by-step walk through of running an ADCIRC test case in both serial and parallel
mode follows.

==== Serial Execution ====

Create a directory where all the files needed to run the serial ADCIRC job will be kept.

<pre>
salk$ mkdir test_sadcirc
salk$ cd test_sadcirc
</pre>

Copy the Shinnecok Inlet example from ADCIRC installation tree and unzip it.

<pre>
salk$ cp /share/apps/adcirc/default/testcase/serial_shinnecock_inlet.zip ./
salk$ unzip ./serial_shinnecock_inlet.zip
Archive: ./serial_shinnecock_inlet.zip
inflating: serial_shinnecock_inlet/fort.14
inflating: serial_shinnecock_inlet/fort.15
inflating: serial_shinnecock_inlet/fort.16
inflating: serial_shinnecock_inlet/fort.63
inflating: serial_shinnecock_inlet/fort.64
</pre>

Change into the unpacked subdirectory.

<pre>
salk$ cd serial_shinnecock_inlet/
</pre>

There you should find the following files:

<pre>
salk$ ls
fort.14 fort.15 fort.16 fort.63 fort.64
</pre>

Next, create a PBS script with the following lines in it to be used to submit the serial
ADCIRC job to the Cray (SALK) PBS queues. Note that on SALK running a serial job
requires allocating (and wasting most of) 16 processors because fractional compute
nodes cannot be allocated on SALK.

<pre>
#!/bin/bash
#PBS -q production
#PBS -N SADCIRC.test
#PBS -l select=16:ncpus=1:mem=2048mb
#PBS -l place=free
#PBS -j oe
#PBS -o sadcirc.out
#PBS -V

# Find out name of master execution host (compute node)
echo ""
echo -n ">>>> PBS Master compute node is: "
hostname

# You must explicitly change to the working directory in PBS
cd $PBS_O_WORKDIR

echo ">>>> Begin ADCRIC Serial Run ..."
aprun -n 1 /share/apps/adcirc/default/bin/adcirc
echo ">>>> End ADCRIC Serial Run ..."
</pre>

And finally to submit the serial job to the PBS queue enter:

<pre>
salk$ qsub sadcirc.job
</pre>

==== Parallel Execution ====

The steps required to run ADCIRC in parallel include some additional mesh partitioning
and decomposition steps based on the number processors planned for the job. As before,
create a directory where all the files needed for the job will be kept:

<pre>
salk$ mkdir test_padcirc
salk$ cd test_padcirc
</pre>

Again, copy the Shinnecok Inlet example from ADCIRC installation tree and unzip it. The
starting point for the serial and parallel tests is the same, but for the parallel case the serial
data set used above is partitioned and decomposed for the parallel run.

<pre>
salk$ cp /share/apps/adcirc/default/testcase/serial_shinnecock_inlet.zip ./
salk$ unzip ./serial_shinnecock_inlet.zip
Archive: ./serial_shinnecock_inlet.zip
inflating: serial_shinnecock_inlet/fort.14
inflating: serial_shinnecock_inlet/fort.15
inflating: serial_shinnecock_inlet/fort.16
inflating: serial_shinnecock_inlet/fort.63
inflating: serial_shinnecock_inlet/fort.64
</pre>

Rename and change into directory you just unpacked:

<pre>
salk$ mv serial_shinnecock_inlet parallel_shinnecock_inlet
salk$ cd parallel_shinnecock_inlet/
</pre>

Now we need to run the ADCIRC preparation program 'adcprep' to partition the serial domain
and decompose problem:

<pre>
salk$ /share/apps/adcirc/default/bin/adcprep
</pre>

When prompted, enter 8 for number of processors to be used in our parallel example here:

<pre>

*****************************************
ADCPREP Fortran90 Version 2.3 10/18/2006
Serial version of ADCIRC Pre-processor
*****************************************

Input number of processors for parallel ADCIRC run:
8

</pre>

Next, enter 1 to complete partitioning the domain for 8 processors using METIS:

<pre>

#-------------------------------------------------------
Preparing input files for subdomains.
Select number or action:
1. partmesh
- partition mesh using metis ( perform this first)

2. prepall
- Full pre-process using default names (i.e., fort.14)

...

#-------------------------------------------------------

calling: prepinput

use_default = F
partition = T
prep_all = F
prep_15 = F
prep_13 = F
hot_local = F
hot_global = F

</pre>

Next, provide that name of the unpartitioned file unzipped from the serial test case, fort.14:

<pre>
Enter the name of the ADCIRC UNIT 14 (Grid) file:
fort.14
</pre>

This will generate some additional output to your terminal and complete the mesh partition step.

You must then run 'adcprep' again to decompose the problem. When prompted enter 8, number
of processors as before, but this time followed by a 2 to decompose the problem. When this preparation
step completes you will find the following files and directories in your working directory:

<pre>
salk$ ls
fort.15 fort.63 fort.80 partmesh.txt PE0001 PE0003 PE0005 PE0007
fort.14 fort.16 fort.64 metis_graph.txt PE0000 PE0002 PE0004 PE0006
</pre>

The 8 subdirectories created in the second 'adcprep' run contain the partitioned and decomposed
problem that each MPI processor (8 in this case) will work on.

Copy the parallel ADCIRC binary to the working directory.

<pre>
# cp /share/apps/adcirc/default/bin/padcirc ./
</pre>

At this point you'll have all the files needed to run the parallel job. The
files and directories created and required for this 8 core parallel run are
shown here:

<pre>
# ls
adc fort.14 fort.15 fort.16 fort.80 metis_graph.txt partmesh.txt
PE0000/ PE0001/ PE0002/ PE0003/ PE0004/ PE0005/ PE0006/ PE0007/
</pre>

Create a PBS script with the following lines in it to be used to submit the parallel
ADCIRC job to the Cray (SALK) PBS queues:

<pre>
#!/bin/bash
#PBS -q production
#PBS -N PADCIRC.test
#PBS -l select=16:ncpus=1:mem=2048mb
#PBS -l place=free
#PBS -j oe
#PBS -o padcirc.out
#PBS -V

# Find out name of master execution host (compute node)
echo ""
echo -n ">>>> PBS Master compute node is: "
hostname

# Change to working directory
cd $PBS_O_WORKDIR

echo ">>>> Begin PADCRIC MPI Parallel Run ..."
aprun -n 8 /share/apps/adcirc/default/bin/padcirc
echo ">>>> End PADCRIC MPI Parallel Run ..."
</pre>

And finally to submit the parallel job to the PBS queue enter:

<pre>
salk$ qsub padcirc.job
</pre>

The CUNY HPC Center has also built and provided a parallel-coupled version
of ADCIRC and SWAN to include surface wave affects in the simulation. This
executable is called 'padcswan' and can be run with largely the same preparation
steps and the same PBS script shown above for 'padcirc'. Details on the minor
differences and additional input files required are available at the SWAN websites
given above.

Applications Environment/ADCIRC - Revision history

James: Text replacement - "[pP][bB][sS]" to "SLURM"