James: Text replacement - "[pP][bB][sS]" to "SLURM"

2022-10-27T19:31:36Z

Text replacement - "[pP][bB][sS]" to "SLURM"

← Older revision		Revision as of 19:31, 27 October 2022
Line 19:		Line 19:

	As is the case with most long running applications run at the CUNY HPC Center, whether parallel		As is the case with most long running applications run at the CUNY HPC Center, whether parallel
	or serial, LS-DYNA jobs are run using a ~~PBS~~ batch job submission script. Here we provide some		or serial, LS-DYNA jobs are run using a SLURM batch job submission script. Here we provide some
	example scripts for both serial and parallel execution.		example scripts for both serial and parallel execution.

James: Created page with " Both 32-bit and 64-bit executables in both serial and parallel versions are provided. The MPI parallel versions use OpenMPI as their MPI parallel library, the HPC Center's default version of MPI. The serial executable can also be run in OpenMP (not to be confused with OpenMPI) node- local SMP-parallel mode. The names of the executable files in the '/share/apps/lsdyna/default/bin' directory are:

 ls-dyna_32.exe  ls-dyna_64.exe  ls-dyna-mpp32.exe  ls-dyna_mpp64.e..."

2022-10-20T20:08:28Z

Created page with " Both 32-bit and 64-bit executables in both serial and parallel versions are provided. The MPI parallel versions use OpenMPI as their MPI parallel library, the HPC Center's default version of MPI. The serial executable can also be run in OpenMP (not to be confused with OpenMPI) node- local SMP-parallel mode. The names of the executable files in the '/share/apps/lsdyna/default/bin' directory are: <pre> ls-dyna_32.exe ls-dyna_64.exe ls-dyna-mpp32.exe ls-dyna_mpp64.e..."

New page

Both 32-bit and 64-bit executables in both serial and parallel versions are provided. The MPI
parallel versions use OpenMPI as their MPI parallel library, the HPC Center's default version of
MPI. The serial executable can also be run in OpenMP (not to be confused with OpenMPI) node-
local SMP-parallel mode. The names of the executable files in the '/share/apps/lsdyna/default/bin'
directory are:

<pre>
ls-dyna_32.exe ls-dyna_64.exe ls-dyna-mpp32.exe ls-dyna_mpp64.exe
</pre>

Those with the string 'mpp' in the name are the MPI distributed parallel versions of the code. The
integer (32 or 64) designates the precision of the build. In the examples below, depending on the
type of script being submitted (serial or parallel, 32- or 64-bit), a different executable will be
chosen. The scaling properties of LS-DYNA in parallel mode are limited, and users should not
carelessly submit parallel jobs requesting large numbers of cores without understanding how
their job will scale. A large 128 core job that runs only 5% faster than a 64 core job is a waste
of resources. Please examine the scaling properties of your particular job before scaling up.

As is the case with most long running applications run at the CUNY HPC Center, whether parallel
or serial, LS-DYNA jobs are run using a PBS batch job submission script. Here we provide some
example scripts for both serial and parallel execution.

Note that before using this script you will need to setup the environment for LS-DYNA. On Andy
"[[Main_Page#Modules.2C_Managing_Your_CUNY_HPC_Center_Environment|modules]]" is used to manage environments. Setting up LS-DYNA is done with
<pre>
module load ls-dyna
</pre>

First an example serial execution script
(called say 'airbag.job') run at 64-bits using the LS-DYNA 'airbag' example ('airbag.deplo.k') from
the examples directory above as the input.

<pre>
#!/bin/bash
#SBATCH --partition production_qdr
#SBATCH --job-name ls-dyna_serial
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mem=2880

# Find out name of master execution host (compute node)
echo -n ">>>> SLURM Master compute node is: "
hostname

# You must explicitly change to the working directory in SLURM
cd $SLURM_SUBMIT_DIR

# Point to the execution directory to run
echo ">>>> Begin LS-DYNA Serial Run ..."
ls-dyna_64.exe i=airbag.deploy.k memory=2000m
echo ">>>> End LS-DYNA Serial Run ..."
</pre>

Details on the SLURM options at the head of the this script file are discussion below,
but in summary '-q production_qdr' selects the routing queue into which the job will
be placed, '-N ls-dyna_serial' sets this job's name, 'nodes=1 ntasks=1 mem=2880'
nodes-requests 1 SLURM resource chunk that includes 1 cpu and 2880 MBytes of memory,
allows SLURM to put the resources needed for the job any where on the
ANDY system.

The LS-DYNA command line sets the input file to be used and the amount of in-core
memory that is available to the job. Note that this executable does NOT include the
string 'mpp' which means that it is not the MPI executable. Users can copy the 'airbag.deploy.k'
file from the examples directory and cut-and-paste this script to run this job. It
takes a relatively sort time to run. The SLURM command for submitting the job would be:

<pre>
qsub airbag.job
</pre>

Here is a SLURM script that runs a 16 processor (core) MPI job. This script is
set to run the TopCrunch '3cars' benchmark which is relatively long-running using
MPI on 16 processors. There are a few important differences in this script.

<pre>
#!/bin/bash
#SBATCH --partition production_qdr
#SBATCH --job name ls-dyna_mpi
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mem=2880

# Find out name of master execution host (compute node)
echo -n ">>>> SLURM Master compute node is: "
hostname

# You must explicitly change to the working directory in SLURM
cd $SLURM_SUBMIT_DIR

# Point to the execution directory to run
echo ">>>> Begin LS-DYNA MPI Parallel Run ..."
mpirun -np 16 ls-dyna_mpp64.exe i=3cars_shell2_150ms.k ntasks=16 memory=2000m
echo ">>>> End LS-DYNA MPI Parallel Run ..."
</pre>

Focusing on the difference in this script relative to the serial SLURM script above. First, the '-l select' line
requests not 1 SLURM resource chunk, but 16 each with 1 cpu (core) and 2880 MBytes of memory. This
provides the necessary resources to run our 16 processor MPI-parallel job. Next, the LS-DYNA command
line is different. The LS-DYNA MPI-parallel executable is used (ls-dyna_mpp64.exe), and it is run with
the help of the OpenMPI job submission command 'mpirun' which sets the number of processors
and the location of those processors on the system. The actually LS-DYNA key words also add the
string 'ncpu=16' to instruct LS-DYNA that this is to be a parallel run.

Running in parallel on 16 cores in 64-bit mode on ANDY, the '3cars' case takes about 9181 seconds
of elapsed time to complete. If the user would like to run this job, they can grab the input files out
of the directory '/share/apps/lsdyna/default/examples/3cars' on ANDY and use the above script.

LSDYNA - Revision history

James: Text replacement - "[pP][bB][sS]" to "SLURM"