LSDYNA
Both 32-bit and 64-bit executables in both serial and parallel versions are provided. The MPI parallel versions use OpenMPI as their MPI parallel library, the HPC Center's default version of MPI. The serial executable can also be run in OpenMP (not to be confused with OpenMPI) node- local SMP-parallel mode. The names of the executable files in the '/share/apps/lsdyna/default/bin' directory are:
ls-dyna_32.exe ls-dyna_64.exe ls-dyna-mpp32.exe ls-dyna_mpp64.exe
Those with the string 'mpp' in the name are the MPI distributed parallel versions of the code. The integer (32 or 64) designates the precision of the build. In the examples below, depending on the type of script being submitted (serial or parallel, 32- or 64-bit), a different executable will be chosen. The scaling properties of LS-DYNA in parallel mode are limited, and users should not carelessly submit parallel jobs requesting large numbers of cores without understanding how their job will scale. A large 128 core job that runs only 5% faster than a 64 core job is a waste of resources. Please examine the scaling properties of your particular job before scaling up.
As is the case with most long running applications run at the CUNY HPC Center, whether parallel or serial, LS-DYNA jobs are run using a SLURM batch job submission script. Here we provide some example scripts for both serial and parallel execution.
Note that before using this script you will need to setup the environment for LS-DYNA. On Andy
"modules" is used to manage environments. Setting up LS-DYNA is done with
module load ls-dyna
First an example serial execution script (called say 'airbag.job') run at 64-bits using the LS-DYNA 'airbag' example ('airbag.deplo.k') from the examples directory above as the input.
#!/bin/bash #SBATCH --partition production_qdr #SBATCH --job-name ls-dyna_serial #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --mem=2880 # Find out name of master execution host (compute node) echo -n ">>>> SLURM Master compute node is: " hostname # You must explicitly change to the working directory in SLURM cd $SLURM_SUBMIT_DIR # Point to the execution directory to run echo ">>>> Begin LS-DYNA Serial Run ..." ls-dyna_64.exe i=airbag.deploy.k memory=2000m echo ">>>> End LS-DYNA Serial Run ..."
Details on the SLURM options at the head of the this script file are discussion below, but in summary '-q production_qdr' selects the routing queue into which the job will be placed, '-N ls-dyna_serial' sets this job's name, 'nodes=1 ntasks=1 mem=2880' nodes-requests 1 SLURM resource chunk that includes 1 cpu and 2880 MBytes of memory, allows SLURM to put the resources needed for the job any where on the ANDY system.
The LS-DYNA command line sets the input file to be used and the amount of in-core memory that is available to the job. Note that this executable does NOT include the string 'mpp' which means that it is not the MPI executable. Users can copy the 'airbag.deploy.k' file from the examples directory and cut-and-paste this script to run this job. It takes a relatively sort time to run. The SLURM command for submitting the job would be:
qsub airbag.job
Here is a SLURM script that runs a 16 processor (core) MPI job. This script is set to run the TopCrunch '3cars' benchmark which is relatively long-running using MPI on 16 processors. There are a few important differences in this script.
#!/bin/bash #SBATCH --partition production_qdr #SBATCH --job name ls-dyna_mpi #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --mem=2880 # Find out name of master execution host (compute node) echo -n ">>>> SLURM Master compute node is: " hostname # You must explicitly change to the working directory in SLURM cd $SLURM_SUBMIT_DIR # Point to the execution directory to run echo ">>>> Begin LS-DYNA MPI Parallel Run ..." mpirun -np 16 ls-dyna_mpp64.exe i=3cars_shell2_150ms.k ntasks=16 memory=2000m echo ">>>> End LS-DYNA MPI Parallel Run ..."
Focusing on the difference in this script relative to the serial SLURM script above. First, the '-l select' line requests not 1 SLURM resource chunk, but 16 each with 1 cpu (core) and 2880 MBytes of memory. This provides the necessary resources to run our 16 processor MPI-parallel job. Next, the LS-DYNA command line is different. The LS-DYNA MPI-parallel executable is used (ls-dyna_mpp64.exe), and it is run with the help of the OpenMPI job submission command 'mpirun' which sets the number of processors and the location of those processors on the system. The actually LS-DYNA key words also add the string 'ncpu=16' to instruct LS-DYNA that this is to be a parallel run.
Running in parallel on 16 cores in 64-bit mode on ANDY, the '3cars' case takes about 9181 seconds of elapsed time to complete. If the user would like to run this job, they can grab the input files out of the directory '/share/apps/lsdyna/default/examples/3cars' on ANDY and use the above script.