- First download the correct binary,
You can download the LS-Dyna binaries here (password protected): https://ftp.lstc.com/user/mpp-dyna/
Note: You'll want to make sure you're using the MPP binaries and not the SMP ones for multi-instance runs. These are contained in two seperate folders on the FTP server. Navigate to
mpp-dyna/instead ofls-dyna/to grab them.
You'll need to download two files, as shown in the example below, you'll need the binary as well as the sharelib for each version you want. In this example I'm using Intel MPI 4.1.3 but the same applies for Open MPI.
$ cd /fsx
$ mkdir -p ls-dyna && cd ls-dyna
$ wget https://ftp.lstc.com/user/mpp-dyna/R11.0.0/x86-64/ifort_160_avx2/ls-dyna_mpp_s_R11_0_0_x64_centos65_ifort160_avx2_intelmpi-413_sharelib.tar.gz_extractor.sh --user user --password computer
$ wget https://ftp.lstc.com/user/mpp-dyna/R11.0.0/x86-64/ifort_160_avx2/ls-dyna_mpp_s_R11_0_0_x64_centos65_ifort160_avx2_intelmpi-413.tar.gz_extractor.sh --user user --password computer
- Install LS-Dyna by running the extractor script in the
/fsx/ls-dynadirectory. This will open the terms and conditions, type 'q' to continue and then select 'y' to agree then 'n' to not create a new folder.
$ bash ls-dyna_mpp_s_R11_0_0_x64_centos65_ifort160_avx2_intelmpi-413_sharelib.tar.gz_extractor.sh
# you should now see the extracted file, make sure to copy this path for later use.
$ ls
ls-dyna_mpp_s_r7_1_3_107967_x64_redhat54_ifort131_avx2_intelmpi-413- Import the application profile by going to Application Management > Import/Export, then download the file linked here (Ctrl+S or Command+S) as
soca_app_LS-Dyna.jsonand upload it like so:
Optional:
wget https://gist.githubusercontent.com/sean-smith/0a947c6af987c3f35b12d0aa3b995fa1/raw/b713fd8560b728c74dfd44829b3ce87e43d243b4/soca_app_LS-Dyna.json
- Navigate to Application Management > Edit Application Profile > LS-Dyna > Step 1: Design the form
Change the following values in the GUI form:
- binary_location point this to the path of the LS-Dyna install, see step 1
- scratch_location make sure you have
/fsxmounted - MPI Install MPI as a module file on an ami
- fsx_id Make sure this is a valid FSx filesystem in your account
- custom_ami this AMI needs to have intelmpi/openmpi (depending on what you're using) installed. See below
- Create a file
/apps/lstc/licenseon the scheduler instance for the license server that contains the following:
#LICENSE_TYPE:network
#LICENSE_SERVER:31111@10.0.0.1
- Create a directory in the home directory with the
.keyfile(s). This is important as the first step in the script is to copy that entire directory up to/fsxand then it runs the job from there. - Submit the job, you can use the web based simulator and submit it as follows:
Once you choose your input files, you’ll be presented with a screen that looks like:
- Choose Compute most of the time. this is a c5.24xlarge instance. It has 48 cores and 192 GB of RAM.
- Change the number of cores but make sure it’s a multiple of 48 (to utilize all the cores on the instance)
- You can change the job name to suite your liking, it defaults to the input file name. This will change the output directory as well. Job output is written to:
/fsx/[job_name]_[jobid]_[host] - Input Location: Leave as is.
- Queue: Use On Demand this ensure the instances are launched when your job is submitted then terminated when the job exits. The “Always On” queue is for debugging application configuration and is beyond the scope of this guide.
- Application Parameters:
- Binary Location: Select the version you want to run. Note we’ve benchmarked a bunch of different binaries and found that the best binaries are 9.3.1 for version 9 and 7.1.3 for version 7. See table below for different versions.
- Memory: Leave this as
2G. This is the decomposition memory. - Memory 2: Leave this as
400M - Scratch Location: Leave this as
/fsx - Version of MPI: Change this to match the binary’s MPI version. This must match!
After you submit you'll be taken to a screen with a job queue, the job will be in submitted, before it goes to capacity provisioning then running. If it doesn't start see this page.
- Navigate to
/fsxand search by the job ID:
- Go to the folder it created. You'll see a bunch of files, runtime info is contained in
output.log, click "view or edit icon on that page to see the results:
- In the output.log you'll see:
PBS Script
Note this is what gets added by the JSON file, however if you need to use it independent of SOCA or without the GUI, you can start with this script.
#!/bin/bash
#PBS -N %job_name%
#PBS -P %project_name%
# https://awslabs.github.io/scale-out-computing-on-aws/tutorials/integration-ec2-job-parameters/
#PBS -l instance_type=%instance_type%
#PBS -l instance_ami=%custom_ami%
#PBS -l ht_support=False
#PBS -l fsx_lustre=%fsx_id%
#PBS -q %queue_name%
#PBS -l system_metrics=True
INPUT_FILE=%input_file%
SCRATCH=%scratch_location%
# Create Working Directory
TMP_DIR=${SCRATCH}/${PBS_JOBNAME}_${PBS_JOBID}
mkdir -p $TMP_DIR
cp -R $(dirname "${INPUT_FILE}")/* $TMP_DIR
cd $TMP_DIR
# MPI Variables
export OMP_DYNAMIC=off
export KMP_AFFINITY="granularity=thread,balanced"
export I_MPI_FALLBACK=0
export I_MPI_PIN_PROCS=allcores
export I_MPI_PIN_MODE=lib
export I_MPI_EAGER_THRESHOLD=134000
export I_MPI_PIN_AFFINITY=sock
export I_MPI_ADJUST_COLLECTIVES="bcast:1\;alltoall:1\;allreduce:1-1024,2-500000"
export I_MPI_PIN_DOMAIN=omp
####### LICENSE ##########
export LSTC_FILE="/apps/lstc/license"
####### LICENSE ##########
cores=%cpus%
instances=$(cat $PBS_NODEFILE | uniq | wc -l)
ppn=$(echo ${cores}/${instances} | bc)
memory=%memory%
memory2=%memory2%
mpi=%mpi_version%
module load $mpi
if [[ $mpi =~ "intelmpi" ]]
then
mpiargs="-n ${cores} -ppn ${ppn} -hostfile $PBS_NODEFILE"
elif [[ $mpi =~ "openmpi/1.5.1" ]]
then
mpiargs="-np ${cores} -hostfile $PBS_NODEFILE"
else
mpiargs="-np ${cores} --map-by ppr:${ppn}:node -hostfile $PBS_NODEFILE --nooversubscribe"
fi
echo "MPI=$mpi" >> output.log
echo "BINARY=%binary_location%" >> output.log
echo "INSTANCES=${instances}" >> output.log
echo "MPIARGS=${mpiargs}" >> output.log
$(which mpirun) ${mpiargs} %binary_location% I=%input_file% MEMORY=${memory} MEMORY2=${memory2} NCPU=${cores} >> output.log 2>&1
# Copy data back to /fsx
if [ $SCRATCH != "/fsx" ]
then
echo "Writing output to /fsx/$TMPDIR"
cp -R $TMP_DIR /fsx
fiInstall IntelMPI on a custom AMI:
2019.7
#!/bin/bash
sudo su -
INTELMPI_VERSION="2019.7.217"
INTELMPI_URL="https://registrationcenter-download.intel.com/akdlm/irc_nas/tec/16546/l_mpi_2019.7.217.tgz"
INTELMPI_VERSION='2019.7.217'
INTELMPI_MODULEFILE="/opt/intel/impi/${INTELMPI_VERSION}/intel64/modulefiles/mpi"
INTELMPI_MODULEFILE_DIR="/opt/intel/impi/${INTELMPI_VERSION}/intel64/modulefiles"
wget $INTELMPI_URL
tar -xf l_mpi_$INTELMPI_VERSION.tgz
cd l_mpi_$INTELMPI_VERSION/
./install.sh -s silent.cfg --accept_eula
# create modulefile
mv $INTELMPI_MODULEFILE $INTELMPI_MODULEFILE_DIR/intelmpi
echo "$INTELMPI_MODULEFILE_DIR" >> /usr/share/Modules/init/.modulespath
exitInstall Open MPI on a custom AMI:
sudo yum update -y
curl -O https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz
tar -xf aws-efa-installer-latest.tar.gz
cd aws-efa-installer
sudo ./efa_installer.sh -y