Skip to content

Instantly share code, notes, and snippets.

@vivek-bala

vivek-bala/jsrun_runs_summitdev.md

Created October 30, 2018 20:52
Show Gist options
  • Save vivek-bala/76c4c1767a9a37c07b51ed3bcb893e7e to your computer and use it in GitHub Desktop.
Save vivek-bala/76c4c1767a9a37c07b51ed3bcb893e7e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
jsrun_runs_summitdev.md

OpenMP (env variables)=4, smt level=4:

jsrun -n1 -a1 -c4 -g1 /ccs/home/vivekb/tools/Hello_jsrun/jsrun_layout
########################################################################

*** MPI Ranks: 1, OpenMP Threads: 4, GPUs per Resource Set: 1 ***
========================================================================
MPI Rank 000, OMP_thread 00 on HWThread 002 of Node summitdev-r0c2n02 - RT_GPU_id 0 : GPU_id 0 
MPI Rank 000, OMP_thread 03 on HWThread 000 of Node summitdev-r0c2n02 - RT_GPU_id 0 : GPU_id 0 
MPI Rank 000, OMP_thread 01 on HWThread 003 of Node summitdev-r0c2n02 - RT_GPU_id 0 : GPU_id 0 
MPI Rank 000, OMP_thread 02 on HWThread 001 of Node summitdev-r0c2n02 - RT_GPU_id 0 : GPU_id 0 

jsrun -n1 -a1 /ccs/home/vivekb/tools/Hello_jsrun/jsrun_layout
########################################################################

*** MPI Ranks: 1, OpenMP Threads: 4, GPUs per Resource Set: 4 ***
========================================================================
MPI Rank 000, OMP_thread 00 on HWThread 003 of Node summitdev-r0c1n02 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 02 on HWThread 002 of Node summitdev-r0c1n02 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 03 on HWThread 001 of Node summitdev-r0c1n02 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 01 on HWThread 000 of Node summitdev-r0c1n02 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 

 jsrun -n1 -a2 /ccs/home/vivekb/tools/Hello_jsrun/jsrun_layout
CUDA Error - cudaDeviceGetPCIBusId: all CUDA-capable devices are busy or unavailable/n########################################################################

*** MPI Ranks: 2, OpenMP Threads: 4, GPUs per Resource Set: 4 ***
========================================================================
MPI Rank 000, OMP_thread 00 on HWThread 001 of Node summitdev-r0c1n02 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 02 on HWThread 003 of Node summitdev-r0c1n02 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 01 on HWThread 002 of Node summitdev-r0c1n02 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 03 on HWThread 000 of Node summitdev-r0c1n02 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3

OpenMP=4, smt level=1:

jsrun -n1 -a1 -c4 -g1 /ccs/home/vivekb/tools/Hello_jsrun/jsrun_layout
########################################################################

*** MPI Ranks: 1, OpenMP Threads: 4, GPUs per Resource Set: 1 ***
========================================================================
MPI Rank 000, OMP_thread 03 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 : GPU_id 0 
MPI Rank 000, OMP_thread 02 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 : GPU_id 0 
MPI Rank 000, OMP_thread 01 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 : GPU_id 0 
MPI Rank 000, OMP_thread 00 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 : GPU_id 0

jsrun -n1 -a1 /ccs/home/vivekb/tools/Hello_jsrun/jsrun_layout
########################################################################

*** MPI Ranks: 1, OpenMP Threads: 4, GPUs per Resource Set: 4 ***
========================================================================
MPI Rank 000, OMP_thread 03 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 02 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 01 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 00 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 

jsrun -n1 -a2 /ccs/home/vivekb/tools/Hello_jsrun/jsrun_layout
CUDA Error - cudaDeviceGetPCIBusId: all CUDA-capable devices are busy or unavailable/n########################################################################

*** MPI Ranks: 2, OpenMP Threads: 4, GPUs per Resource Set: 4 ***
========================================================================
MPI Rank 000, OMP_thread 03 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 02 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 01 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 
MPI Rank 000, OMP_thread 00 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3

OpenMP=None, smt level=1:

$ jsrun -n1 -a1 /ccs/home/vivekb/tools/Hello_jsrun/jsrun_layout
########################################################################

*** MPI Ranks: 1, OpenMP Threads: 1, GPUs per Resource Set: 4 ***
========================================================================
MPI Rank 000, OMP_thread 00 on HWThread 008 of Node summitdev-r0c2n05 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3 

$ jsrun -n1 -a1 -c4 /ccs/home/vivekb/tools/Hello_jsrun/jsrun_layout
########################################################################

*** MPI Ranks: 1, OpenMP Threads: 1, GPUs per Resource Set: 4 ***
========================================================================
MPI Rank 000, OMP_thread 00 on HWThread 000 of Node summitdev-r0c2n05 - RT_GPU_id 0 1 2 3 : GPU_id 0 1 2 3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment