jsrun_layout.md

Testing jsrun in layout mode on Summitdev

1) Task needs to be assigned node 1, physical core 1, HW thread 1

Status: Done

$ cat resource-file
RS 0: { host: 1, cpu: 0}
$ export OMP_NUM_THREADS=1
$ jsrun -n 1 -a 1 -U /ccs/home/vivekb/resource-file-1 /ccs/proj/csc190/Hello_jsrun/jsrun_layout
*** MPI Ranks: 1, OpenMP Threads: 1, GPUs per Resource Set: 0 ***
========================================================================
MPI Rank 000 of 001 on HWThread 000 of Node summitdev-r0c1n08, OMP_threadID 0 of 1

2) Task needs to be assigned node 1, physical core 1, HW threads 1 and 2 (

Status: Needs additional testing

Need to use OMP_PLACES and '-b' flags

3) Task needs to be assigned node 1, physical cores 1 and 2, HW thread 1 of each physical core

Status: Done

$ cat resource-file
RS 0: { host: 1, cpu: 0 1}

2 MPI tasks, 1 OMP thread per task

$ export OMP_NUM_THREADS=1
$ jsrun -n 1 -a 2 -U /ccs/home/vivekb/resource-file-1 /ccs/proj/csc190/Hello_jsrun/jsrun_layout
*** MPI Ranks: 2, OpenMP Threads: 1, GPUs per Resource Set: 0 ***
========================================================================
MPI Rank 000 of 002 on HWThread 000 of Node summitdev-r0c1n08, OMP_threadID 0 of 1
MPI Rank 001 of 002 on HWThread 008 of Node summitdev-r0c1n08, OMP_threadID 0 of 1

Note: When we specify cpu 1, we actually seem to be getting HWthread 008. Summitdev has 8 HWthreads in a physical core (Summit has 4 in 1). HWthread 008 is the first HW thread in the second core.

1 MPI task, 2 OMP threads per task

$ export OMP_NUM_THREADS=2
$ jsrun -n 1 -a 1 -U /ccs/home/vivekb/resource-file-1 /ccs/proj/csc190/Hello_jsrun/jsrun_layout
*** MPI Ranks: 1, OpenMP Threads: 2, GPUs per Resource Set: 0 ***
========================================================================
MPI Rank 000 of 001 on HWThread 000 of Node summitdev-r0c1n08, OMP_threadID 0 of 2
MPI Rank 000 of 001 on HWThread 000 of Node summitdev-r0c1n08, OMP_threadID 1 of 2

Note: Both threads use the same HWthread. This seems to be an issue only on Summitdev, not Summit (as checked by Brian from ORNL). Needs to be confirmed by us as well.

4) Task needs to be assigned node 1, physical cores 1 and 2, HW threads 1 and 2 of each physical core

Status: Needs additional testing

Need to use OMP_PLACES and '-b' flags

5) Task needs to be assigned nodes 1 and 2, physical core 1 of each node, HW thread 1 of each physical core

Status: Done

$ cat resource-file
RS 0: { host: 1, cpu: 0}
RS 1: { host: 2, cpu: 0}

2 MPI tasks, 1 OMP thread per task

$ export OMP_NUM_THREADS=1
jsrun -n 2 -a 1 -U /ccs/home/vivekb/resource-file-1 /ccs/proj/csc190/Hello_jsrun/jsrun_layout
MPI Rank 001 of 002 on HWThread 000 of Node summitdev-r0c1n09, OMP_threadID 0 of 1
*** MPI Ranks: 2, OpenMP Threads: 1, GPUs per Resource Set: 0 ***
========================================================================
MPI Rank 000 of 002 on HWThread 000 of Node summitdev-r0c1n08, OMP_threadID 0 of 1

6) Task needs to be assigned nodes 1 and 2, physical cores 1 and 2 of each node, HW threads 1 and 2 of each physical core

Status: Done

$ cat resource-file
RS 0: { host: 1, cpu: 0 1}
RS 1: { host: 2, cpu: 0 1}

2 MPI tasks, 2 OMP threads per task

$ export OMP_NUM_THREADS=2
$ jsrun -n 2 -a 1 -U /ccs/home/vivekb/resource-file-1 /ccs/proj/csc190/Hello_jsrun/jsrun_layout
MPI Rank 001 of 002 on HWThread 000 of Node summitdev-r0c1n04, OMP_threadID 0 of 2
MPI Rank 001 of 002 on HWThread 000 of Node summitdev-r0c1n04, OMP_threadID 1 of 2
*** MPI Ranks: 2, OpenMP Threads: 2, GPUs per Resource Set: 0 ***
========================================================================
MPI Rank 000 of 002 on HWThread 000 of Node summitdev-r0c1n02, OMP_threadID 0 of 2
MPI Rank 000 of 002 on HWThread 000 of Node summitdev-r0c1n02, OMP_threadID 1 of 2

Note: Both threads use the same HWthread. This seems to be an issue only on Summitdev, not Summit (as checked by Brian from ORNL). Needs to be confirmed by us as well.