gistfile1.txt

2020-11-23 00:10:06,935 ERROR      mympirun.RunAsyncMPI MainThread  _post_exitcode: problem occured with cmd ['mpirun', '--file=/user/gent/438/vsc43806/.mympirun_m4jtq7/33429921_20201123_001005/mpdboot', '--machinefile', '/user/gent/438/vsc43806/.mympirun_m4jtq7/33429921_20201123_001005/nodes', '-rmk', 'slurm', '-bootstrap', 'slurm', '-genv', 'MKL_NUM_THREADS', '1', '-genv', 'MODULEPATH', '/apps/gent/CO7/haswell-ib/modules/all:/etc/modulefiles/vsc', '-genv', 'LOADEDMODULES', 'cluster/swalot:GCCcore/9.3.0:zlib/1.2.11-GCCcore-9.3.0:binutils/2.34-GCCcore-9.3.0:iccifort/2020.1.217:numactl/2.0.13-GCCcore-9.3.0:UCX/1.8.0-GCCcore-9.3.0:impi/2019.7.217-iccifort-2020.1.217:iimpi/2020a:imkl/2020.1.217-iimpi-2020a:intel/2020a:FDS/6.7.5-intel-2020a:vsc-mympirun/5.2.5', '-genv', 'MODULESHOME', '/usr/share/lmod/lmod', '-genv', 'I_MPI_FALLBACK_DEVICE', '0', '-genv', 'I_MPI_DAT_LIBRARY', 'libdat2.so', '-genv', 'I_MPI_NETMASK', '10.143.0.0/255.255.0.0', '-genv', 'I_MPI_PIN', '1', '-genv', 'I_MPI_FALLBACK', 'disable', '-genv', 'I_MPI_FABRICS', 'shm:dapl', '-genv', 'I_MPI_DAPL_SCALABLE_PROGRESS', '0', '-np', '9', '-envlist', 'LD_LIBRARY_PATH,PATH,PYTHONPATH,I_MPI_HYDRA_TOPOLIB,I_MPI_ROOT,I_MPI_TMPDIR,FI_PROVIDER_PATH,MKL_EXAMPLES,OMP_NUM_THREADS', 'fds', 'PretrelC2.fds']: (shellcmd ['mpirun', '--file=/user/gent/438/vsc43806/.mympirun_m4jtq7/33429921_20201123_001005/mpdboot', '--machinefile', '/user/gent/438/vsc43806/.mympirun_m4jtq7/33429921_20201123_001005/nodes', '-rmk', 'slurm', '-bootstrap', 'slurm', '-genv', 'MKL_NUM_THREADS', '1', '-genv', 'MODULEPATH', '/apps/gent/CO7/haswell-ib/modules/all:/etc/modulefiles/vsc', '-genv', 'LOADEDMODULES', 'cluster/swalot:GCCcore/9.3.0:zlib/1.2.11-GCCcore-9.3.0:binutils/2.34-GCCcore-9.3.0:iccifort/2020.1.217:numactl/2.0.13-GCCcore-9.3.0:UCX/1.8.0-GCCcore-9.3.0:impi/2019.7.217-iccifort-2020.1.217:iimpi/2020a:imkl/2020.1.217-iimpi-2020a:intel/2020a:FDS/6.7.5-intel-2020a:vsc-mympirun/5.2.5', '-genv', 'MODULESHOME', '/usr/share/lmod/lmod', '-genv', 'I_MPI_FALLBACK_DEVICE', '0', '-genv', 'I_MPI_DAT_LIBRARY', 'libdat2.so', '-genv', 'I_MPI_NETMASK', '10.143.0.0/255.255.0.0', '-genv', 'I_MPI_PIN', '1', '-genv', 'I_MPI_FALLBACK', 'disable', '-genv', 'I_MPI_FABRICS', 'shm:dapl', '-genv', 'I_MPI_DAPL_SCALABLE_PROGRESS', '0', '-np', '9', '-envlist', 'LD_LIBRARY_PATH,PATH,PYTHONPATH,I_MPI_HYDRA_TOPOLIB,I_MPI_ROOT,I_MPI_TMPDIR,FI_PROVIDER_PATH,MKL_EXAMPLES,OMP_NUM_THREADS', 'fds', 'PretrelC2.fds']) output Assertion failed in file ../../src/util/intel/shm_heap/impi_shm_heap.c at line 926: group_id < group_num
Assertion failed in file ../../src/util/intel/shm_heap/impi_shm_heap.c at line 926: group_id < group_num
Assertion failed in file ../../src/util/intel/shm_heap/impi_shm_heap.c at line 926: group_id < group_num
Assertion failed in file ../../src/util/intel/shm_heap/impi_shm_heap.c at line 926: group_id < group_num
Assertion failed in file ../../src/util/intel/shm_heap/impi_shm_heap.c at line 926: group_id < group_num
Assertion failed in file ../../src/util/intel/shm_heap/impi_shm_heap.c at line 926: group_id < group_num
Assertion failed in file ../../src/util/intel/shm_heap/impi_shm_heap.c at line 926: group_id < group_num
Assertion failed in file ../../src/util/intel/shm_heap/impi_shm_heap.c at line 926: group_id < group_num
Assertion failed in file ../../src/util/intel/shm_heap/impi_shm_heap.c at line 926: group_id < group_num
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x1c) [0x2af68d949f7c]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x1c) [0x2ba97063af7c]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x2ba97006cfc1]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x3cb7cf) [0x2ba96ffb37cf]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x821826) [0x2ba970409826]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x62770b) [0x2ba97020f70b]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x70fb46) [0x2ba9702f7b46]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x188163) [0x2ba96fd70163]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(PMPI_Init_thread+0xe5) [0x2ba96ffe9395]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/libmpifort.so.12(MPI_INIT_THREAD+0x2c) [0x2ba96f91ff4c]
fds() [0xc898e0]
fds() [0x404c92]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x2ba970f95555]
fds() [0x404ba9]
Abort(1) on node 2: Internal error
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x1c) [0x2b2bbd3a0f7c]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x1c) [0x2af407a23f7c]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x1c) [0x2b1272e8cf7c]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x2b12728befc1]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x3cb7cf) [0x2b12728057cf]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x821826) [0x2b1272c5b826]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x62770b) [0x2b1272a6170b]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x70fb46) [0x2b1272b49b46]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x188163) [0x2b12725c2163]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(PMPI_Init_thread+0xe5) [0x2b127283b395]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/libmpifort.so.12(MPI_INIT_THREAD+0x2c) [0x2b1272171f4c]
fds() [0xc898e0]
fds() [0x404c92]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x2b12737e7555]
fds() [0x404ba9]
Abort(1) on node 4: Internal error
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x1c) [0x2b3d5a64cf7c]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x2b3d5a07efc1]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x3cb7cf) [0x2b3d59fc57cf]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x821826) [0x2b3d5a41b826]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x62770b) [0x2b3d5a22170b]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x70fb46) [0x2b3d5a309b46]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x188163) [0x2b3d59d82163]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(PMPI_Init_thread+0xe5) [0x2b3d59ffb395]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/libmpifort.so.12(MPI_INIT_THREAD+0x2c) [0x2b3d59931f4c]
fds() [0xc898e0]
fds() [0x404c92]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x2b3d5afa7555]
fds() [0x404ba9]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x1c) [0x2ae8ffac5f7c]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x2ae8ff4f7fc1]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x3cb7cf) [0x2ae8ff43e7cf]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x821826) [0x2ae8ff894826]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x62770b) [0x2ae8ff69a70b]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x70fb46) [0x2ae8ff782b46]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x188163) [0x2ae8ff1fb163]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(PMPI_Init_thread+0xe5) [0x2ae8ff474395]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/libmpifort.so.12(MPI_INIT_THREAD+0x2c) [0x2ae8fedaaf4c]
fds() [0xc898e0]
fds() [0x404c92]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x2ae900420555]
fds() [0x404ba9]
Abort(1) on node 5: Internal error
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x2b2bbcdd2fc1]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x1c) [0x2b155ef89f7c]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x2b155e9bbfc1]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x3cb7cf) [0x2b155e9027cf]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x821826) [0x2b155ed58826]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x62770b) [0x2b155eb5e70b]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x70fb46) [0x2b155ec46b46]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x188163) [0x2b155e6bf163]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(PMPI_Init_thread+0xe5) [0x2b155e938395]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/libmpifort.so.12(MPI_INIT_THREAD+0x2c) [0x2b155e26ef4c]
Abort(1) on node 8: Internal error
fds() [0xc898e0]
fds() [0x404c92]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x2b155f8e4555]
fds() [0x404ba9]
Abort(1) on node 7: Internal error
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x2af407455fc1]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x3cb7cf) [0x2b2bbcd197cf]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x3cb7cf) [0x2af40739c7cf]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x821826) [0x2b2bbd16f826]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x62770b) [0x2b2bbcf7570b]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x821826) [0x2af4077f2826]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x62770b) [0x2af4075f870b]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x70fb46) [0x2b2bbd05db46]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x70fb46) [0x2af4076e0b46]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x188163) [0x2b2bbcad6163]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x188163) [0x2af407159163]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(PMPI_Init_thread+0xe5) [0x2b2bbcd4f395]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(PMPI_Init_thread+0xe5) [0x2af4073d2395]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x1c) [0x2ad23c538f7c]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x2ad23bf6afc1]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x3cb7cf) [0x2ad23beb17cf]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x821826) [0x2ad23c307826]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x62770b) [0x2ad23c10d70b]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x70fb46) [0x2ad23c1f5b46]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(+0x188163) [0x2ad23bc6e163]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/release/libmpi.so.12(PMPI_Init_thread+0xe5) [0x2ad23bee7395]
/apps/gent/CO7/haswell-ib/software/impi/2019.7.217-iccifort-2020.1.217/intel64/lib/libmpifort.so.12(MPI_INIT_THREAD+0x2c) [0x2ad23b81df4c]
fds() [0xc898e0]
fds() [0x404c92]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x2ad23ce93555]
fds() [0x404ba9]
Abort(1) on node 1: Internal error

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 6 PID 5089 RUNNING AT node2707.swalot.os
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 0 PID 15921 RUNNING AT node2649.swalot.os

=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 3 PID 8619 RUNNING AT node2694.swalot.os
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

2020-11-23 00:10:06,936 WARNING    mympirun.Coupler_IntelMPI2019_SLURM MainThread  main: exitcode 255 > 0; cmd ['mpirun', '--file=/user/gent/438/vsc43806/.mympirun_m4jtq7/33429921_20201123_001005/mpdboot', '--machinefile', '/user/gent/438/vsc43806/.mympirun_m4jtq7/33429921_20201123_001005/nodes', '-rmk', 'slurm', '-bootstrap', 'slurm', '-genv', 'MKL_NUM_THREADS', '1', '-genv', 'MODULEPATH', '/apps/gent/CO7/haswell-ib/modules/all:/etc/modulefiles/vsc', '-genv', 'LOADEDMODULES', 'cluster/swalot:GCCcore/9.3.0:zlib/1.2.11-GCCcore-9.3.0:binutils/2.34-GCCcore-9.3.0:iccifort/2020.1.217:numactl/2.0.13-GCCcore-9.3.0:UCX/1.8.0-GCCcore-9.3.0:impi/2019.7.217-iccifort-2020.1.217:iimpi/2020a:imkl/2020.1.217-iimpi-2020a:intel/2020a:FDS/6.7.5-intel-2020a:vsc-mympirun/5.2.5', '-genv', 'MODULESHOME', '/usr/share/lmod/lmod', '-genv', 'I_MPI_FALLBACK_DEVICE', '0', '-genv', 'I_MPI_DAT_LIBRARY', 'libdat2.so', '-genv', 'I_MPI_NETMASK', '10.143.0.0/255.255.0.0', '-genv', 'I_MPI_PIN', '1', '-genv', 'I_MPI_FALLBACK', 'disable', '-genv', 'I_MPI_FABRICS', 'shm:dapl', '-genv', 'I_MPI_DAPL_SCALABLE_PROGRESS', '0', '-np', '9', '-envlist', 'LD_LIBRARY_PATH,PATH,PYTHONPATH,I_MPI_HYDRA_TOPOLIB,I_MPI_ROOT,I_MPI_TMPDIR,FI_PROVIDER_PATH,MKL_EXAMPLES,OMP_NUM_THREADS', 'fds', 'PretrelC2.fds']
2020-11-23 00:10:06,938 ERROR      mympirun        MainThread  Main failed: main: exitcode 255 > 0; cmd ['mpirun', '--file=/user/gent/438/vsc43806/.mympirun_m4jtq7/33429921_20201123_001005/mpdboot', '--machinefile', '/user/gent/438/vsc43806/.mympirun_m4jtq7/33429921_20201123_001005/nodes', '-rmk', 'slurm', '-bootstrap', 'slurm', '-genv', 'MKL_NUM_THREADS', '1', '-genv', 'MODULEPATH', '/apps/gent/CO7/haswell-ib/modules/all:/etc/modulefiles/vsc', '-genv', 'LOADEDMODULES', 'cluster/swalot:GCCcore/9.3.0:zlib/1.2.11-GCCcore-9.3.0:binutils/2.34-GCCcore-9.3.0:iccifort/2020.1.217:numactl/2.0.13-GCCcore-9.3.0:UCX/1.8.0-GCCcore-9.3.0:impi/2019.7.217-iccifort-2020.1.217:iimpi/2020a:imkl/2020.1.217-iimpi-2020a:intel/2020a:FDS/6.7.5-intel-2020a:vsc-mympirun/5.2.5', '-genv', 'MODULESHOME', '/usr/share/lmod/lmod', '-genv', 'I_MPI_FALLBACK_DEVICE', '0', '-genv', 'I_MPI_DAT_LIBRARY', 'libdat2.so', '-genv', 'I_MPI_NETMASK', '10.143.0.0/255.255.0.0', '-genv', 'I_MPI_PIN', '1', '-genv', 'I_MPI_FALLBACK', 'disable', '-genv', 'I_MPI_FABRICS', 'shm:dapl', '-genv', 'I_MPI_DAPL_SCALABLE_PROGRESS', '0', '-np', '9', '-envlist', 'LD_LIBRARY_PATH,PATH,PYTHONPATH,I_MPI_HYDRA_TOPOLIB,I_MPI_ROOT,I_MPI_TMPDIR,FI_PROVIDER_PATH,MKL_EXAMPLES,OMP_NUM_THREADS', 'fds', 'PretrelC2.fds']
Traceback (most recent call last):
  File "/kyukon/home/apps/CO7/haswell-ib/software/vsc-mympirun/5.2.5/lib/python3.6/site-packages/vsc_mympirun-5.2.5-py3.6.egg/vsc/mympirun/main.py", line 120, in main
    instance.main()
  File "/kyukon/home/apps/CO7/haswell-ib/software/vsc-mympirun/5.2.5/lib/python3.6/site-packages/vsc_mympirun-5.2.5-py3.6.egg/vsc/mympirun/mpi/mpi.py", line 291, in main
    self.log.raiseException("main: exitcode %s > 0; cmd %s" % (exitcode, self.mpirun_cmd))
  File "/kyukon/home/apps/CO7/haswell-ib/software/vsc-mympirun/5.2.5/lib/python3.6/site-packages/vsc_base-3.1.4-py3.6.egg/vsc/utils/fancylogger.py", line 346, in raiseException
    raise_with_traceback(exception(message))
  File "/apps/gent/CO7/haswell-ib/software/vsc-mympirun/5.2.5/lib/python3.6/site-packages/future-0.18.2-py3.6.egg/future/utils/__init__.py", line 446, in raise_with_traceback
    raise exc.with_traceback(traceback)
Exception: main: exitcode 255 > 0; cmd ['mpirun', '--file=/user/gent/438/vsc43806/.mympirun_m4jtq7/33429921_20201123_001005/mpdboot', '--machinefile', '/user/gent/438/vsc43806/.mympirun_m4jtq7/33429921_20201123_001005/nodes', '-rmk', 'slurm', '-bootstrap', 'slurm', '-genv', 'MKL_NUM_THREADS', '1', '-genv', 'MODULEPATH', '/apps/gent/CO7/haswell-ib/modules/all:/etc/modulefiles/vsc', '-genv', 'LOADEDMODULES', 'cluster/swalot:GCCcore/9.3.0:zlib/1.2.11-GCCcore-9.3.0:binutils/2.34-GCCcore-9.3.0:iccifort/2020.1.217:numactl/2.0.13-GCCcore-9.3.0:UCX/1.8.0-GCCcore-9.3.0:impi/2019.7.217-iccifort-2020.1.217:iimpi/2020a:imkl/2020.1.217-iimpi-2020a:intel/2020a:FDS/6.7.5-intel-2020a:vsc-mympirun/5.2.5', '-genv', 'MODULESHOME', '/usr/share/lmod/lmod', '-genv', 'I_MPI_FALLBACK_DEVICE', '0', '-genv', 'I_MPI_DAT_LIBRARY', 'libdat2.so', '-genv', 'I_MPI_NETMASK', '10.143.0.0/255.255.0.0', '-genv', 'I_MPI_PIN', '1', '-genv', 'I_MPI_FALLBACK', 'disable', '-genv', 'I_MPI_FABRICS', 'shm:dapl', '-genv', 'I_MPI_DAPL_SCALABLE_PROGRESS', '0', '-np', '9', '-envlist', 'LD_LIBRARY_PATH,PATH,PYTHONPATH,I_MPI_HYDRA_TOPOLIB,I_MPI_ROOT,I_MPI_TMPDIR,FI_PROVIDER_PATH,MKL_EXAMPLES,OMP_NUM_THREADS', 'fds', 'PretrelC2.fds']