Skip to content

Instantly share code, notes, and snippets.

@dmarce1

dmarce1/gist:ac22dbf4d651b1764ec2

Created September 17, 2015 19:07
Show Gist options
  • Save dmarce1/ac22dbf4d651b1764ec2 to your computer and use it in GitHub Desktop.
Save dmarce1/ac22dbf4d651b1764ec2 to your computer and use it in GitHub Desktop.
Download ZIP
Seg fault in parcelport
Raw
gistfile1.txt
[dmarce1@smic027 release]$ ./job
+ cd /work/dmarce1/octotiger/release
+ make -k all -j21
make: Nothing to be done for `all'.
+ mkdir 49309.smic3
mkdir: cannot create directory `49309.smic3': File exists
+ cp ./octotiger ./49309.smic3
+ cd 49309.smic3
+ uniq /var/spool/torque/aux//49309.smic3
+ export HPX_NODEFILE=actual.nodes.49309.smic3
+ HPX_NODEFILE=actual.nodes.49309.smic3
+ unset PBS_NODEFILE
+ mpirun -f actual.nodes.49309.smic3 ./octotiger -t20 -Ihpx.stacks.use_guard_pages=0
WARNING: Error in initializing MVAPICH2 ptmalloc library.Continuing without InfiniBand registration cache support.
Regrid gather
Regrid scatter
Regrid done
---------------Created Level 1---------------
Regrid gather
Regrid scatter
Regrid done
---------------Created Level 2---------------
Regrid gather
Regrid scatter
Regrid done
---------------Created Level 3---------------
Regrid gather
Regrid scatter
Regrid done
---------------Created Level 4---------------
Regrid gather
Regrid scatter
Regrid done
---------------Created Level 5---------------
Forming tree connections------------
...done
Waiting 3 seconds...
Starting...
--- begin checkpoint ---
--- end checkpoint ---
--- begin output ---
--- end output ---
{stack-trace}: 18 frames:
0x2ac246f5fc88 : hpx::termination_handler(int) + 0x158 in /home/dmarce1/lib/libhpx.so.0
0x361c40f710 : ??? + 0x361c40f710 in /lib64/libpthread.so.0
0x2ac24a54dd6b : handle_cqe + 0x83b in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2ac24a54acbe : MPIDI_CH3I_MRAILI_Cq_poll_ib + 0x28e in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2ac24a51dedb : MPIDI_CH3I_read_progress + 0xcb in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2ac24a51d869 : MPIDI_CH3I_Progress_test + 0x249 in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2ac24a6765e1 : MPIR_Test_impl + 0x71 in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2ac24a67634c : PMPI_Test + 0xfc in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2ac247c0bbf6 : ??? + 0x2ac247c0bbf6 in /home/dmarce1/lib/libhpx.so.0
0x2ac247c0a59c : hpx::parcelset::parcelport_impl<hpx::parcelset::policies::mpi::parcelport>::do_background_work(unsigned long) + 0x13c in /home/dmarce1/lib/libhpx.so.0
0x2ac24702457d : hpx::parcelset::parcelhandler::do_background_work(unsigned long, bool) + 0x8d in /home/dmarce1/lib/libhpx.so.0
0x2ac24715e73e : ??? + 0x2ac24715e73e in /home/dmarce1/lib/libhpx.so.0
0x2ac2470bd191 : ??? + 0x2ac2470bd191 in /home/dmarce1/lib/libhpx.so.0
0x2ac247143692 : hpx::threads::detail::thread_pool<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, hpx::threads::topology const&, boost::barrier&) + 0x702 in /home/dmarce1/lib/libhpx.so.0
0x2ac245d185a1 : ??? + 0x2ac245d185a1 in /home/dmarce1/lib/libboost_thread.so.1.59.0
0x361c4079d1 : ??? + 0x361c4079d1 in /lib64/libpthread.so.0
0x361bce88fd : clone + 0x6d in /lib64/libc.so.6
{what}: Segmentation fault
{config}:
HPX_HAVE_NATIVE_TLS=ON
HPX_HAVE_STACKTRACES=ON
HPX_HAVE_COMPRESSION_BZIP2=OFF
HPX_HAVE_COMPRESSION_SNAPPY=OFF
HPX_HAVE_COMPRESSION_ZLIB=OFF
HPX_HAVE_PARCEL_COALESCING=ON
HPX_HAVE_PARCELPORT_TCP=ON
HPX_HAVE_PARCELPORT_MPI=ON (MPICH V3.1, MPI V3.0)
HPX_HAVE_PARCELPORT_IPC=OFF
HPX_HAVE_PARCELPORT_IBVERBS=OFF
HPX_HAVE_VERIFY_LOCKS=ON
HPX_HAVE_HWLOC=ON
HPX_HAVE_ITTNOTIFY=OFF
HPX_HAVE_RUN_MAIN_EVERYWHERE=OFF
HPX_LIMIT=5
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
HPX_HAVE_MALLOC=tbbmalloc
HPX_PREFIX (configured)=/work/dmarce1/builds/hpx_build
HPX_PREFIX=/home/dmarce1
{version}: V0.9.11-trunk (AGAS: V3.0), Git: f75d2b0ff4
{boost}: V1.59.0
{build-type}: release
{date}: Sep 16 2015 17:08:10
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1500
{stdlib}: GNU libstdc++ version 20140422
[smic087:mpi_rank_25][error_sighandler] Caught error: Aborted (signal 6)
{stack-trace}: 18 frames:
0x2b3920353c88 : hpx::termination_handler(int) + 0x158 in /home/dmarce1/lib/libhpx.so.0
0x3730a0f710 : ??? + 0x3730a0f710 in /lib64/libpthread.so.0
0x2b3923941d6b : handle_cqe + 0x83b in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b392393ecbe : MPIDI_CH3I_MRAILI_Cq_poll_ib + 0x28e in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b3923911edb : MPIDI_CH3I_read_progress + 0xcb in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b3923911869 : MPIDI_CH3I_Progress_test + 0x249 in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b3923a6a5e1 : MPIR_Test_impl + 0x71 in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b3923a6a34c : PMPI_Test + 0xfc in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b3920fffbf6 : ??? + 0x2b3920fffbf6 in /home/dmarce1/lib/libhpx.so.0
0x2b3920ffe59c : hpx::parcelset::parcelport_impl<hpx::parcelset::policies::mpi::parcelport>::do_background_work(unsigned long) + 0x13c in /home/dmarce1/lib/libhpx.so.0
0x2b392041857d : hpx::parcelset::parcelhandler::do_background_work(unsigned long, bool) + 0x8d in /home/dmarce1/lib/libhpx.so.0
0x2b392055273e : ??? + 0x2b392055273e in /home/dmarce1/lib/libhpx.so.0
0x2b39204b1191 : ??? + 0x2b39204b1191 in /home/dmarce1/lib/libhpx.so.0
0x2b3920537692 : hpx::threads::detail::thread_pool<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, hpx::threads::topology const&, boost::barrier&) + 0x702 in /home/dmarce1/lib/libhpx.so.0
0x2b391f10c5a1 : ??? + 0x2b391f10c5a1 in /home/dmarce1/lib/libboost_thread.so.1.59.0
0x3730a079d1 : ??? + 0x3730a079d1 in /lib64/libpthread.so.0
0x37302e88fd : clone + 0x6d in /lib64/libc.so.6
{what}: Segmentation fault
{config}:
HPX_HAVE_NATIVE_TLS=ON
HPX_HAVE_STACKTRACES=ON
HPX_HAVE_COMPRESSION_BZIP2=OFF
HPX_HAVE_COMPRESSION_SNAPPY=OFF
HPX_HAVE_COMPRESSION_ZLIB=OFF
HPX_HAVE_PARCEL_COALESCING=ON
HPX_HAVE_PARCELPORT_TCP=ON
HPX_HAVE_PARCELPORT_MPI=ON (MPICH V3.1, MPI V3.0)
HPX_HAVE_PARCELPORT_IPC=OFF
HPX_HAVE_PARCELPORT_IBVERBS=OFF
HPX_HAVE_VERIFY_LOCKS=ON
HPX_HAVE_HWLOC=ON
HPX_HAVE_ITTNOTIFY=OFF
HPX_HAVE_RUN_MAIN_EVERYWHERE=OFF
HPX_LIMIT=5
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
HPX_HAVE_MALLOC=tbbmalloc
HPX_PREFIX (configured)=/work/dmarce1/builds/hpx_build
HPX_PREFIX=/home/dmarce1
{version}: V0.9.11-trunk (AGAS: V3.0), Git: f75d2b0ff4
{boost}: V1.59.0
{build-type}: release
{date}: Sep 16 2015 17:08:10
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1500
{stdlib}: GNU libstdc++ version 20140422
[smic056:mpi_rank_9][error_sighandler] Caught error: Aborted (signal 6)
{stack-trace}: 18 frames:
0x2b2132d45c88 : hpx::termination_handler(int) + 0x158 in /home/dmarce1/lib/libhpx.so.0
0x34fdc0f710 : ??? + 0x34fdc0f710 in /lib64/libpthread.so.0
0x2b2136333d6b : handle_cqe + 0x83b in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b2136330cbe : MPIDI_CH3I_MRAILI_Cq_poll_ib + 0x28e in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b2136303edb : MPIDI_CH3I_read_progress + 0xcb in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b2136303869 : MPIDI_CH3I_Progress_test + 0x249 in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b213645c5e1 : MPIR_Test_impl + 0x71 in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b213645c34c : PMPI_Test + 0xfc in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b21339f3c8b : ??? + 0x2b21339f3c8b in /home/dmarce1/lib/libhpx.so.0
0x2b21339f05b1 : hpx::parcelset::parcelport_impl<hpx::parcelset::policies::mpi::parcelport>::do_background_work(unsigned long) + 0x151 in /home/dmarce1/lib/libhpx.so.0
0x2b2132e0a57d : hpx::parcelset::parcelhandler::do_background_work(unsigned long, bool) + 0x8d in /home/dmarce1/lib/libhpx.so.0
0x2b2132f4473e : ??? + 0x2b2132f4473e in /home/dmarce1/lib/libhpx.so.0
0x2b2132ea3191 : ??? + 0x2b2132ea3191 in /home/dmarce1/lib/libhpx.so.0
0x2b2132f29692 : hpx::threads::detail::thread_pool<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, hpx::threads::topology const&, boost::barrier&) + 0x702 in /home/dmarce1/lib/libhpx.so.0
0x2b2131afe5a1 : ??? + 0x2b2131afe5a1 in /home/dmarce1/lib/libboost_thread.so.1.59.0
0x34fdc079d1 : ??? + 0x34fdc079d1 in /lib64/libpthread.so.0
0x34fd4e88fd : clone + 0x6d in /lib64/libc.so.6
{what}: Segmentation fault
{config}:
HPX_HAVE_NATIVE_TLS=ON
HPX_HAVE_STACKTRACES=ON
HPX_HAVE_COMPRESSION_BZIP2=OFF
HPX_HAVE_COMPRESSION_SNAPPY=OFF
HPX_HAVE_COMPRESSION_ZLIB=OFF
HPX_HAVE_PARCEL_COALESCING=ON
HPX_HAVE_PARCELPORT_TCP=ON
HPX_HAVE_PARCELPORT_MPI=ON (MPICH V3.1, MPI V3.0)
HPX_HAVE_PARCELPORT_IPC=OFF
HPX_HAVE_PARCELPORT_IBVERBS=OFF
HPX_HAVE_VERIFY_LOCKS=ON
HPX_HAVE_HWLOC=ON
HPX_HAVE_ITTNOTIFY=OFF
HPX_HAVE_RUN_MAIN_EVERYWHERE=OFF
HPX_LIMIT=5
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
HPX_HAVE_MALLOC=tbbmalloc
HPX_PREFIX (configured)=/work/dmarce1/builds/hpx_build
HPX_PREFIX=/home/dmarce1
{version}: V0.9.11-trunk (AGAS: V3.0), Git: f75d2b0ff4
{boost}: V1.59.0
{build-type}: release
{date}: Sep 16 2015 17:08:10
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1500
{stdlib}: GNU libstdc++ version 20140422
[smic118:mpi_rank_50][error_sighandler] Caught error: Aborted (signal 6)
{stack-trace}: 18 frames:
0x2b2facf00c88 : hpx::termination_handler(int) + 0x158 in /home/dmarce1/lib/libhpx.so.0
0x3382c0f710 : ??? + 0x3382c0f710 in /lib64/libpthread.so.0
0x2b2fb04eed6b : handle_cqe + 0x83b in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b2fb04ebcbe : MPIDI_CH3I_MRAILI_Cq_poll_ib + 0x28e in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b2fb04beedb : MPIDI_CH3I_read_progress + 0xcb in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b2fb04be869 : MPIDI_CH3I_Progress_test + 0x249 in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b2fb06175e1 : MPIR_Test_impl + 0x71 in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b2fb061734c : PMPI_Test + 0xfc in /usr/local/packages/mvapich2/2.0/INTEL-14.0.2/lib/libmpich.so.12
0x2b2fadbacbf6 : ??? + 0x2b2fadbacbf6 in /home/dmarce1/lib/libhpx.so.0
0x2b2fadbab59c : hpx::parcelset::parcelport_impl<hpx::parcelset::policies::mpi::parcelport>::do_background_work(unsigned long) + 0x13c in /home/dmarce1/lib/libhpx.so.0
0x2b2facfc557d : hpx::parcelset::parcelhandler::do_background_work(unsigned long, bool) + 0x8d in /home/dmarce1/lib/libhpx.so.0
0x2b2fad0ff73e : ??? + 0x2b2fad0ff73e in /home/dmarce1/lib/libhpx.so.0
0x2b2fad05e191 : ??? + 0x2b2fad05e191 in /home/dmarce1/lib/libhpx.so.0
0x2b2fad0e4692 : hpx::threads::detail::thread_pool<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, hpx::threads::topology const&, boost::barrier&) + 0x702 in /home/dmarce1/lib/libhpx.so.0
0x2b2fabcb95a1 : ??? + 0x2b2fabcb95a1 in /home/dmarce1/lib/libboost_thread.so.1.59.0
0x3382c079d1 : ??? + 0x3382c079d1 in /lib64/libpthread.so.0
0x33824e88fd : clone + 0x6d in /lib64/libc.so.6
{what}: Segmentation fault
{config}:
HPX_HAVE_NATIVE_TLS=ON
HPX_HAVE_STACKTRACES=ON
HPX_HAVE_COMPRESSION_BZIP2=OFF
HPX_HAVE_COMPRESSION_SNAPPY=OFF
HPX_HAVE_COMPRESSION_ZLIB=OFF
HPX_HAVE_PARCEL_COALESCING=ON
HPX_HAVE_PARCELPORT_TCP=ON
HPX_HAVE_PARCELPORT_MPI=ON (MPICH V3.1, MPI V3.0)
HPX_HAVE_PARCELPORT_IPC=OFF
HPX_HAVE_PARCELPORT_IBVERBS=OFF
HPX_HAVE_VERIFY_LOCKS=ON
HPX_HAVE_HWLOC=ON
HPX_HAVE_ITTNOTIFY=OFF
HPX_HAVE_RUN_MAIN_EVERYWHERE=OFF
HPX_LIMIT=5
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
HPX_HAVE_MALLOC=tbbmalloc
HPX_PREFIX (configured)=/work/dmarce1/builds/hpx_build
HPX_PREFIX=/home/dmarce1
{version}: V0.9.11-trunk (AGAS: V3.0), Git: f75d2b0ff4
{boost}: V1.59.0
{build-type}: release
{date}: Sep 16 2015 17:08:10
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1500
{stdlib}: GNU libstdc++ version 20140422
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 121971 RUNNING AT smic087
= EXIT CODE: 6
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0:0@smic027] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:0@smic027] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:0@smic027] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:1@smic044] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:1@smic044] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:1@smic044] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:2@smic047] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:2@smic047] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:2@smic047] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:6@smic052] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:6@smic052] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:6@smic052] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:10@smic057] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:10@smic057] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:10@smic057] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:12@smic061] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:12@smic061] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:12@smic061] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:27@smic089] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:27@smic089] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:27@smic089] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:35@smic100] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:35@smic100] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:35@smic100] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:49@smic117] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:49@smic117] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:49@smic117] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:50@smic118] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:50@smic118] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:50@smic118] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:59@smic135] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:59@smic135] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:59@smic135] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:69@smic147] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:69@smic147] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:69@smic147] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:71@smic150] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:71@smic150] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:71@smic150] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:72@smic152] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:72@smic152] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:72@smic152] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[mpiexec@smic027] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting
[mpiexec@smic027] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
[mpiexec@smic027] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion
[mpiexec@smic027] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion
[dmarce1@smic027 release]$
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment