Skip to content

Instantly share code, notes, and snippets.

@dmarce1
Created December 28, 2015 17:00
Show Gist options
  • Save dmarce1/14f30c0d2ed8fc0f2c2f to your computer and use it in GitHub Desktop.
Save dmarce1/14f30c0d2ed8fc0f2c2f to your computer and use it in GitHub Desktop.
+ cat /var/spool/torque/aux//77455.smic3
+ awk 'NR % 20 == 0'
+ export HPX_NODEFILE=node.list
+ HPX_NODEFILE=node.list
+ export PBS_NODEFILE=node.list
+ PBS_NODEFILE=node.list
++ wc -l node.list
++ gawk '//{print $1}'
+ export NPROCS=64
+ NPROCS=64
+ rm data.bin
+ mpirun -np 64 --machinefile node.list ./octotiger -t20 restart.chk '--hpx:print-counter=/statistics{/threads{locality#1/total}/count/instantaneous/all}/average' --hpx:print-counter-interval 1000
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,1,0.123332,[s],-9.22337e+18
Running
Loading from restart.chk...
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,2,1.124625,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,3,2.124338,[s],-9.22337e+18
Loaded checkpoint file
-----------------------------------------------
regridding
rebalancing 6793 nodes
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,4,3.124988,[s],-9.22337e+18
forming tree connections
solving gravity
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,5,4.126374,[s],-9.22337e+18
regrid done
-----------------------------------------------
Done.
Forming tree connections------------
...done
Starting...
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,6,5.126101,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,7,6.126270,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,8,7.127267,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,9,8.127862,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,10,9.128303,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,11,10.128714,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,12,11.129590,[s],-9.22337e+18
Saved 6793 grids to checkpoint file
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,13,12.131332,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,14,13.131009,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,15,14.132628,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,16,15.132512,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,17,16.134185,[s],-9.22337e+18
0 1.756506e+01 1.316241e-03 1.144979e+01 1.327136e-01 1.156828e+00
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,18,17.134883,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,19,18.134410,[s],-9.22337e+18
1 1.756767e+01 1.291751e-03 2.100582e+00 1.327140e-01 1.156999e+00
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,20,19.139610,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,21,20.140624,[s],-9.22337e+18
2 1.757023e+01 1.272653e-03 2.033617e+00 1.327144e-01 1.157168e+00
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,22,21.142092,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,23,22.142209,[s],-9.22337e+18
3 1.757276e+01 1.259911e-03 2.072346e+00 1.327148e-01 1.157336e+00
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,24,23.142602,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,25,24.143749,[s],-9.22337e+18
4 1.757527e+01 1.248242e-03 2.076644e+00 1.327153e-01 1.157501e+00
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,26,25.144936,[s],-9.22337e+18
/statistics{/threads{locality#1/total}/count/instantaneous/all}/average,27,26.144706,[s],-9.22337e+18
5 1.757776e+01 1.237933e-03 2.046060e+00 1.327157e-01 1.157666e+00
{stack-trace}: 7 frames:
0x2ab25f54f1e8 : hpx::termination_handler(int) + 0x158 in /home/dmarce1/debug/lib/libhpxd.so.0
0x3b83e0f710 : ??? + 0x3b83e0f710 in /lib64/libpthread.so.0
0x2ab25ff3570a : hpx::performance_counters::server::statistics_counter<boost::accumulators::tag::mean>::get_counter_value(bool) + 0x17a in /home/dmarce1/debug/lib/libhpxd.so.0
0x2ab26001fa63 : ??? + 0x2ab26001fa63 in /home/dmarce1/debug/lib/libhpxd.so.0
0x2ab26001088f : ??? + 0x2ab26001088f in /home/dmarce1/debug/lib/libhpxd.so.0
0x2ab25f48d9ef : ??? + 0x2ab25f48d9ef in /home/dmarce1/debug/lib/libhpxd.so.0
{what}: Floating point exception
{config}:
HPX_HAVE_NATIVE_TLS=ON
HPX_HAVE_STACKTRACES=ON
HPX_HAVE_COMPRESSION_BZIP2=OFF
HPX_HAVE_COMPRESSION_SNAPPY=OFF
HPX_HAVE_COMPRESSION_ZLIB=OFF
HPX_HAVE_PARCEL_COALESCING=ON
HPX_HAVE_PARCELPORT_TCP=ON
HPX_HAVE_PARCELPORT_MPI=ON (MPICH V3.1.2, MPI V3.0)
HPX_HAVE_PARCELPORT_IPC=OFF
HPX_HAVE_PARCELPORT_IBVERBS=OFF
HPX_HAVE_VERIFY_LOCKS=ON
HPX_HAVE_HWLOC=ON
HPX_HAVE_ITTNOTIFY=OFF
HPX_HAVE_RUN_MAIN_EVERYWHERE=OFF
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
HPX_HAVE_MALLOC=custom
HPX_PREFIX (configured)=/work/dmarce1/builds/hpx_build
HPX_PREFIX=/home/dmarce1/debug
{version}: V0.9.12-trunk (AGAS: V3.0), Git: unknown
{boost}: V1.59.0
{build-type}: debug
{date}: Dec 28 2015 10:01:23
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1500
{stdlib}: GNU libstdc++ version 20140422
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 119390 RUNNING AT smic026
= EXIT CODE: 6
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment