octo-tiger seg fault output

gistfile1.txt

--------------------------------------
Running PBS prologue script
--------------------------------------
User and Job Data:
--------------------------------------
Job ID:    103947.smic3
Username:  dmarce1
Group:     hpcusers
Date:      23-Feb-2016 13:29
Node:      smic015 (117759)
--------------------------------------
PBS has allocated the following nodes:

smic015
smic016
smic020
smic021
smic022
smic024
smic025
smic027
smic028
smic033
smic034
smic036
smic037
smic041
smic042
smic045
smic046
smic047
smic048
smic049
smic050
smic053
smic056
smic057
smic060
smic065
smic068
smic069
smic070
smic071
smic073
smic074
smic075
smic076
smic080
smic083
smic084
smic088
smic089
smic094
smic096
smic097
smic098
smic099
smic101
smic102
smic104
smic105
smic106
smic108
smic110
smic113
smic114
smic116
smic118
smic119
smic122
smic125
smic126
smic128
smic131
smic133
smic135
smic136

A total of 1280 processors on 64 nodes allocated
---------------------------------------------
Check nodes and clean them of stray processes
---------------------------------------------
Checking node smic015 13:29:05 
Checking node smic016 13:29:07 
Checking node smic020 13:29:09 
Checking node smic021 13:29:12 
Checking node smic022 13:29:14 
Checking node smic024 13:29:16 
Checking node smic025 13:29:19 
Checking node smic027 13:29:21 
Checking node smic028 13:29:23 
Checking node smic033 13:29:26 
Checking node smic034 13:29:28 
Checking node smic036 13:29:30 
Checking node smic037 13:29:33 
Checking node smic041 13:29:35 
Checking node smic042 13:29:37 
Checking node smic045 13:29:39 
Checking node smic046 13:29:42 
Checking node smic047 13:29:44 
Checking node smic048 13:29:46 
Checking node smic049 13:29:49 
Checking node smic050 13:29:51 
Checking node smic053 13:29:53 
Checking node smic056 13:29:56 
Checking node smic057 13:29:58 
Checking node smic060 13:30:00 
Checking node smic065 13:30:03 
Checking node smic068 13:30:05 
Checking node smic069 13:30:07 
Checking node smic070 13:30:10 
Checking node smic071 13:30:12 
Checking node smic073 13:30:15 
Checking node smic074 13:30:17 
Checking node smic075 13:30:19 
Checking node smic076 13:30:22 
Checking node smic080 13:30:24 
Checking node smic083 13:30:26 
Checking node smic084 13:30:29 
Checking node smic088 13:30:31 
Checking node smic089 13:30:33 
Checking node smic094 13:30:36 
Checking node smic096 13:30:38 
Checking node smic097 13:30:40 
Checking node smic098 13:30:42 
Checking node smic099 13:30:45 
Checking node smic101 13:30:47 
Checking node smic102 13:30:49 
Checking node smic104 13:30:52 
Checking node smic105 13:30:54 
Checking node smic106 13:30:56 
Checking node smic108 13:30:59 
Checking node smic110 13:31:01 
Checking node smic113 13:31:03 
Checking node smic114 13:31:05 
Checking node smic116 13:31:08 
Checking node smic118 13:31:10 
Checking node smic119 13:31:12 
Checking node smic122 13:31:15 
Checking node smic125 13:31:17 
Checking node smic126 13:31:19 
Checking node smic128 13:31:21 
Checking node smic131 13:31:24 
Checking node smic133 13:31:26 
Checking node smic135 13:31:28 
Checking node smic136 13:31:31 
Done clearing all the allocated nodes
------------------------------------------------------
Concluding PBS prologue script - 23-Feb-2016 13:31:31
------------------------------------------------------
+ cd /work/dmarce1/octotiger
+ make clean
+ make -j20
[  5%] [ 11%] [ 17%] [ 23%] [ 29%] [ 35%] [ 41%] [ 47%] Building CXX object CMakeFiles/octotiger_exe.dir/src/node_client.cpp.o
Building CXX object CMakeFiles/octotiger_exe.dir/src/main.cpp.o
[ 52%] Building CXX object CMakeFiles/octotiger_exe.dir/src/grid.cpp.o
Building CXX object CMakeFiles/octotiger_exe.dir/src/node_server.cpp.o
[ 58%] Building CXX object CMakeFiles/octotiger_exe.dir/src/grid_fmm.cpp.o
[ 70%] [ 70%] Building CXX object CMakeFiles/octotiger_exe.dir/src/node_server_decomp.cpp.o
[ 76%] [ 82%] Building CXX object CMakeFiles/octotiger_exe.dir/src/node_server_output.cpp.o
Building CXX object CMakeFiles/octotiger_exe.dir/src/node_octstruct.cpp.o
[ 88%] [ 94%] Building CXX object CMakeFiles/octotiger_exe.dir/src/grid_scf.cpp.o
[100%] Building CXX object CMakeFiles/octotiger_exe.dir/src/grid_output.cpp.o
Building CXX object CMakeFiles/octotiger_exe.dir/src/lane_emden.cpp.o
Building CXX object CMakeFiles/octotiger_exe.dir/src/node_location.cpp.o
Building CXX object CMakeFiles/octotiger_exe.dir/src/problem.cpp.o
Building CXX object CMakeFiles/octotiger_exe.dir/src/geometry.cpp.o
Building CXX object CMakeFiles/octotiger_exe.dir/src/old_scf_read.cpp.o
Building CXX object CMakeFiles/octotiger_exe.dir/src/taylor.cpp.o
Building CXX object CMakeFiles/octotiger_exe.dir/src/roe.cpp.o
Linking CXX executable octotiger
[100%] Built target octotiger_exe
+ cat /var/spool/torque/aux//103947.smic3
+ awk 'NR % 20 == 0'
+ export HPX_NODEFILE=node.list
+ HPX_NODEFILE=node.list
+ export PBS_NODEFILE=node.list
+ PBS_NODEFILE=node.list
++ wc -l node.list
++ gawk '//{print $1}'
+ export NPROCS=64
+ NPROCS=64
+ rm data.bin
rm: cannot remove `data.bin': No such file or directory
+ mpirun -np 64 --machinefile node.list ./octotiger restart.chk -t20 -Ihpx.stacks.small_size=2097152 -Ihpx.stacks.medium_size=2097152 -Ihpx.stacks.large_size=2097152 -Ihpx.stacks.huge_size=2097152
Running
Loading from restart.chk...
Loaded checkpoint file
-----------------------------------------------
regridding
rebalancing 4217 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
Done. 
Forming tree connections------------
...done
Starting...
Saved 4217 grids to checkpoint file
0 3.085544e+01 1.182342e-03 7.435340e+00 2.891326e-01 4.451645e+00
1 3.085781e+01 1.185190e-03 7.712713e-01 2.891309e-01 4.451988e+00
2 3.086018e+01 1.187842e-03 7.591811e-01 2.891292e-01 4.452331e+00
3 3.086256e+01 1.190492e-03 7.731159e-01 2.891275e-01 4.452676e+00
4 3.086494e+01 1.193776e-03 7.765362e-01 2.891258e-01 4.453021e+00
5 3.086734e+01 1.198690e-03 7.687805e-01 2.891240e-01 4.453367e+00
6 3.086974e+01 1.204649e-03 7.649755e-01 2.891223e-01 4.453716e+00
7 3.087216e+01 1.215826e-03 7.629803e-01 2.891206e-01 4.454067e+00
8 3.087459e+01 1.218355e-03 7.861523e-01 2.891188e-01 4.454419e+00
9 3.087703e+01 1.218075e-03 7.587103e-01 2.891171e-01 4.454772e+00
10 3.087947e+01 1.217739e-03 7.982307e-01 2.891153e-01 4.455124e+00
11 3.088190e+01 1.217527e-03 7.659862e-01 2.891135e-01 4.455476e+00
12 3.088434e+01 1.217470e-03 7.546157e-01 2.891118e-01 4.455828e+00
13 3.088677e+01 1.217637e-03 7.635465e-01 2.891100e-01 4.456180e+00
14 3.088921e+01 1.218025e-03 8.101343e-01 2.891083e-01 4.456532e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4185 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
15 3.089164e+01 1.218596e-03 7.630411e-01 2.891065e-01 4.456884e+00
16 3.089408e+01 1.219333e-03 7.521396e-01 2.891047e-01 4.457237e+00
17 3.089652e+01 1.220827e-03 7.481843e-01 2.891029e-01 4.457590e+00
18 3.089897e+01 1.223693e-03 7.586251e-01 2.891012e-01 4.457943e+00
19 3.090142e+01 1.225651e-03 7.419421e-01 2.890994e-01 4.458298e+00
20 3.090387e+01 1.225435e-03 7.513301e-01 2.890976e-01 4.458652e+00
21 3.090632e+01 1.225265e-03 7.585401e-01 2.890958e-01 4.459006e+00
22 3.090877e+01 1.225130e-03 7.653723e-01 2.890940e-01 4.459360e+00
23 3.091122e+01 1.225036e-03 7.750566e-01 2.890922e-01 4.459714e+00
24 3.091367e+01 1.224951e-03 7.812746e-01 2.890904e-01 4.460069e+00
25 3.091612e+01 1.224317e-03 7.626488e-01 2.890886e-01 4.460423e+00
26 3.091856e+01 1.223064e-03 7.898739e-01 2.890868e-01 4.460776e+00
27 3.092101e+01 1.223232e-03 7.600080e-01 2.890850e-01 4.461130e+00
28 3.092346e+01 1.223349e-03 7.623880e-01 2.890832e-01 4.461483e+00
29 3.092590e+01 1.223368e-03 7.649504e-01 2.890814e-01 4.461837e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4169 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
30 3.092835e+01 1.223554e-03 7.576628e-01 2.890796e-01 4.462191e+00
31 3.093079e+01 1.218072e-03 7.653483e-01 2.890778e-01 4.462543e+00
32 3.093322e+01 1.212036e-03 7.500903e-01 2.890760e-01 4.462893e+00
33 3.093564e+01 1.207380e-03 7.715479e-01 2.890743e-01 4.463242e+00
34 3.093805e+01 1.205065e-03 7.878082e-01 2.890725e-01 4.463591e+00
35 3.094046e+01 1.204837e-03 7.638843e-01 2.890707e-01 4.463939e+00
36 3.094288e+01 1.206259e-03 7.729775e-01 2.890689e-01 4.464288e+00
37 3.094528e+01 1.203123e-03 7.587805e-01 2.890671e-01 4.464635e+00
38 3.094769e+01 1.202081e-03 7.674781e-01 2.890653e-01 4.464983e+00
39 3.095009e+01 1.202880e-03 7.577544e-01 2.890636e-01 4.465331e+00
40 3.095249e+01 1.196759e-03 7.631840e-01 2.890618e-01 4.465676e+00
41 3.095488e+01 1.190120e-03 7.834789e-01 2.890600e-01 4.466020e+00
42 3.095726e+01 1.184107e-03 7.699794e-01 2.890582e-01 4.466363e+00
43 3.095962e+01 1.176253e-03 7.525242e-01 2.890565e-01 4.466703e+00
44 3.096196e+01 1.168501e-03 7.697695e-01 2.890547e-01 4.467041e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4169 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
45 3.096429e+01 1.162086e-03 8.026125e-01 2.890530e-01 4.467376e+00
46 3.096661e+01 1.157555e-03 7.874698e-01 2.890512e-01 4.467711e+00
47 3.096892e+01 1.154231e-03 7.567593e-01 2.890495e-01 4.468045e+00
48 3.097123e+01 1.149734e-03 7.520545e-01 2.890478e-01 4.468377e+00
49 3.097352e+01 1.147608e-03 7.470258e-01 2.890460e-01 4.468709e+00
50 3.097582e+01 1.147655e-03 7.600249e-01 2.890443e-01 4.469040e+00
51 3.097812e+01 1.149719e-03 7.862055e-01 2.890426e-01 4.469373e+00
52 3.098042e+01 1.153559e-03 7.748213e-01 2.890409e-01 4.469706e+00
53 3.098273e+01 1.158900e-03 7.753882e-01 2.890391e-01 4.470041e+00
54 3.098505e+01 1.161968e-03 7.807141e-01 2.890374e-01 4.470377e+00
55 3.098738e+01 1.161793e-03 7.538716e-01 2.890357e-01 4.470713e+00
56 3.098970e+01 1.161303e-03 7.532132e-01 2.890339e-01 4.471048e+00
57 3.099202e+01 1.161012e-03 7.591544e-01 2.890322e-01 4.471384e+00
58 3.099435e+01 1.161957e-03 7.609845e-01 2.890304e-01 4.471720e+00
59 3.099667e+01 1.164260e-03 7.585059e-01 2.890287e-01 4.472056e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4169 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
60 3.099900e+01 1.164512e-03 7.533820e-01 2.890269e-01 4.472393e+00
61 3.100133e+01 1.166447e-03 7.503094e-01 2.890251e-01 4.472730e+00
62 3.100367e+01 1.170010e-03 7.953796e-01 2.890234e-01 4.473068e+00
63 3.100601e+01 1.175059e-03 7.640493e-01 2.890216e-01 4.473408e+00
64 3.100837e+01 1.180671e-03 7.592386e-01 2.890198e-01 4.473749e+00
65 3.101073e+01 1.181956e-03 8.028988e-01 2.890180e-01 4.474091e+00
66 3.101309e+01 1.179080e-03 8.271700e-01 2.890162e-01 4.474431e+00
67 3.101545e+01 1.181078e-03 7.625140e-01 2.890144e-01 4.474773e+00
68 3.101782e+01 1.181807e-03 7.763999e-01 2.890127e-01 4.475114e+00
69 3.102018e+01 1.181747e-03 7.744834e-01 2.890109e-01 4.475456e+00
70 3.102254e+01 1.179866e-03 8.060312e-01 2.890091e-01 4.475797e+00
71 3.102490e+01 1.177607e-03 7.364833e-01 2.890073e-01 4.476137e+00
72 3.102725e+01 1.176167e-03 7.589659e-01 2.890055e-01 4.476477e+00
73 3.102960e+01 1.175247e-03 7.656593e-01 2.890037e-01 4.476817e+00
74 3.103195e+01 1.174561e-03 8.086443e-01 2.890019e-01 4.477156e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4169 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
75 3.103430e+01 1.174019e-03 7.675743e-01 2.890001e-01 4.477496e+00
76 3.103665e+01 1.172965e-03 7.581880e-01 2.889983e-01 4.477835e+00
77 3.103899e+01 1.172573e-03 7.726473e-01 2.889965e-01 4.478173e+00
78 3.104134e+01 1.171608e-03 7.718287e-01 2.889947e-01 4.478512e+00
79 3.104368e+01 1.169096e-03 7.696402e-01 2.889929e-01 4.478850e+00
80 3.104602e+01 1.167247e-03 7.586761e-01 2.889911e-01 4.479187e+00
81 3.104835e+01 1.165710e-03 7.403909e-01 2.889893e-01 4.479524e+00
82 3.105068e+01 1.164403e-03 7.822551e-01 2.889875e-01 4.479861e+00
83 3.105301e+01 1.163638e-03 7.435672e-01 2.889857e-01 4.480197e+00
84 3.105533e+01 1.163166e-03 7.912764e-01 2.889840e-01 4.480533e+00
85 3.105766e+01 1.163544e-03 7.676927e-01 2.889822e-01 4.480869e+00
86 3.105999e+01 1.162408e-03 7.609710e-01 2.889804e-01 4.481205e+00
87 3.106231e+01 1.161590e-03 7.720763e-01 2.889786e-01 4.481541e+00
88 3.106463e+01 1.161859e-03 7.686933e-01 2.889768e-01 4.481877e+00
89 3.106695e+01 1.159235e-03 7.480946e-01 2.889750e-01 4.482212e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4169 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
90 3.106927e+01 1.157562e-03 7.338286e-01 2.889732e-01 4.482546e+00
91 3.107159e+01 1.156604e-03 7.497324e-01 2.889714e-01 4.482880e+00
92 3.107390e+01 1.156403e-03 7.662259e-01 2.889696e-01 4.483214e+00
93 3.107621e+01 1.156615e-03 7.880385e-01 2.889678e-01 4.483549e+00
94 3.107853e+01 1.157022e-03 7.720828e-01 2.889660e-01 4.483883e+00
95 3.108084e+01 1.156666e-03 7.681459e-01 2.889642e-01 4.484217e+00
96 3.108315e+01 1.155151e-03 7.596065e-01 2.889624e-01 4.484551e+00
97 3.108546e+01 1.153619e-03 7.445093e-01 2.889606e-01 4.484884e+00
98 3.108777e+01 1.152587e-03 7.920162e-01 2.889588e-01 4.485217e+00
99 3.109007e+01 1.152304e-03 7.743239e-01 2.889571e-01 4.485550e+00
100 3.109237e+01 1.152201e-03 7.542311e-01 2.889553e-01 4.485883e+00
101 3.109468e+01 1.151646e-03 7.666350e-01 2.889535e-01 4.486216e+00
102 3.109698e+01 1.151714e-03 7.731126e-01 2.889517e-01 4.486549e+00
103 3.109929e+01 1.152167e-03 7.721276e-01 2.889499e-01 4.486882e+00
104 3.110159e+01 1.153015e-03 7.708499e-01 2.889481e-01 4.487215e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4169 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
105 3.110390e+01 1.151885e-03 7.727483e-01 2.889463e-01 4.487548e+00
106 3.110620e+01 1.149893e-03 7.899428e-01 2.889445e-01 4.487880e+00
107 3.110850e+01 1.148123e-03 7.619522e-01 2.889427e-01 4.488212e+00
108 3.111079e+01 1.146631e-03 7.414718e-01 2.889409e-01 4.488543e+00
109 3.111308e+01 1.145657e-03 7.503404e-01 2.889391e-01 4.488874e+00
110 3.111537e+01 1.145321e-03 7.714902e-01 2.889373e-01 4.489205e+00
111 3.111766e+01 1.144937e-03 7.804732e-01 2.889355e-01 4.489536e+00
112 3.111995e+01 1.145149e-03 7.625827e-01 2.889337e-01 4.489867e+00
113 3.112225e+01 1.145952e-03 7.818301e-01 2.889319e-01 4.490198e+00
114 3.112454e+01 1.147285e-03 8.071382e-01 2.889301e-01 4.490529e+00
115 3.112683e+01 1.147473e-03 7.631064e-01 2.889283e-01 4.490861e+00
116 3.112913e+01 1.146382e-03 7.748115e-01 2.889264e-01 4.491192e+00
117 3.113142e+01 1.145434e-03 8.128144e-01 2.889246e-01 4.491523e+00
118 3.113371e+01 1.144724e-03 7.563336e-01 2.889228e-01 4.491854e+00
119 3.113600e+01 1.144054e-03 7.763035e-01 2.889210e-01 4.492184e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4185 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
120 3.113829e+01 1.143817e-03 8.069003e-01 2.889192e-01 4.492515e+00
121 3.114057e+01 1.144013e-03 7.758667e-01 2.889174e-01 4.492845e+00
122 3.114286e+01 1.145050e-03 7.901549e-01 2.889156e-01 4.493176e+00
123 3.114515e+01 1.147017e-03 7.697271e-01 2.889138e-01 4.493507e+00
124 3.114745e+01 1.149341e-03 7.737468e-01 2.889120e-01 4.493840e+00
125 3.114975e+01 1.151222e-03 7.754550e-01 2.889101e-01 4.494172e+00
126 3.115205e+01 1.151859e-03 7.510789e-01 2.889083e-01 4.494505e+00
127 3.115436e+01 1.151841e-03 7.582638e-01 2.889065e-01 4.494838e+00
128 3.115666e+01 1.151908e-03 7.672210e-01 2.889047e-01 4.495170e+00
129 3.115897e+01 1.152139e-03 7.512505e-01 2.889028e-01 4.495503e+00
130 3.116127e+01 1.152409e-03 7.976784e-01 2.889010e-01 4.495836e+00
131 3.116358e+01 1.152689e-03 7.634847e-01 2.888992e-01 4.496169e+00
132 3.116588e+01 1.153030e-03 7.640664e-01 2.888973e-01 4.496502e+00
133 3.116819e+01 1.153622e-03 7.857030e-01 2.888955e-01 4.496836e+00
134 3.117050e+01 1.154637e-03 7.829220e-01 2.888937e-01 4.497169e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4201 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
135 3.117281e+01 1.156106e-03 7.607135e-01 2.888918e-01 4.497503e+00
136 3.117512e+01 1.158209e-03 7.612680e-01 2.888900e-01 4.497838e+00
137 3.117744e+01 1.160250e-03 7.823895e-01 2.888881e-01 4.498173e+00
138 3.117976e+01 1.160838e-03 7.815610e-01 2.888863e-01 4.498508e+00
139 3.118208e+01 1.160993e-03 7.663486e-01 2.888844e-01 4.498844e+00
140 3.118441e+01 1.161614e-03 7.600089e-01 2.888826e-01 4.499179e+00
141 3.118673e+01 1.162620e-03 8.325222e-01 2.888807e-01 4.499515e+00
142 3.118906e+01 1.163757e-03 7.785559e-01 2.888789e-01 4.499851e+00
143 3.119138e+01 1.164923e-03 7.754858e-01 2.888770e-01 4.500188e+00
144 3.119372e+01 1.165877e-03 7.654652e-01 2.888751e-01 4.500525e+00
145 3.119605e+01 1.165568e-03 7.591686e-01 2.888733e-01 4.500861e+00
146 3.119838e+01 1.165648e-03 7.756241e-01 2.888714e-01 4.501198e+00
147 3.120071e+01 1.165957e-03 7.861761e-01 2.888695e-01 4.501535e+00
148 3.120304e+01 1.166401e-03 7.640410e-01 2.888677e-01 4.501872e+00
149 3.120538e+01 1.166856e-03 7.519192e-01 2.888658e-01 4.502209e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4217 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
150 3.120771e+01 1.167136e-03 7.754973e-01 2.888639e-01 4.502546e+00
151 3.121004e+01 1.167529e-03 7.902348e-01 2.888621e-01 4.502883e+00
152 3.121238e+01 1.168132e-03 7.719954e-01 2.888602e-01 4.503221e+00
153 3.121472e+01 1.168896e-03 7.691818e-01 2.888583e-01 4.503558e+00
154 3.121706e+01 1.169343e-03 7.698354e-01 2.888564e-01 4.503896e+00
155 3.121939e+01 1.168899e-03 7.810022e-01 2.888546e-01 4.504234e+00
156 3.122173e+01 1.168626e-03 7.910360e-01 2.888527e-01 4.504571e+00
157 3.122407e+01 1.168524e-03 7.806767e-01 2.888508e-01 4.504909e+00
158 3.122641e+01 1.168693e-03 7.828159e-01 2.888489e-01 4.505246e+00
159 3.122874e+01 1.169084e-03 7.630737e-01 2.888471e-01 4.505584e+00
160 3.123108e+01 1.169601e-03 7.645037e-01 2.888452e-01 4.505922e+00
161 3.123342e+01 1.170234e-03 7.859167e-01 2.888433e-01 4.506260e+00
162 3.123576e+01 1.170979e-03 7.844194e-01 2.888414e-01 4.506598e+00
163 3.123811e+01 1.171831e-03 7.456921e-01 2.888395e-01 4.506937e+00
164 3.124045e+01 1.173549e-03 7.797819e-01 2.888376e-01 4.507276e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4201 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
165 3.124280e+01 1.174468e-03 7.831745e-01 2.888358e-01 4.507615e+00
166 3.124515e+01 1.174617e-03 7.688859e-01 2.888339e-01 4.507954e+00
167 3.124750e+01 1.174781e-03 7.972022e-01 2.888320e-01 4.508293e+00
168 3.124985e+01 1.175100e-03 7.876715e-01 2.888301e-01 4.508633e+00
169 3.125220e+01 1.175547e-03 8.089244e-01 2.888282e-01 4.508972e+00
170 3.125455e+01 1.176153e-03 7.834593e-01 2.888263e-01 4.509312e+00
171 3.125690e+01 1.176991e-03 7.902545e-01 2.888244e-01 4.509652e+00
172 3.125926e+01 1.178056e-03 8.167954e-01 2.888225e-01 4.509992e+00
173 3.126161e+01 1.178940e-03 7.709183e-01 2.888206e-01 4.510333e+00
174 3.126397e+01 1.179602e-03 7.995571e-01 2.888187e-01 4.510673e+00
175 3.126633e+01 1.180493e-03 8.095876e-01 2.888168e-01 4.511014e+00
176 3.126870e+01 1.181315e-03 7.614509e-01 2.888149e-01 4.511356e+00
177 3.127106e+01 1.182270e-03 7.773515e-01 2.888130e-01 4.511697e+00
178 3.127342e+01 1.183023e-03 7.736930e-01 2.888111e-01 4.512039e+00
179 3.127579e+01 1.183765e-03 7.775540e-01 2.888092e-01 4.512381e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4225 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
180 3.127816e+01 1.184456e-03 7.830502e-01 2.888073e-01 4.512723e+00
181 3.128053e+01 1.185534e-03 7.672997e-01 2.888054e-01 4.513065e+00
182 3.128290e+01 1.185373e-03 7.738819e-01 2.888035e-01 4.513407e+00
183 3.128527e+01 1.185374e-03 7.826453e-01 2.888016e-01 4.513750e+00
184 3.128764e+01 1.185738e-03 7.630048e-01 2.887996e-01 4.514092e+00
185 3.129001e+01 1.186377e-03 8.034799e-01 2.887977e-01 4.514435e+00
186 3.129239e+01 1.187279e-03 7.981289e-01 2.887958e-01 4.514778e+00
187 3.129476e+01 1.187779e-03 7.841929e-01 2.887939e-01 4.515121e+00
188 3.129714e+01 1.187975e-03 8.092952e-01 2.887920e-01 4.515464e+00
189 3.129951e+01 1.188176e-03 7.707662e-01 2.887901e-01 4.515807e+00
190 3.130189e+01 1.188097e-03 7.767486e-01 2.887882e-01 4.516150e+00
191 3.130427e+01 1.187742e-03 7.948201e-01 2.887862e-01 4.516493e+00
192 3.130664e+01 1.187745e-03 7.875691e-01 2.887843e-01 4.516836e+00
193 3.130902e+01 1.188344e-03 8.132866e-01 2.887824e-01 4.517179e+00
194 3.131140e+01 1.188337e-03 7.667217e-01 2.887805e-01 4.517522e+00
-----------------------------------------------
checking for refinement
regridding
rebalancing 4241 nodes
forming tree connections
solving gravity
regrid done
-----------------------------------------------
195 3.131377e+01 1.188347e-03 7.884178e-01 2.887786e-01 4.517866e+00
196 3.131615e+01 1.188406e-03 7.685781e-01 2.887767e-01 4.518209e+00
197 3.131853e+01 1.188465e-03 7.946079e-01 2.887747e-01 4.518552e+00
198 3.132090e+01 1.188528e-03 7.678213e-01 2.887728e-01 4.518895e+00
199 3.132328e+01 1.188704e-03 7.787766e-01 2.887709e-01 4.519238e+00
200 3.132566e+01 1.188919e-03 7.623276e-01 2.887690e-01 4.519582e+00
201 3.132804e+01 1.189169e-03 7.781843e-01 2.887671e-01 4.519925e+00
Saved 4241 grids to checkpoint file
202 3.133041e+01 1.189210e-03 7.353921e+00 2.887652e-01 4.520269e+00
{stack-trace}: 7 frames:
0x2b66f99abd48  : hpx::termination_handler(int) + 0x158 in /home/dmarce1/debug/lib/libhpxd.so.0
0x332300f710    : ??? + 0x332300f710 in /lib64/libpthread.so.0
0x7e7cf3        : hpx::lcos::base_lco_with_value<hpx::util::unused_type, hpx::util::unused_type>::set_value_nonvirt(hpx::util::unused_type&&) + 0x3 in ./octotiger
0x7939e5        : hpx::util::detail::bound_result_of<hpx::util::detail::one_shot_wrapper<hpx::actions::basic_action<hpx::lcos::base_lco_with_value<hpx::util::unused_type, hpx::util::unused_type>, void (hpx::util::unused_type&&), hpx::lcos::base_lco_with_value<hpx::util::unused_type, hpx::util::unused_type>::set_value_action>::thread_function>, hpx::util::tuple<unsigned long, hpx::util::unused_type>, hpx::util::tuple<hpx::threads::thread_state_ex_enum&&> >::type hpx::util::detail::bound_impl<hpx::util::detail::one_shot_wrapper<hpx::actions::basic_action<hpx::lcos::base_lco_with_value<hpx::util::unused_type, hpx::util::unused_type>, void (hpx::util::unused_type&&), hpx::lcos::base_lco_with_value<hpx::util::unused_type, hpx::util::unused_type>::set_value_action>::thread_function>, hpx::util::tuple<unsigned long, hpx::util::unused_type>, hpx::util::tuple<hpx::threads::thread_state_ex_enum&&>, 0ul, 1ul>(hpx::util::detail::one_shot_wrapper<hpx::actions::basic_action<hpx::lcos::base_lco_with_value<hpx::util::unused_type, hpx::util::unused_type>, void (hpx::util::unused_type&&), hpx::lcos::base_lco_with_value<hpx::util::unused_type, hpx::util::unused_type>::set_value_action>::thread_function>&, hpx::util::tuple<unsigned long, hpx::util::unused_type>&, hpx::util::tuple<hpx::threads::thread_state_ex_enum&&>&&, hpx::util::detail::pack_c<unsigned long, 0ul>) + 0x2d5 in ./octotiger
0x7936f3        : hpx::threads::thread_state_enum hpx::util::detail::callable_vtable<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>::invoke<hpx::util::detail::bound<hpx::util::detail::one_shot_wrapper<hpx::actions::basic_action<hpx::lcos::base_lco_with_value<hpx::util::unused_type, hpx::util::unused_type>, void (hpx::util::unused_type&&), hpx::lcos::base_lco_with_value<hpx::util::unused_type, hpx::util::unused_type>::set_value_action>::thread_function> (unsigned long&, hpx::util::unused_type&&)> >(void**, hpx::threads::thread_state_ex_enum&&) + 0x23 in ./octotiger
0x2b66f98eab6e  : ??? + 0x2b66f98eab6e in /home/dmarce1/debug/lib/libhpxd.so.0
{what}: Segmentation fault
{config}:
  HPX_HAVE_NATIVE_TLS=ON
  HPX_HAVE_STACKTRACES=ON
  HPX_HAVE_COMPRESSION_BZIP2=OFF
  HPX_HAVE_COMPRESSION_SNAPPY=OFF
  HPX_HAVE_COMPRESSION_ZLIB=OFF
  HPX_HAVE_PARCEL_COALESCING=ON
  HPX_HAVE_PARCELPORT_TCP=ON
  HPX_HAVE_PARCELPORT_MPI=ON (MPICH V3.1.2, MPI V3.0)
  HPX_HAVE_PARCELPORT_IPC=OFF
  HPX_HAVE_PARCELPORT_IBVERBS=OFF
  HPX_HAVE_VERIFY_LOCKS=ON
  HPX_HAVE_HWLOC=ON
  HPX_HAVE_ITTNOTIFY=OFF
  HPX_HAVE_RUN_MAIN_EVERYWHERE=OFF
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_AGAS_LOCAL_CACHE_SIZE=4096
  HPX_HAVE_MALLOC=custom
  HPX_PREFIX (configured)=/work/dmarce1/builds/hpx_build
  HPX_PREFIX=/home/dmarce1/debug
{version}: V0.9.12-trunk (AGAS: V3.0), Git: unknown
{boost}: V1.59.0
{build-type}: debug
{date}: Feb 18 2016 11:40:46
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1500
{stdlib}: GNU libstdc++ version 20140422

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 93444 RUNNING AT smic045
=   EXIT CODE: 6
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
------------------------------------------------------
Running PBS epilogue script    - 23-Feb-2016 13:37:10
------------------------------------------------------
Checking node smic015 (MS)
Checking node smic136 ok
Checking node smic135 ok
Checking node smic133 ok
Checking node smic131 ok
Checking node smic128 ok
Checking node smic126 ok
Checking node smic125 ok
Checking node smic122 ok
Checking node smic119 ok
Checking node smic118 ok
Checking node smic116 ok
Checking node smic114 ok
Checking node smic113 ok
Checking node smic110 ok
Checking node smic108 ok
Checking node smic106 ok
Checking node smic105 ok
Checking node smic104 ok
Checking node smic102 ok
Checking node smic101 ok
Checking node smic099 ok
Checking node smic098 ok
Checking node smic097 ok
Checking node smic096 ok
Checking node smic094 ok
Checking node smic089 ok
Checking node smic088 ok
Checking node smic084 ok
Checking node smic083 ok
Checking node smic080 ok
Checking node smic076 ok
Checking node smic075 ok
Checking node smic074 ok
Checking node smic073 ok
Checking node smic071 ok
Checking node smic070 ok
Checking node smic069 ok
Checking node smic068 ok
Checking node smic065 ok
Checking node smic060 ok
Checking node smic057 ok
Checking node smic056 ok
Checking node smic053 ok
Checking node smic050 ok
Checking node smic049 ok
Checking node smic048 ok
Checking node smic047 ok
Checking node smic046 ok
Checking node smic045 ok
Checking node smic042 ok
Checking node smic041 ; modules change (oprofile 47133 <;) ; ok
Checking node smic037 ok
Checking node smic036 ok
Checking node smic034 ok
Checking node smic033 ok
Checking node smic028 ok
Checking node smic027 ok
Checking node smic025 ok
Checking node smic024 ok
Checking node smic022 ok
Checking node smic021 ok
Checking node smic020 ok
Checking node smic016 ok
Checking node smic015 ok
------------------------------------------------------
Concluding PBS epilogue script - 23-Feb-2016 13:39:42
------------------------------------------------------
Exit Status:     6
Job ID:          103947.smic3
Username:        dmarce1
Group:           hpcusers
Job Name:        job
Session Id:      117758
Resource Limits: ncpus=1,neednodes=64:ppn=20,nodes=64:ppn=20,walltime=24:00:00
Resources Used:  cput=01:13:09,mem=16537640kb,vmem=21928104kb,walltime=00:05:39
Queue Used:      checkpt
Account String:  hpc_supermic01
Node:            smic015
Process id:      119364
------------------------------------------------------