Skip to content

Instantly share code, notes, and snippets.

@scivision
Last active December 17, 2018 07:34
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save scivision/59d62f02bc73e84f514a71489ad177ef to your computer and use it in GitHub Desktop.
Gemini failures GNU+MKL
$ ./gemini initialize/test2d/config.ini /tmp/2dt
Process: 0 of: 0 online...
initialize/test2d/config.ini: simulation ymd is: 2013/02/20
start time is: 18000.000
duration is: 300.000
output every: 60.000
...using input data files:
../simulations/zenodo2d/inputs/simsize.dat
../simulations/zenodo2d/inputs/simgrid.dat
../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
Precipitation file input cadence (s): 5.000
Precipitation file input source directory: ../simulations/zenodo2d/inputs/prec_inputs/
Electric field file input cadence (s): 1.0000000000000000
Electric field file input source directory: ../simulations/zenodo2d/inputs/Efield_inputs/
Grid has size: 64 40 1
Detected a 2D run request... swapping x2 an x3 sizes to maintain parallelization.
Starting grid input from file: ../simulations/zenodo2d/inputs/simgrid.dat
Detected a 2D grid, so will permute the dimensions of the input
Done reading in grid data...
Done sending common variables to workers...
Done sending slabbed variables to workers...
Done computing null grid points... Process: 0 has: 0
mkdir: created directory '/tmp/2dt'
mkdir: created directory '/tmp/2dt/inputs'
mkdir: created directory '/tmp/2dt/inputs/prec_inputs'
mkdir: created directory '/tmp/2dt/inputs/Efield_inputs'
Assembling initial condition on root using ../simulations/zenodo2d/inputs/simsize.dat ../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
Input file has size: 64 40 1
Target grid structure has size 64 1 40
2D simulations grid detected, swapping input file dimension sizes and permuting input arrays
Done gathering input...
Done setting initial conditions...
Min/max input density: 0.0000000000000000 311058103128.71814
Min/max input velocity: -175.63834785418018 66.673464467868655
Min/max input temperature: 0.0000000000000000 1500.0304895266988
Done sending ICs to workers... CPU elapsed time: 8.6599999999999178E-004
dt figured to be: 9.9999999999999995E-007
x1,x2,x3 courant numbers (root process only!):
0.22, 0.00, 0.00
0.16, 0.00, 0.00
0.16, 0.00, 0.00
0.16, 0.00, 0.00
0.23, 0.00, 0.00
0.90, 0.00, 0.00
0.00, 0.00, 0.00
Min and max density: 0.0000000000000000 311058103128.71814
Neutral background calculated in time: 1.0658999999999988E-002
Conductivities for time step: 0.0000000000000000 took 2.6629999999999987E-003 seconds...
Inputting electric field data size from file: ../simulations/zenodo2d/inputs/Efield_inputs//simsize.dat
Electric field data has llon,llat size: 100 1
Inputting electric field grid from file: ../simulations/zenodo2d/inputs/Efield_inputs//simgrid.dat
Electric field data has mlon,mlat extent: 256.66078848318182 258.32500381597765 67.252718387721629 67.252718387721629
Grid has mlon,mlat range: 256.67710431977787 258.30868797938160 67.252718387721629 67.252718387721629
Grid has size: 40
tprev,tnow,tnext: 0.0000000000000000 4.9999999999999998E-007 0.0000000000000000
Pulling electric field data from file: ../simulations/zenodo2d/inputs/Efield_inputs//20130220_18001.000000.dat
Min/max values for E0xp: 0.0000000000000000 0.0000000000000000
Min/max values for E0yp: 0.0000000000000000 0.0000000000000000
Min/max values for Vminx1p: 0.0000000000000000 0.0000000000000000
Min/max values for Vmaxx1p: -1266.0379018899018 1266.0379018899018
Min/max values for Vminx2pslice: 0.0000000000000000 0.0000000000000000
Min/max values for Vmaxx2pslice: 0.0000000000000000 0.0000000000000000
Min/max values for Vminx3pslice: 0.0000000000000000 0.0000000000000000
Min/max values for Vmaxx3pslice: 0.0000000000000000 0.0000000000000000
Initiating electric field boundary condition spatial interpolations for date: 2013 2 20 18001.000000000000
Singleton latitude dimension detected; interpolating in longitude...
Min/max values for E0xi: 0.0000000000000000 0.0000000000000000
Min/max values for E0yi: 0.0000000000000000 0.0000000000000000
Min/max values for Vminx1i: 0.0000000000000000 0.0000000000000000
Min/max values for Vmaxx1i: -1266.0375210727786 1266.0375210727786
Solve type: 1
tprev,t,tnext: 0.0000000000000000 4.9999999999999998E-007 1.0000000000000000
Min/max values for E0xinow: 0.0000000000000000 0.0000000000000000
Min/max values for E0yinow: 0.0000000000000000 0.0000000000000000
Min/max values for Vminx1inow: 0.0000000000000000 0.0000000000000000
Min/max values for Vmaxx1inow: -6.3301876053638924E-004 6.3301876053638924E-004
Root has computed BCs in time: 3.0000000000000859E-004
Root has communicated type of solve to workers: 1
Root has computed background field source terms... 0.0000000000000000 0.0000000000000000
Root has computed wind source terms... 0.0000000000000000 0.0000000000000000
Beginning field-resolved 2D solve... Type; 1
MUMPS will attempt a solve of size: 64 40
Total unknowns and nonzero entries in matrix: 2560 12024
Number of entries used: 12024
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x7fc9f16282da in ???
#1 0x7fc9f1627503 in ???
#2 0x7fc9f0c65f1f in ???
#3 0x7fc9f032f3a1 in ???
#4 0x7fc9f1057aca in ???
#5 0x555eaa632128 in ???
#6 0x555eaa5f4a1c in __potential_mumps_MOD_elliptic2d_nonint_curv
at ~/gemini/numerical/potential/potential_mumps.F90:1587
#7 0x555eaa596794 in __potential_comm_MOD_potential_root_mpi_curv
at ~/gemini/numerical/potential/potential_comm_mumps.f90:558
#8 0x555eaa5a219a in __potential_comm_MOD_electrodynamics_curv
at ~/gemini/numerical/potential/potential_comm_mumps.f90:136
#9 0x555eaa4de27e in gemini
at ~/gemini/gemini.f90:214
#10 0x555eaa4dba1e in main
at ~/gemini/gemini.f90:7
Segmentation fault (core dumped)
$ ctest -R 2D --output-on-failure
Test project ~/gemini/objects
Start 1: Gemini2D
1/6 Test #1: Gemini2D .........................***Failed 0.57 sec
Process: 0 of: 0 online...
Process: 0 of: 0 online...
Process: 0 of: 0 online...
Process: 0 of: 0 online...
~/gemini/initialize/test2d/config.ini: simulation ymd is: 2013/02/20
~/gemini/initialize/test2d/config.ini: simulation ymd is: 2013/02/20
start time is: 18000.000
start time is: 18000.000
duration is: 300.000
duration is: 300.000
output every: 60.000
...using input data files:
output every: 60.000
...using input data files:
../simulations/zenodo2d/inputs/simsize.dat
../simulations/zenodo2d/inputs/simgrid.dat
../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
../simulations/zenodo2d/inputs/simsize.dat
../simulations/zenodo2d/inputs/simgrid.dat
../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
~/gemini/initialize/test2d/config.ini: simulation ymd is: 2013/02/20
Precipitation file input cadence (s): 5.000
Precipitation file input source directory: ../simulations/zenodo2d/inputs/prec_inputs/
start time is: 18000.000
duration is: 300.000
Precipitation file input cadence (s): 5.000
Precipitation file input source directory: ../simulations/zenodo2d/inputs/prec_inputs/
output every: 60.000
...using input data files:
../simulations/zenodo2d/inputs/simsize.dat
../simulations/zenodo2d/inputs/simgrid.dat
../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
Electric field file input cadence (s): 1.0000000000000000
Electric field file input source directory: ../simulations/zenodo2d/inputs/Efield_inputs/
Electric field file input cadence (s): 1.0000000000000000
Electric field file input source directory: ../simulations/zenodo2d/inputs/Efield_inputs/
Precipitation file input cadence (s): 5.000
Precipitation file input source directory: ../simulations/zenodo2d/inputs/prec_inputs/
~/gemini/initialize/test2d/config.ini: simulation ymd is: 2013/02/20
Electric field file input cadence (s): 1.0000000000000000
Electric field file input source directory: ../simulations/zenodo2d/inputs/Efield_inputs/
start time is: 18000.000
duration is: 300.000
output every: 60.000
...using input data files:
../simulations/zenodo2d/inputs/simsize.dat
../simulations/zenodo2d/inputs/simgrid.dat
../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
Grid has size: 64 40 1
Detected a 2D run request... swapping x2 an x3 sizes to maintain parallelization.
Grid has size: 64 40 1
Detected a 2D run request... swapping x2 an x3 sizes to maintain parallelization.
Grid has size: 64 40 1
Detected a 2D run request... swapping x2 an x3 sizes to maintain parallelization.
Precipitation file input cadence (s): 5.000
Precipitation file input source directory: ../simulations/zenodo2d/inputs/prec_inputs/
Electric field file input cadence (s): 1.0000000000000000
Electric field file input source directory: ../simulations/zenodo2d/inputs/Efield_inputs/
Starting grid input from file: ../simulations/zenodo2d/inputs/simgrid.dat
Detected a 2D grid, so will permute the dimensions of the input
Grid has size: 64 40 1
Detected a 2D run request... swapping x2 an x3 sizes to maintain parallelization.
Starting grid input from file: ../simulations/zenodo2d/inputs/simgrid.dat
Starting grid input from file: ../simulations/zenodo2d/inputs/simgrid.dat
Detected a 2D grid, so will permute the dimensions of the input
Detected a 2D grid, so will permute the dimensions of the input
Starting grid input from file: ../simulations/zenodo2d/inputs/simgrid.dat
Detected a 2D grid, so will permute the dimensions of the input
Done reading in grid data...
Done sending common variables to workers...
Done reading in grid data...
Done sending common variables to workers...
Done reading in grid data...
Done sending common variables to workers...
Done reading in grid data...
Done sending common variables to workers...
Done sending slabbed variables to workers...
Done computing null grid points... Process: 0 has: 0
Done sending slabbed variables to workers...
Done sending slabbed variables to workers...
Done computing null grid points... Process: 0 has: 0
Done computing null grid points... Process: 0 has: 0
Done sending slabbed variables to workers...
Done computing null grid points... Process: 0 has: 0
mkdir: created directory '~/gemini/objects/test2d'
mkdir: created directory '~/gemini/objects/test2d/inputs'
cp: cannot create regular file '~/gemini/objects/test2d/inputs/config.ini'cp: cannot create regular file '~/gemini/objects/test2d/inputs/config.ini'cp: cannot create regular file '~/gemini/objects/test2d/inputs/config.ini': File exists
: File exists
: File exists
ERROR STOP error copying input parameters to output directory
Error termination. Backtrace:
ERROR STOP error copying input parameters to output directory
Error termination. Backtrace:
ERROR STOP error copying input parameters to output directory
Error termination. Backtrace:
mkdir: created directory '~/gemini/objects/test2d/inputs/prec_inputs'
mkdir: created directory '~/gemini/objects/test2d/inputs/Efield_inputs'
Assembling initial condition on root using ../simulations/zenodo2d/inputs/simsize.dat ../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
Input file has size: 64 40 1
Target grid structure has size 64 1 40
2D simulations grid detected, swapping input file dimension sizes and permuting input arrays
Done gathering input...
Done setting initial conditions...
Min/max input density: 0.0000000000000000 311058103128.71814
Min/max input velocity: -175.63834785418018 66.673464467868655
Min/max input temperature: 0.0000000000000000 1500.0304895266988
Done sending ICs to workers... CPU elapsed time: 8.4299999999999653E-004
dt figured to be: 9.9999999999999995E-007
x1,x2,x3 courant numbers (root process only!):
0.22, 0.00, 0.00
0.16, 0.00, 0.00
0.16, 0.00, 0.00
0.16, 0.00, 0.00
0.23, 0.00, 0.00
0.90, 0.00, 0.00
0.00, 0.00, 0.00
Min and max density: 0.0000000000000000 311058103128.71814
Neutral background calculated in time: 1.0806999999999997E-002
Conductivities for time step: 0.0000000000000000 took 2.7519999999999975E-003 seconds...
Inputting electric field data size from file: ../simulations/zenodo2d/inputs/Efield_inputs//simsize.dat
Electric field data has llon,llat size: 100 1
Inputting electric field grid from file: ../simulations/zenodo2d/inputs/Efield_inputs//simgrid.dat
Electric field data has mlon,mlat extent: 256.66078848318182 258.32500381597765 67.252718387721629 67.252718387721629
Grid has mlon,mlat range: 256.67710431977787 258.30868797938160 67.252718387721629 67.252718387721629
Grid has size: 40
tprev,tnow,tnext: 0.0000000000000000 4.9999999999999998E-007 0.0000000000000000
Pulling electric field data from file: ../simulations/zenodo2d/inputs/Efield_inputs//20130220_18001.000000.dat
Min/max values for E0xp: 0.0000000000000000 0.0000000000000000
Min/max values for E0yp: 0.0000000000000000 0.0000000000000000
Min/max values for Vminx1p: 0.0000000000000000 0.0000000000000000
Min/max values for Vmaxx1p: -1266.0379018899018 1266.0379018899018
Min/max values for Vminx2pslice: 0.0000000000000000 0.0000000000000000
Min/max values for Vmaxx2pslice: 0.0000000000000000 0.0000000000000000
Min/max values for Vminx3pslice: 0.0000000000000000 0.0000000000000000
Min/max values for Vmaxx3pslice: 0.0000000000000000 0.0000000000000000
Initiating electric field boundary condition spatial interpolations for date: 2013 2 20 18001.000000000000
Singleton latitude dimension detected; interpolating in longitude...
Min/max values for E0xi: 0.0000000000000000 0.0000000000000000
Min/max values for E0yi: 0.0000000000000000 0.0000000000000000
Min/max values for Vminx1i: 0.0000000000000000 0.0000000000000000
Min/max values for Vmaxx1i: -1266.0375210727786 1266.0375210727786
Solve type: 1
tprev,t,tnext: 0.0000000000000000 4.9999999999999998E-007 1.0000000000000000
Min/max values for E0xinow: 0.0000000000000000 0.0000000000000000
Min/max values for E0yinow: 0.0000000000000000 0.0000000000000000
Min/max values for Vminx1inow: 0.0000000000000000 0.0000000000000000
Min/max values for Vmaxx1inow: -6.3301876053638924E-004 6.3301876053638924E-004
Root has computed BCs in time: 2.4400000000000810E-004
Root has communicated type of solve to workers: 1
Root has computed background field source terms... 0.0000000000000000 0.0000000000000000
Root has computed wind source terms... 0.0000000000000000 0.0000000000000000
Beginning field-resolved 2D solve... Type; 1
MUMPS will attempt a solve of size: 64 40
Total unknowns and nonzero entries in matrix: 2560 12024
Number of entries used: 12024
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x7fc8775252da in ???
#1 0x7fc877525ec5 in ???
#2 0x7fc877526f76 in ???
#0 0x7fe4e71682da in ???
#1 0x7fe4e7168ec5 in ???
#2 0x7fe4e7169f76 in ???
#0 0x7f87023962da in ???
#1 0x7f8702396ec5 in ???
#2 0x7f8702397f76 in ???
#3 0x5654f0f8c936 in __io_MOD_create_outdir
at ~/gemini/io/io.F90:171
#3 0x5601f1f5a936 in __io_MOD_create_outdir
at ~/gemini/io/io.F90:171
#3 0x55970c1e6936 in __io_MOD_create_outdir
at ~/gemini/io/io.F90:171
#4 0x5654f0f6f2fe in gemini
at ~/gemini/gemini.f90:116
#5 0x5654f0f6ba1e in main
#4 0x5601f1f3d2fe in gemini
at ~/gemini/gemini.f90:116
#5 0x5601f1f39a1e in main
at ~/gemini/gemini.f90:7
at ~/gemini/gemini.f90:7
#4 0x55970c1c92fe in gemini
at ~/gemini/gemini.f90:116
#5 0x55970c1c5a1e in main
at ~/gemini/gemini.f90:7
#0 0x7fe2874532da in ???
#1 0x7fe287452503 in ???
#2 0x7fe286a90f1f in ???
#3 0x7fe28615a3a1 in ???
#4 0x7fe286e82aca in ???
#5 0x56406a6c8128 in ???
#6 0x56406a68aa1c in __potential_mumps_MOD_elliptic2d_nonint_curv
at ~/gemini/numerical/potential/potential_mumps.F90:1587
#7 0x56406a62c794 in __potential_comm_MOD_potential_root_mpi_curv
at ~/gemini/numerical/potential/potential_comm_mumps.f90:558
#8 0x56406a63819a in __potential_comm_MOD_electrodynamics_curv
at ~/gemini/numerical/potential/potential_comm_mumps.f90:136
#9 0x56406a57427e in gemini
at ~/gemini/gemini.f90:214
#10 0x56406a571a1e in main
at ~/gemini/gemini.f90:7
Start 2: Compare2D
2/6 Test #2: Compare2D ........................***Failed 3.00 sec
sim grid dimensions: 64 40 1
~/gemini/objects/test2d/20130220_18000.000001.dat does not exist
$ ctest --output-on-failure -R 3D
Test project ~/gemini/objects
Start 3: Gemini3D
1/2 Test #3: Gemini3D ......................... Passed 226.41 sec
Start 4: Compare3D
2/2 Test #4: Compare3D ........................***Failed 3.64 sec
sim grid dimensions: 64 20 20
sim grid dimensions: 64 20 20
sim grid dimensions: 64 20 20
sim grid dimensions: 64 20 20
warning: AssertionError: Ne UTsec 18060 20.51% failed accuracy. maximum error magnitude 12577723571.5503 Actual: 48312807218.6193 Desired: 35735083647.0691 atol: 1000000 rtol: 1e-05
warning: AssertionError: Te UTsec 18060 18.00% failed accuracy. maximum error magnitude 1360.5535 Actual: 122.0907 Desired: 1482.6442 atol: 1 rtol: 1e-05
warning: AssertionError: J1 UTsec 18060 15.45% failed accuracy. maximum error magnitude 4.8742e-06 Actual: 7.2111e-06 Desired: 2.3369e-06 atol: 1e-09 rtol: 1e-05
warning: AssertionError: J2 UTsec 18060 2.78% failed accuracy. maximum error magnitude 2.755e-08 Actual: -4.8508e-06 Desired: -4.8783e-06 atol: 1e-09 rtol: 1e-05
warning: AssertionError: J3 UTsec 18060 0.38% failed accuracy. maximum error magnitude 5.0262e-08 Actual: 1.0458e-05 Desired: 1.0509e-05 atol: 1e-09 rtol: 1e-05
sim grid dimensions: 64 20 20
sim grid dimensions: 64 20 20
warning: AssertionError: Ne UTsec 18120 32.66% failed accuracy. maximum error magnitude 76252575851.3119 Actual: 87639633435.7617 Desired: 11387057584.4498 atol: 1000000 rtol: 1e-05
warning: AssertionError: Te UTsec 18120 26.68% failed accuracy. maximum error magnitude 1494.3927 Actual: 154.6212 Desired: 1649.0139 atol: 1 rtol: 1e-05
warning: AssertionError: J1 UTsec 18120 26.28% failed accuracy. maximum error magnitude 6.5516e-06 Actual: 9.7726e-06 Desired: 3.221e-06 atol: 1e-09 rtol: 1e-05
warning: AssertionError: J2 UTsec 18120 6.09% failed accuracy. maximum error magnitude 5.061e-08 Actual: -8.0994e-06 Desired: -8.15e-06 atol: 1e-09 rtol: 1e-05
warning: AssertionError: J3 UTsec 18120 1.84% failed accuracy. maximum error magnitude 8.2165e-08 Actual: 1.1324e-05 Desired: 1.1406e-05 atol: 1e-09 rtol: 1e-05
sim grid dimensions: 64 20 20
sim grid dimensions: 64 20 20
warning: AssertionError: Ne UTsec 18180 38.31% failed accuracy. maximum error magnitude 100478510186.6472 Actual: 109483558912.2997 Desired: 9005048725.6525 atol: 1000000 rtol: 1e-05
warning: AssertionError: Te UTsec 18180 28.46% failed accuracy. maximum error magnitude 1549.8259 Actual: 158.2692 Desired: 1708.0951 atol: 1 rtol: 1e-05
warning: AssertionError: J1 UTsec 18180 33.06% failed accuracy. maximum error magnitude 7.4646e-06 Actual: 1.0851e-05 Desired: 3.386e-06 atol: 1e-09 rtol: 1e-05
warning: AssertionError: J2 UTsec 18180 7.93% failed accuracy. maximum error magnitude 6.0759e-08 Actual: -8.0978e-06 Desired: -8.1586e-06 atol: 1e-09 rtol: 1e-05
warning: AssertionError: J3 UTsec 18180 6.08% failed accuracy. maximum error magnitude 8.9486e-08 Actual: 1.1423e-05 Desired: 1.1513e-05 atol: 1e-09 rtol: 1e-05
sim grid dimensions: 64 20 20
sim grid dimensions: 64 20 20
warning: AssertionError: Ne UTsec 18240 40.70% failed accuracy. maximum error magnitude 92338807983.58 Actual: 107300904923.245 Desired: 14962096939.665 atol: 1000000 rtol: 1e-05
warning: AssertionError: Te UTsec 18240 29.33% failed accuracy. maximum error magnitude 1570.3262 Actual: 156.1014 Desired: 1726.4276 atol: 1 rtol: 1e-05
warning: AssertionError: J1 UTsec 18240 36.20% failed accuracy. maximum error magnitude 7.9495e-06 Actual: 1.1354e-05 Desired: 3.4046e-06 atol: 1e-09 rtol: 1e-05
warning: AssertionError: J2 UTsec 18240 9.06% failed accuracy. maximum error magnitude 6.6393e-08 Actual: -8.0937e-06 Desired: -8.1601e-06 atol: 1e-09 rtol: 1e-05
warning: AssertionError: J3 UTsec 18240 7.97% failed accuracy. maximum error magnitude 9.3068e-08 Actual: 1.1443e-05 Desired: 1.1536e-05 atol: 1e-09 rtol: 1e-05
sim grid dimensions: 64 20 20
sim grid dimensions: 64 20 20
warning: AssertionError: Ne UTsec 18300 42.03% failed accuracy. maximum error magnitude 90035261455.9563 Actual: 137922340047.0673 Desired: 47887078591.111 atol: 1000000 rtol: 1e-05
warning: AssertionError: Te UTsec 18300 29.71% failed accuracy. maximum error magnitude 1579.969 Actual: 172.6757 Desired: 1752.6448 atol: 1 rtol: 1e-05
warning: AssertionError: J1 UTsec 18300 37.48% failed accuracy. maximum error magnitude 8.2234e-06 Actual: 1.1623e-05 Desired: 3.3991e-06 atol: 1e-09 rtol: 1e-05
warning: AssertionError: J2 UTsec 18300 9.81% failed accuracy. maximum error magnitude 7.0397e-08 Actual: -8.0897e-06 Desired: -8.1601e-06 atol: 1e-09 rtol: 1e-05
warning: AssertionError: J3 UTsec 18300 9.07% failed accuracy. maximum error magnitude 9.4394e-08 Actual: 1.1448e-05 Desired: 1.1542e-05 atol: 1e-09 rtol: 1e-05
25 compare errors
50% tests passed, 1 tests failed out of 2
Total Test time (real) = 230.17 sec
The following tests FAILED:
4 - Compare3D (Failed)
Errors while running CTest
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment