Gemini failures Intel compiler

2d_single.log

$ ./gemini initialize/test2d/config.ini /tmp/2dt
 Process:             0  of:             0  online...
initialize/test2d/config.ini: simulation ymd is:    2013/02/20
                                   start time is:   18000.000
                                     duration is:     300.000
                                    output every:      60.000
 ...using input data files:  
   ../simulations/zenodo2d/inputs/simsize.dat
   ../simulations/zenodo2d/inputs/simgrid.dat
   ../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
Precipitation file input cadence (s):       5.000
 Precipitation file input source directory:  ../simulations/zenodo2d/inputs/prec
 _inputs/
 Electric field file input cadence (s):     1.00000000000000     
 Electric field file input source directory:  ../simulations/zenodo2d/inputs/Efi
 eld_inputs/
 Grid has size:            64          40           1
 Detected a 2D run request...  swapping x2 an x3 sizes to maintain parallelizati
 on.
 Starting grid input from file: ../simulations/zenodo2d/inputs/simgrid.dat
 Detected a 2D grid, so will permute the dimensions of the input
 Done reading in grid data...
 Done sending common variables to workers...
 Done sending slabbed variables to workers...
 Done computing null grid points...  Process:             0  has:             0
 Assembling initial condition on root using ../simulations/zenodo2d/inputs/simsi
 ze.dat ../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
 Input file has size:            64          40           1
 Target grid structure has size          64           1          40
 2D simulations grid detected, swapping input file dimension sizes and permuting
  input arrays
 Done gathering input...
 Done setting initial conditions...
 Min/max input density:    0.000000000000000E+000   311058103128.718     
 Min/max input velocity:    -175.638347854180        66.6734644678687     
 Min/max input temperature:    0.000000000000000E+000   1500.03048952670     
 Done sending ICs to workers...  CPU elapsed time:    1.071000000000003E-003
 dt figured to be:    1.000000000000000E-006
 x1,x2,x3 courant numbers (root process only!):  
    0.22, 0.00, 0.00
    0.16, 0.00, 0.00
    0.16, 0.00, 0.00
    0.16, 0.00, 0.00
    0.23, 0.00, 0.00
    0.90, 0.00, 0.00
    0.00, 0.00, 0.00
 Min and max density:    0.000000000000000E+000   311058103128.718     
 Neutral background calculated in time:    1.080500000000000E-002
 Conductivities for time step:    0.000000000000000E+000  took 
  1.002000000000003E-003  seconds...
 Inputting electric field data size from file:  
 ../simulations/zenodo2d/inputs/Efield_inputs//simsize.dat
 Electric field data has llon,llat size:           100           1
 Inputting electric field grid from file:  
 ../simulations/zenodo2d/inputs/Efield_inputs//simgrid.dat
 Electric field data has mlon,mlat extent:   256.660788483182     
   258.325003815978        67.2527183877216        67.2527183877216     
 Grid has mlon,mlat range:     256.677104319778        258.308687979382     
   67.2527183877216        67.2527183877216     
 Grid has size:            40
 tprev,tnow,tnext:    0.000000000000000E+000  5.000000000000000E-007
  0.000000000000000E+000
 Pulling electric field data from file:  
 ../simulations/zenodo2d/inputs/Efield_inputs//20130220_18001.000000.dat
 Min/max values for E0xp:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for E0yp:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for Vminx1p:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for Vmaxx1p:    -1266.03790188990        1266.03790188990     
 Min/max values for Vminx2pslice:    0.000000000000000E+000
  0.000000000000000E+000
 Min/max values for Vmaxx2pslice:    0.000000000000000E+000
  0.000000000000000E+000
 Min/max values for Vminx3pslice:    0.000000000000000E+000
  0.000000000000000E+000
 Min/max values for Vmaxx3pslice:    0.000000000000000E+000
  0.000000000000000E+000
 Initiating electric field boundary condition spatial interpolations for date:  
        2013           2          20     18001.0000000000     
 Singleton latitude dimension detected; interpolating in longitude...
 Min/max values for E0xi:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for E0yi:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for Vminx1i:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for Vmaxx1i:    -1266.03752107278        1266.03752107278     
 Solve type:            1
 tprev,t,tnext:    0.000000000000000E+000  5.000000000000000E-007
   1.00000000000000     
 Min/max values for E0xinow:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for E0yinow:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for Vminx1inow:    0.000000000000000E+000
  0.000000000000000E+000
 Min/max values for Vmaxx1inow:   -6.330187605363892E-004
  6.330187605363892E-004
 Root has computed BCs in time:    4.250000000000018E-004
 Root has communicated type of solve to workers:             1
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
gemini             000000000092E8C9  Unknown               Unknown  Unknown
libpthread-2.27.s  00007FCEB580C890  Unknown               Unknown  Unknown
gemini             000000000092E4AC  Unknown               Unknown  Unknown
libpthread-2.27.s  00007FCEB580C890  Unknown               Unknown  Unknown

Stack trace terminated abnormally.

3d_single.log

$ ./gemini initialize/test3d/config.ini /tmp/3dt
 Process:             0  of:             0  online...
initialize/test3d/config.ini: simulation ymd is:    2013/02/20
                                   start time is:   18000.000
                                     duration is:     300.000
                                    output every:      60.000
 ...using input data files:  
   ../simulations/zenodo3d/inputs/simsize.dat
   ../simulations/zenodo3d/inputs/simgrid.dat
   ../simulations/zenodo3d/inputs/3Dtest_ICs.dat
Precipitation file input cadence (s):       5.000
 Precipitation file input source directory:  ../simulations/zenodo3d/inputs/prec
 _inputs/
 Electric field file input cadence (s):     1.00000000000000     
 Electric field file input source directory:  ../simulations/zenodo3d/inputs/Efi
 eld_inputs/
 Grid has size:            64          20          20
 Detected a 3D run requuest...
 Starting grid input from file: ../simulations/zenodo3d/inputs/simgrid.dat
 Done reading in grid data...
 Done sending common variables to workers...
 Done sending slabbed variables to workers...
 Done computing null grid points...  Process:             0  has:             0
mkdir: created directory '/tmp/3dt'
mkdir: created directory '/tmp/3dt/inputs'
mkdir: created directory '/tmp/3dt/inputs/prec_inputs'
mkdir: created directory '/tmp/3dt/inputs/Efield_inputs'
 Assembling initial condition on root using ../simulations/zenodo3d/inputs/simsi
 ze.dat ../simulations/zenodo3d/inputs/3Dtest_ICs.dat
 Input file has size:            64          20          20
 Target grid structure has size          64          20          20
 Done gathering input...
 Done setting initial conditions...
 Min/max input density:    0.000000000000000E+000   311057875896.448     
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
gemini             000000000092E823  Unknown               Unknown  Unknown
libpthread-2.27.s  00007FEC4A3FB890  Unknown               Unknown  Unknown
gemini             0000000000420818  io_mp_input_root_         372  io.F90
gemini             000000000041CC16  io_mp_input_plasm         241  io.F90
gemini             0000000000408B67  MAIN__                    140  gemini.f90
gemini             0000000000405DC2  Unknown               Unknown  Unknown
libc-2.27.so       00007FEC46D98B97  __libc_start_main     Unknown  Unknown
gemini             0000000000405CAA  Unknown               Unknown  Unknown

out_mpi.log

$ ctest --output-on-failure
Test project ~/gemini/objects
    Start 1: Gemini2D
1/6 Test #1: Gemini2D .........................***Failed    0.21 sec
 Process:             0  of:             3  online...
 Process:             1  of:             3  online...
 Process:             2  of:             3  online...
 Process:             3  of:             3  online...
~/gemini/initialize/test2d/config.ini: simulation ymd is:    2013/02/20
                                   start time is:   18000.000
                                     duration is:     300.000
                                    output every:      60.000
 ...using input data files:  
   ../simulations/zenodo2d/inputs/simsize.dat
   ../simulations/zenodo2d/inputs/simgrid.dat
   ../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
Precipitation file input cadence (s):       5.000
 Precipitation file input source directory:  ../simulations/zenodo2d/inputs/prec
 _inputs/
 Electric field file input cadence (s):     1.00000000000000     
 Electric field file input source directory:  ../simulations/zenodo2d/inputs/Efi
 eld_inputs/
 Grid has size:            64          40           1
 Detected a 2D run request...  swapping x2 an x3 sizes to maintain parallelizati
 on.
 Starting grid input from file: ../simulations/zenodo2d/inputs/simgrid.dat
 Detected a 2D grid, so will permute the dimensions of the input
 Done reading in grid data...
 Done sending common variables to workers...
 Done sending slabbed variables to workers...
 Done computing null grid points...  Process:             1  has:             0
 Done computing null grid points...  Process:             2  has:             0
 Done computing null grid points...  Process:             3  has:             0
 Done computing null grid points...  Process:             0  has:             0
mkdir: created directory '~/gemini/objects/test2d'
mkdir: created directory '~/gemini/objects/test2d/inputs'
mkdir: created directory '~/gemini/objects/test2d/inputs/prec_inputs'
mkdir: created directory '~/gemini/objects/test2d/inputs/Efield_inputs'
 Assembling initial condition on root using ../simulations/zenodo2d/inputs/simsi
 ze.dat ../simulations/zenodo2d/inputs/2Dtest_lowres_ICs.dat
 Input file has size:            64          40           1
 Target grid structure has size          64           1          40
 2D simulations grid detected, swapping input file dimension sizes and permuting
  input arrays
 Done gathering input...
 Done setting initial conditions...
 Min/max input density:    0.000000000000000E+000   311058103128.718     
 Min/max input velocity:    -175.638347854180        66.6734644678687     
 Min/max input temperature:    0.000000000000000E+000   1500.03048952670     
 Done sending ICs to workers...  CPU elapsed time:    6.880000000000011E-004
 dt figured to be:    1.000000000000000E-006
 x1,x2,x3 courant numbers (root process only!):  
    0.22, 0.00, 0.00
    0.16, 0.00, 0.00
    0.16, 0.00, 0.00
    0.16, 0.00, 0.00
    0.23, 0.00, 0.00
    0.90, 0.00, 0.00
    0.00, 0.00, 0.00
 Min and max density:    0.000000000000000E+000   311058103128.718     
 Neutral background calculated in time:    2.858999999999993E-003
 Conductivities for time step:    0.000000000000000E+000  took 
  2.710000000000004E-004  seconds...
 Inputting electric field data size from file:  
 ../simulations/zenodo2d/inputs/Efield_inputs//simsize.dat
 Electric field data has llon,llat size:           100           1
 Inputting electric field grid from file:  
 ../simulations/zenodo2d/inputs/Efield_inputs//simgrid.dat
 Electric field data has mlon,mlat extent:   256.660788483182     
   258.325003815978        67.2527183877216        67.2527183877216     
 Grid has mlon,mlat range:     256.677104319778        258.308687979382     
   67.2527183877216        67.2527183877216     
 Grid has size:            40
 tprev,tnow,tnext:    0.000000000000000E+000  5.000000000000000E-007
  0.000000000000000E+000
 Pulling electric field data from file:  
 ../simulations/zenodo2d/inputs/Efield_inputs//20130220_18001.000000.dat
 Min/max values for E0xp:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for E0yp:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for Vminx1p:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for Vmaxx1p:    -1266.03790188990        1266.03790188990     
 Min/max values for Vminx2pslice:    0.000000000000000E+000
  0.000000000000000E+000
 Min/max values for Vmaxx2pslice:    0.000000000000000E+000
  0.000000000000000E+000
 Min/max values for Vminx3pslice:    0.000000000000000E+000
  0.000000000000000E+000
 Min/max values for Vmaxx3pslice:    0.000000000000000E+000
  0.000000000000000E+000
 Initiating electric field boundary condition spatial interpolations for date:  
        2013           2          20     18001.0000000000     
 Singleton latitude dimension detected; interpolating in longitude...
 Min/max values for E0xi:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for E0yi:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for Vminx1i:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for Vmaxx1i:    -1266.03752107278        1266.03752107278     
 Solve type:            1
 tprev,t,tnext:    0.000000000000000E+000  5.000000000000000E-007
   1.00000000000000     
 Min/max values for E0xinow:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for E0yinow:    0.000000000000000E+000  0.000000000000000E+000
 Min/max values for Vminx1inow:    0.000000000000000E+000
  0.000000000000000E+000
 Min/max values for Vmaxx1inow:   -6.330187605363892E-004
  6.330187605363892E-004
 Root has computed BCs in time:    3.570000000000031E-004
 Root has communicated type of solve to workers:             1
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
gemini             000000000092E8C9  Unknown               Unknown  Unknown
libpthread-2.27.s  00007FD75D320890  Unknown               Unknown  Unknown
gemini             000000000092E4AC  Unknown               Unknown  Unknown
libpthread-2.27.s  00007FD75D320890  Unknown               Unknown  Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
gemini             000000000092E8C9  Unknown               Unknown  Unknown
libpthread-2.27.s  00007F69C5BA9890  Unknown               Unknown  Unknown
gemini             000000000092E4AC  Unknown               Unknown  Unknown
libpthread-2.27.s  00007F69C5BA9890  Unknown               Unknown  Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
gemini             000000000092E8C9  Unknown               Unknown  Unknown
libpthread-2.27.s  00007F87021FD890  Unknown               Unknown  Unknown
gemini             000000000092E4AC  Unknown               Unknown  Unknown
libpthread-2.27.s  00007F87021FD890  Unknown               Unknown  Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
gemini             000000000092E8C9  Unknown               Unknown  Unknown
libpthread-2.27.s  00007F3AAF5A3890  Unknown               Unknown  Unknown
gemini             000000000092E4AC  Unknown               Unknown  Unknown
libpthread-2.27.s  00007F3AAF5A3890  Unknown               Unknown  Unknown

    Start 2: Compare2D
2/6 Test #2: Compare2D ........................***Failed    2.84 sec
sim grid dimensions: 64  40   1
~/gemini/objects/test2d/20130220_18000.000001.dat does not exist

    Start 3: Gemini3D
3/6 Test #3: Gemini3D .........................***Failed    0.59 sec
 Process:             1  of:             3  online...
 Process:             2  of:             3  online...
 Process:             3  of:             3  online...
 Process:             0  of:             3  online...
~/gemini/initialize/test3d/config.ini: simulation ymd is:    2013/02/20
                                   start time is:   18000.000
                                     duration is:     300.000
                                    output every:      60.000
 ...using input data files:  
   ../simulations/zenodo3d/inputs/simsize.dat
   ../simulations/zenodo3d/inputs/simgrid.dat
   ../simulations/zenodo3d/inputs/3Dtest_ICs.dat
Precipitation file input cadence (s):       5.000
 Precipitation file input source directory:  ../simulations/zenodo3d/inputs/prec
 _inputs/
 Electric field file input cadence (s):     1.00000000000000     
 Electric field file input source directory:  ../simulations/zenodo3d/inputs/Efi
 eld_inputs/
 Grid has size:            64          20          20
 Detected a 3D run requuest...
 Starting grid input from file: ../simulations/zenodo3d/inputs/simgrid.dat
 Done reading in grid data...
 Done sending common variables to workers...
 Done sending slabbed variables to workers...
 Done computing null grid points...  Process:             1  has:             0
 Done computing null grid points...  Process:             2  has:             0
 Done computing null grid points...  Process:             3  has:             0
 Done computing null grid points...  Process:             0  has:             0
mkdir: created directory '~/gemini/objects/test3d'
mkdir: created directory '~/gemini/objects/test3d/inputs'
mkdir: created directory '~/gemini/objects/test3d/inputs/prec_inputs'
mkdir: created directory '~/gemini/objects/test3d/inputs/Efield_inputs'
 Assembling initial condition on root using ../simulations/zenodo3d/inputs/simsi
 ze.dat ../simulations/zenodo3d/inputs/3Dtest_ICs.dat
 Input file has size:            64          20          20
 Target grid structure has size          64          20          20
 Done gathering input...
 Done setting initial conditions...
 Min/max input density:    0.000000000000000E+000   311057875896.448     
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
gemini             000000000092E823  Unknown               Unknown  Unknown
libpthread-2.27.s  00007FC9DC2FC890  Unknown               Unknown  Unknown
gemini             0000000000420818  io_mp_input_root_         372  io.F90
gemini             000000000041CC16  io_mp_input_plasm         241  io.F90
gemini             0000000000408B67  MAIN__                    140  gemini.f90
gemini             0000000000405DC2  Unknown               Unknown  Unknown
libc-2.27.so       00007FC9D8C99B97  __libc_start_main     Unknown  Unknown
gemini             0000000000405CAA  Unknown               Unknown  Unknown

    Start 4: Compare3D
4/6 Test #4: Compare3D ........................***Failed    2.84 sec
sim grid dimensions: 64  20  20
~/gemini/objects/test3d/20130220_18000.000001.dat does not exist

    Start 5: MPIexist
5/6 Test #5: MPIexist .........................   Passed    0.12 sec
    Start 6: MUMPS
6/6 Test #6: MUMPS ............................   Passed    0.14 sec

33% tests passed, 4 tests failed out of 6

Total Test time (real) =   6.75 sec

The following tests FAILED:
	  1 - Gemini2D (Failed)
	  2 - Compare2D (Failed)
	  3 - Gemini3D (Failed)
	  4 - Compare3D (Failed)
Errors while running CTest