mono crashes during execution of of MaxQuant

mono_crash_maxquant_20181029T091700.log

$ uname -rm
4.18.5-arch1-1-ARCH x86_64

$ mono --version
Mono JIT compiler version 5.21.0 (master/05bc22015c9 Thu Oct 25 13:31:44 CEST 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        Interpreter:   yes
        LLVM:          supported, not enabled.
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

$ maxquant --version
MaxQuantCmd 1.6.0.16

$ maxquant [...]
[...]
Apply FDR (SP) 
couldn't allocate guard page, continue without it
couldn't allocate guard page, continue without it
* Assertion at mini-exceptions.c:2953, condition `tls->signal_stack' not met

Stacktrace:

/proc/self/maps:
41aa8000-41be8000 rwxp 00000000 00:00 0 
41fe7000-41ff7000 rwxp 00000000 00:00 0 
56353d8d8000-56353d903000 r--p 00000000 00:17 391951                     /usr/bin/mono-sgen
56353d903000-56353dc72000 r-xp 0002b000 00:17 391951                     /usr/bin/mono-sgen
56353dc72000-56353dda5000 r--p 0039a000 00:17 391951                     /usr/bin/mono-sgen
56353dda6000-56353ddad000 r--p 004cd000 00:17 391951                     /usr/bin/mono-sgen
56353ddad000-56353ddb2000 rw-p 004d4000 00:17 391951                     /usr/bin/mono-sgen
56353ddb2000-56353de43000 rw-p 00000000 00:00 0 
56353f85f000-563540165000 rw-p 00000000 00:00 0                          [heap]
7f227cd04000-7f227cd20000 rw-p 00000000 00:00 0 
7f227cd28000-7f227cd38000 rw-p 00000000 00:00 0 
7f227cd40000-7f227cd44000 rw-p 00000000 00:00 0 
7f227cd54000-7f227cd5c000 rw-p 00000000 00:00 0 
7f227cd78000-7f227cda0000 rw-p 00000000 00:00 0 
7f227cda8000-7f227cdac000 rw-p 00000000 00:00 0 
7f227cdb4000-7f227cdbc000 rw-p 00000000 00:00 0 
7f227cddc000-7f227cde0000 rw-p 00000000 00:00 0 
7f227cde4000-7f227cdec000 rw-p 00000000 00:00 0 
7f227cdf4000-7f227ce00000 rw-p 00000000 00:00 0 
7f227ce08000-7f227ce10000 rw-p 00000000 00:00 0 
7f227ce18000-7f227ce24000 rw-p 00000000 00:00 0 
7f227ce2c000-7f227ce30000 rw-p 00000000 00:00 0 
7f227ce40000-7f227ce44000 rw-p 00000000 00:00 0 
7f227ce4c000-7f227ce50000 rw-p 00000000 00:00 0 
7f227ce60000-7f227ce64000 rw-p 00000000 00:00 0 
Memory around native instruction pointer (0x7f268d36cd7f):
0x7f268d36cd6f  d2 4c 89 ce bf 02 00 00 00 b8 0e 00 00 00 0f 05  .L..............
0x7f268d36cd7f  48 8b 8c 24 08 01 00 00 64 48 33 0c 25 28 00 00  H..$....dH3.%(..
0x7f268d36cd8f  00 44 89 c0 75 19 48 81 c4 10 01 00 00 5b c3 66  .D..u.H......[.f
0x7f268d36cd9f  90 48 8b 15 99 60 18 00 f7 d8 64 89 02 eb ba 67  .H...`....d....g

Native stacktrace:

	/usr/bin/mono(+0x13addf) [0x56353da12ddf]
	/usr/bin/mono(+0x13b119) [0x56353da13119]
	/usr/bin/mono(+0xcacbc) [0x56353d9a2cbc]
	/usr/lib/libpthread.so.0(+0x123c0) [0x7f268d5253c0]
	/usr/lib/libc.so.6(gsignal+0x10f) [0x7f268d36cd7f]
	/usr/lib/libc.so.6(abort+0x125) [0x7f268d357672]
	/usr/bin/mono(+0x2beb9) [0x56353d903eb9]
	/usr/bin/mono(monoeg_assertion_message+0xda) [0x56353dc66e1a]
	/usr/bin/mono(+0xca860) [0x56353d9a2860]
	/usr/bin/mono(+0x31549) [0x56353d909549]
	/usr/bin/mono(+0x291b60) [0x56353db69b60]
	/usr/lib/libpthread.so.0(+0x7a9d) [0x7f268d51aa9d]
	/usr/lib/libc.so.6(clone+0x43) [0x7f268d430a43]
Pkilling 0x7f26820fb700 from 0x7f2680ef2700
Pkilling 0x7f26822fc700 from 0x7f2680ef2700
Pkilling 0x7f26824fd700 from 0x7f2680ef2700
Pkilling 0x7f268d330780 from 0x7f2680ef2700
Pkilling 0x7f26828ff700 from 0x7f2680ef2700
Pkilling 0x7f268ceab700 from 0x7f2680ef2700
Pkilling 0x7f239cef8700 from 0x7f2680ef2700
Pkilling 0x7f2681efa700 from 0x7f2680ef2700
Entering thread summarizer pause from 0x7f2680ef2700
Finished thread summarizer pause from 0x7f2680ef2700.
jit_tls required for stack walk

Waiting for dumping threads to resume

Didn't have permission to access mono_crash.0.0.json for file dump

Didn't have permission to access mono_crash.0.1.json for file dump

Didn't have permission to access mono_crash.0.2.json for file dump

Didn't have permission to access mono_crash.0.3.json for file dump

Didn't have permission to access mono_crash.0.4.json for file dump

Didn't have permission to access mono_crash.0.5.json for file dump

Didn't have permission to access mono_crash.0.6.json for file dump

Didn't have permission to access mono_crash.0.7.json for file dump

Didn't have permission to access mono_crash.0.8.json for file dump

Didn't have permission to access mono_crash.0.9.json for file dump

Didn't have permission to access mono_crash.0.10.json for file dump

Didn't have permission to access mono_crash.0.11.json for file dump

Didn't have permission to access mono_crash.0.12.json for file dump

Didn't have permission to access mono_crash.0.13.json for file dump

Didn't have permission to access mono_crash.0.14.json for file dump

Didn't have permission to access mono_crash.0.15.json for file dump

Didn't have permission to access mono_crash.0.16.json for file dump

Didn't have permission to access mono_crash.0.17.json for file dump

Didn't have permission to access mono_crash.0.18.json for file dump

Didn't have permission to access mono_crash.0.19.json for file dump

Didn't have permission to access mono_crash.0.20.json for file dump

Didn't have permission to access mono_crash.0.21.json for file dump

Didn't have permission to access mono_crash.0.22.json for file dump

Didn't have permission to access mono_crash.0.23.json for file dump

Didn't have permission to access mono_crash.0.24.json for file dump

Didn't have permission to access mono_crash.0.25.json for file dump

Didn't have permission to access mono_crash.0.26.json for file dump

Didn't have permission to access mono_crash.0.27.json for file dump

Didn't have permission to access mono_crash.0.28.json for file dump

Didn't have permission to access mono_crash.0.29.json for file dump

Didn't have permission to access mono_crash.0.30.json for file dump

Didn't have permission to access mono_crash.0.31.json for file dump

Didn't have permission to access mono_crash.0.32.json for file dump

Didn't have permission to access mono_crash.0.33.json for file dump

Didn't have permission to access mono_crash.0.34.json for file dump

Didn't have permission to access mono_crash.0.35.json for file dump

Didn't have permission to access mono_crash.0.36.json for file dump

Didn't have permission to access mono_crash.0.37.json for file dump

Didn't have permission to access mono_crash.0.38.json for file dump

Didn't have permission to access mono_crash.0.39.json for file dump

Didn't have permission to access mono_crash.0.40.json for file dump

Didn't have permission to access mono_crash.0.41.json for file dump

Didn't have permission to access mono_crash.0.42.json for file dump

Didn't have permission to access mono_crash.0.43.json for file dump

Didn't have permission to access mono_crash.0.44.json for file dump

Didn't have permission to access mono_crash.0.45.json for file dump

Didn't have permission to access mono_crash.0.46.json for file dump

Didn't have permission to access mono_crash.0.47.json for file dump

Didn't have permission to access mono_crash.0.48.json for file dump

Didn't have permission to access mono_crash.0.49.json for file dump

Didn't have permission to access mono_crash.0.50.json for file dump

Didn't have permission to access mono_crash.0.51.json for file dump

Didn't have permission to access mono_crash.0.52.json for file dump

Didn't have permission to access mono_crash.0.53.json for file dump

Didn't have permission to access mono_crash.0.54.json for file dump

Didn't have permission to access mono_crash.0.55.json for file dump

Didn't have permission to access mono_crash.0.56.json for file dump

Didn't have permission to access mono_crash.0.57.json for file dump

Didn't have permission to access mono_crash.0.58.json for file dump

Didn't have permission to access mono_crash.0.59.json for file dump

Didn't have permission to access mono_crash.0.60.json for file dump

Didn't have permission to access mono_crash.0.61.json for file dump

Didn't have permission to access mono_crash.0.62.json for file dump

Didn't have permission to access mono_crash.0.63.json for file dump

Didn't have permission to access mono_crash.0.64.json for file dump

Didn't have permission to access mono_crash.0.65.json for file dump

Didn't have permission to access mono_crash.0.66.json for file dump

Didn't have permission to access mono_crash.0.67.json for file dump

Didn't have permission to access mono_crash.0.68.json for file dump

Didn't have permission to access mono_crash.0.69.json for file dump

Didn't have permission to access mono_crash.0.70.json for file dump

Didn't have permission to access mono_crash.0.71.json for file dump

Didn't have permission to access mono_crash.0.72.json for file dump

Didn't have permission to access mono_crash.0.73.json for file dump

Didn't have permission to access mono_crash.0.74.json for file dump

Didn't have permission to access mono_crash.0.75.json for file dump

Didn't have permission to access mono_crash.0.76.json for file dump

Didn't have permission to access mono_crash.0.77.json for file dump

Didn't have permission to access mono_crash.0.78.json for file dump

Didn't have permission to access mono_crash.0.79.json for file dump

Didn't have permission to access mono_crash.0.80.json for file dump

Didn't have permission to access mono_crash.0.81.json for file dump

Didn't have permission to access mono_crash.0.82.json for file dump

Didn't have permission to access mono_crash.0.83.json for file dump

Didn't have permission to access mono_crash.0.84.json for file dump

Didn't have permission to access mono_crash.0.85.json for file dump

Didn't have permission to access mono_crash.0.86.json for file dump

Didn't have permission to access mono_crash.0.87.json for file dump

Didn't have permission to access mono_crash.0.88.json for file dump

Didn't have permission to access mono_crash.0.89.json for file dump

Didn't have permission to access mono_crash.0.90.json for file dump

Didn't have permission to access mono_crash.0.91.json for file dump

Didn't have permission to access mono_crash.0.92.json for file dump

Didn't have permission to access mono_crash.0.93.json for file dump

Didn't have permission to access mono_crash.0.94.json for file dump

Didn't have permission to access mono_crash.0.95.json for file dump

Didn't have permission to access mono_crash.0.96.json for file dump

Didn't have permission to access mono_crash.0.97.json for file dump

Didn't have permission to access mono_crash.0.98.json for file dump

Didn't have permission to access mono_crash.0.99.json for file dump


Debug info from gdb:

mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================


Unhandled Exception:
System.Exception: Exception during execution of external process: 49404
  at BaseLibS.Util.WorkDispatcher.ProcessSingleRunExternal (System.Int32 taskIndex, System.Int32 threadIndex) [0x000f2] in <889c264bcadd48e1bc1fdbf91f7ed89e>:0 
  at BaseLibS.Util.WorkDispatcher.DoWork (System.Int32 taskIndex, System.Int32 threadIndex) [0x0000c] in <889c264bcadd48e1bc1fdbf91f7ed89e>:0 
  at BaseLibS.Util.WorkDispatcher.Work (System.Object threadIndex) [0x00054] in <889c264bcadd48e1bc1fdbf91f7ed89e>:0 
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00025] in <1a28d678f56741a1893be934d35463ed>:0 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <1a28d678f56741a1893be934d35463ed>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <1a28d678f56741a1893be934d35463ed>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x0002b] in <1a28d678f56741a1893be934d35463ed>:0 
  at System.Threading.ThreadHelper.ThreadStart (System.Object obj) [0x0000f] in <1a28d678f56741a1893be934d35463ed>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.Exception: Exception during execution of external process: 49404
  at BaseLibS.Util.WorkDispatcher.ProcessSingleRunExternal (System.Int32 taskIndex, System.Int32 threadIndex) [0x000f2] in <889c264bcadd48e1bc1fdbf91f7ed89e>:0 
  at BaseLibS.Util.WorkDispatcher.DoWork (System.Int32 taskIndex, System.Int32 threadIndex) [0x0000c] in <889c264bcadd48e1bc1fdbf91f7ed89e>:0 
  at BaseLibS.Util.WorkDispatcher.Work (System.Object threadIndex) [0x00054] in <889c264bcadd48e1bc1fdbf91f7ed89e>:0 
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00025] in <1a28d678f56741a1893be934d35463ed>:0 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <1a28d678f56741a1893be934d35463ed>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <1a28d678f56741a1893be934d35463ed>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x0002b] in <1a28d678f56741a1893be934d35463ed>:0 
  at System.Threading.ThreadHelper.ThreadStart (System.Object obj) [0x0000f] in <1a28d678f56741a1893be934d35463ed>:0  

mono_crash_maxquant_20181030T135000.log

$ uname -rm
4.18.16-arch1-1-ARCH x86_64

$ mono --version
Mono JIT compiler version 5.16.0 (makepkg/6e48ad4f7b1 Mon Oct 15 22:56:48 CEST 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        Interpreter:   yes
        LLVM:          supported, not enabled.
        GC:            sgen (concurrent by default)

$ maxquant --version
MaxQuantCmd 1.6.3.3

$ maxquant -p 20 -e 38 mqpar.xml
Applying FDR
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_obj (intptr,intptr) [0x00000] in <c6745222adcf4e3f90bbf18e4a1af3ac>:0
  at (wrapper alloc) object.AllocSmall (intptr,intptr) <0x000eb>
  at MsLib.Search.AndromedaPeptide..ctor (System.IO.BinaryReader) [0x00055] in <55b3e077616e46b583233307c426d434>:0
  at MaxQuantPLibS.Search.Identifications.Read () [0x000c7] in <bf85706eecb348938b85bf6849ffe5db>:0
  at MaxQuantPLibS.Search.Identifications.get_Count () [0x0000f] in <bf85706eecb348938b85bf6849ffe5db>:0
  at MaxQuantPLibS.Search.AndromedaEnhancement.Correct1 (MsLib.Util.QueryType,System.Collections.Generic.ICollection`1<bool>,System.Collections.Generic.ICollection`1<double>,System.Collections.Generic.ICollection`1<System.Tuple`5<MsLib.Search.ModificationKey, int16, int16, byte, bool>>,MaxQuantPLibS.Basic.GroupParams,bool,string,bool,bool) [0x00016] in <bf85706eecb348938b85bf6849ffe5db>:0
  at MaxQuantPLibS.Search.AndromedaEnhancement/<>c__DisplayClass34_0.<CorrectNmodsNcleave>b__0 (int) [0x00057] in <bf85706eecb348938b85bf6849ffe5db>:0
  at BaseLibS.Util.ThreadDistributor/<>c__DisplayClass15_0.<.ctor>b__0 (int,int) [0x00000] in <b8c34d2ac5944dd7a80ca5910497b476>:0
  at BaseLibS.Util.ThreadDistributor.Work (object) [0x00067] in <b8c34d2ac5944dd7a80ca5910497b476>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00025] in <c6745222adcf4e3f90bbf18e4a1af3ac>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c6745222adcf4e3f90bbf18e4a1af3ac>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c6745222adcf4e3f90bbf18e4a1af3ac>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <c6745222adcf4e3f90bbf18e4a1af3ac>:0
  at System.Threading.ThreadHelper.ThreadStart (object) [0x0000f] in <c6745222adcf4e3f90bbf18e4a1af3ac>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0004f] in <c6745222adcf4e3f90bbf18e4a1af3ac>:0
/proc/self/maps:
40114000-40124000 rwxp 00000000 00:00 0
41667000-416d7000 rwxp 00000000 00:00 0
55c423350000-55c42337a000 r--p 00000000 00:16 503691                     /usr/bin/mono-sgen
55c42337a000-55c42366b000 r-xp 0002a000 00:16 503691                     /usr/bin/mono-sgen
55c42366b000-55c423790000 r--p 0031b000 00:16 503691                     /usr/bin/mono-sgen
55c423791000-55c423798000 r--p 00440000 00:16 503691                     /usr/bin/mono-sgen
55c423798000-55c42379d000 rw-p 00447000 00:16 503691                     /usr/bin/mono-sgen
55c42379d000-55c4237b3000 rw-p 00000000 00:00 0
55c424287000-55c424904000 rw-p 00000000 00:00 0                          [heap]
7f8ef8640000-7f8ef86c0000 rw-p 00000000 00:00 0
7f8ef86c4000-7f8ef8744000 rw-p 00000000 00:00 0
7f8ef8748000-7f8ef87c8000 rw-p 00000000 00:00 0
7f8ef87cc000-7f8ef884c000 rw-p 00000000 00:00 0
7f8ef8850000-7f8ef88d0000 rw-p 00000000 00:00 0
7f8ef88d4000-7f8ef8954000 rw-p 00000000 00:00 0
7f8ef8958000-7f8ef89d8000 rw-p 00000000 00:00 0
7f8ef89dc000-7f8ef8a5c000 rw-p 00000000 00:00 0
7f8ef8a60000-7f8ef8ae0000 rw-p 00000000 00:00 0
7f8ef8ae4000-7f8ef8b64000 rw-p 00000000 00:00 0
7f8ef8b68000-7f8ef8be8000 rw-p 00000000 00:00 0
7f8ef8bec000-7f8ef8c6c000 rw-p 00000000 00:00 0
7f8ef8c70000-7f8ef8cf0000 rw-p 00000000 00:00 0
7f8ef8cf4000-7f8ef8d74000 rw-p 00000000 00:00 0
7f8ef8d78000-7f8ef8df8000 rw-p 00000000 00:00 0
7f8ef8dfc000-7f8ef8e7c000 rw-p 00000000 00:00 0

mono_crash_maxquant_20181030T15310.log

$ uname -rm
4.18.16-arch1-1-ARCH x86_64

$ mono --version
Mono JIT compiler version 5.21.0 (master/adc72216b38 Tue Oct 30 13:25:21 CET 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        Interpreter:   yes
        LLVM:          supported, not enabled.
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

$ maxquant --version
MaxQuantCmd 1.6.3.3

$ maxquant -p 20 -e 38 mqpar.xml
Applying FDR
Error: Garbage collector could not allocate 16384 bytes of memory for major heap section.

Unhandled Exception:
System.Exception: Exception during execution of external process: 95183
  at BaseLibS.Util.WorkDispatcher.ProcessSingleRunExternalProcess (System.Int32 taskIndex, System.Int32 threadIndex) [0x0013b] in <b8c34d2ac5944dd7a80ca5910497b476>:0
  at BaseLibS.Util.WorkDispatcher.DoWork (System.Int32 taskIndex, System.Int32 threadIndex) [0x0001c] in <b8c34d2ac5944dd7a80ca5910497b476>:0
  at BaseLibS.Util.WorkDispatcher.Work (System.Object threadIndex) [0x00054] in <b8c34d2ac5944dd7a80ca5910497b476>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00025] in <004736cb09e745a188841831b0bf9286>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <004736cb09e745a188841831b0bf9286>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <004736cb09e745a188841831b0bf9286>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x0002b] in <004736cb09e745a188841831b0bf9286>:0
  at System.Threading.ThreadHelper.ThreadStart (System.Object obj) [0x0000f] in <004736cb09e745a188841831b0bf9286>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.Exception: Exception during execution of external process: 95183
  at BaseLibS.Util.WorkDispatcher.ProcessSingleRunExternalProcess (System.Int32 taskIndex, System.Int32 threadIndex) [0x0013b] in <b8c34d2ac5944dd7a80ca5910497b476>:0
  at BaseLibS.Util.WorkDispatcher.DoWork (System.Int32 taskIndex, System.Int32 threadIndex) [0x0001c] in <b8c34d2ac5944dd7a80ca5910497b476>:0
  at BaseLibS.Util.WorkDispatcher.Work (System.Object threadIndex) [0x00054] in <b8c34d2ac5944dd7a80ca5910497b476>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00025] in <004736cb09e745a188841831b0bf9286>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <004736cb09e745a188841831b0bf9286>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <004736cb09e745a188841831b0bf9286>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x0002b] in <004736cb09e745a188841831b0bf9286>:0
  at System.Threading.ThreadHelper.ThreadStart (System.Object obj) [0x0000f] in <004736cb09e745a188841831b0bf9286>:0

@pillepalle123
both, sudo and user mono installations did not work for me unless the server is restarted or some other magic things happened.

@pillepalle123 @tobigithub Are you sure you are not running into the old issue where the number of memory mappings vm.max_map_count is too low?
If you are not sure, search the MaxQuant Google user group for the topic. When running many samples, the MaxQuant process hits the limit at some time, especially during the single-core tasks like FDR calculation. On a side note: You might also need to raise the allowed number of open file descriptors, that is what kicked in next on our server.

Raising the memory mapping limit solves the problem, for example like this:
sudo sysctl -w vm.max_map_count=655350

There is a related mono issue:
mono/mono#10415

@elrubio
Oh amazing, this looks very promising! Missed that post.
I ll try this out!!!
Thx!

I have the same problem. I am searching 54 RAW files on our HPC with MaxQuant 1.6.6.0. I have submitted the same report to https://maxquant.myjetbrains.com/youtrack/issue/MaxQuant-315 since it appears to affect a number of people (see two other linked reports below).

This is a CentOS system. Mono has just been upgraded:
mono-data-oracle-5.20.1.19-0.xamarin.3.epel7.x86_64 monodoc-core-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-llvm-tools-6.0+mono20181128173919-0.xamarin.4.epel7.x86_64 dejavu-sans-mono-fonts-2.33-6.el7.noarch mono-extras-5.20.1.19-0.xamarin.3.epel7.x86_64 libmonosgen-2_0-devel-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-winfxcore-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-data-sqlite-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-locale-extras-5.20.1.19-0.xamarin.3.epel7.x86_64 libmonosgen-2_0-1-5.20.1.19-0.xamarin.3.epel7.x86_64 libmono-2_0-devel-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-web-5.20.1.19-0.xamarin.3.epel7.x86_64 libmono-llvm0-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-reactive-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-devel-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-winforms-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-mvc-5.20.1.19-0.xamarin.3.epel7.x86_64 libmonoboehm-2_0-1-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-core-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-data-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-nunit-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-wcf-5.20.1.19-0.xamarin.3.epel7.x86_64 libmonoboehm-2_0-devel-5.20.1.19-0.xamarin.3.epel7.x86_64 liberation-mono-fonts-1.07.2-16.el7.noarch libmono-2_0-1-5.20.1.19-0.xamarin.3.epel7.x86_64 mono-complete-5.20.1.19-0.xamarin.3.epel7.x86_64

MaxQuant got up to Applying FDR (30 CPUs, 360 Gb RAM, 120 threads requested):
Configuring Testing fasta files Testing raw files Feature detection Tmp folder cleanup Calculating peak properties Combining apl files for first search Preparing searches MS/MS first search Read search results for recalibration Mass recalibration Tmp folder cleanup MS/MS preparation for main search Tmp folder cleanup Combining apl files for main search MS/MS main search Preparing combined folder Calculating masses Correcting errors Reading search engine results Preparing reverse hits Finish search engine results Filter identifications (MS/MS) Applying FDR

And then gave this error:

Unhandled Exception: System.Exception: Exception during execution of external process: 752801 at BaseLibS.Util.WorkDispatcher.ProcessSingleRunExternalProcess (System.Int32 taskIndex, System.Int32 threadIndex) [0x0019d] in <a5e2699791c9405ea93be65daa8164cf>:0 at BaseLibS.Util.WorkDispatcher.DoWork (System.Int32 taskIndex, System.Int32 threadIndex) [0x0001c] in <a5e2699791c9405ea93be65daa8164cf>:0 at BaseLibS.Util.WorkDispatcher.Work (System.Object threadIndex) [0x00054] in <a5e2699791c9405ea93be65daa8164cf>:0 at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00025] in <c24f25f8893a4048aab8949db4192cc5>:0 at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <c24f25f8893a4048aab8949db4192cc5>:0 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <c24f25f8893a4048aab8949db4192cc5>:0 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x0002b] in <c24f25f8893a4048aab8949db4192cc5>:0 at System.Threading.ThreadHelper.ThreadStart (System.Object obj) [0x0000f] in <c24f25f8893a4048aab8949db4192cc5>:0 [ERROR] FATAL UNHANDLED EXCEPTION: System.Exception: Exception during execution of external process: 752801 at BaseLibS.Util.WorkDispatcher.ProcessSingleRunExternalProcess (System.Int32 taskIndex, System.Int32 threadIndex) [0x0019d] in <a5e2699791c9405ea93be65daa8164cf>:0 at BaseLibS.Util.WorkDispatcher.DoWork (System.Int32 taskIndex, System.Int32 threadIndex) [0x0001c] in <a5e2699791c9405ea93be65daa8164cf>:0 at BaseLibS.Util.WorkDispatcher.Work (System.Object threadIndex) [0x00054] in <a5e2699791c9405ea93be65daa8164cf>:0 at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00025] in <c24f25f8893a4048aab8949db4192cc5>:0 at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <c24f25f8893a4048aab8949db4192cc5>:0 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <c24f25f8893a4048aab8949db4192cc5>:0 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x0002b] in <c24f25f8893a4048aab8949db4192cc5>:0 at System.Threading.ThreadHelper.ThreadStart (System.Object obj) [0x0000f] in <c24f25f8893a4048aab8949db4192cc5>:0

This is not unique to us (see https://maxquant.myjetbrains.com/youtrack/issue/MaxQuant-293; https://maxquant.myjetbrains.com/youtrack/issue/MaxQuant-304 for similar error at MS/MS First Search step).

I have tried the suggestions given:

• Mono AOT compilation: no difference (though was unable to compile linked library as it resides in /usr and I am on HPC so not root)
• I ran a stack trace and tracked mmap and munmap commands - there were never more than 162 mmaps open, and total mmap space was less than 1 Gb
• With HPC help we increased vm.max_map_count
• I reduced thread count to 1 (unsurprisingly no change since the FDR section is single threaded)

Please let me know if you find a solution here. I will cross post any answers from the exception report I filed on myjetbrains.

Best wishes,

Andrew

@elrubio
thank you again! I think this solved it!
Did you ever manage to run different tasks on different cluster knots in parallel? We have 40CPUs per cluter. When I run 200 files on 200 cores. It will put a load of almost 200 on 1 cluster (40 core, master) and won't use the reamining 4 slaves (160 cores) at all.

Glad it solved it for you. It didn't change it on our HPC. I also ran strace and pulled out mmap and munmap commands and didn't find an excessive number of memory maps at the time of the crash.

I'd be grateful if there are any other ideas floating round!

Thanks,

Andrew

The solution only works when you have sudo rights? What if I don't have sudo rights?

sudo sysctl -w vm.max_map_count=655350

would not work?

That was my problem -- I had to bring this to the attention of our HPC team, who fixed it (but I had to wait some weeks for the fix as machine restarts were required). I found no other solution, sorry.