Manticore crash

manticore crash

root@sloth:~# tail -n 100 /var/log/manticore/searchd.log
[Fri Apr 12 04:15:22.966 2019] [7099] rotating index 'kab_main_index': started
[Fri Apr 12 04:15:22.967 2019] [7099] WARNING: index 'kab_main_index': prealloc: failed to open file '/var/manticore/indices/kab_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.967 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.967 2019] [7099] rotating index 'run_main_index': started
[Fri Apr 12 04:15:22.968 2019] [7099] WARNING: index 'run_main_index': prealloc: failed to open file '/var/manticore/indices/run_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.968 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.968 2019] [7099] rotating index 'aln_main_index': started
[Fri Apr 12 04:15:22.969 2019] [7099] WARNING: index 'aln_main_index': prealloc: failed to open file '/var/manticore/indices/aln_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.969 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.969 2019] [7099] rotating index 'akl_main_index': started
[Fri Apr 12 04:15:22.971 2019] [7099] WARNING: index 'akl_main_index': prealloc: failed to open file '/var/manticore/indices/akl_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.971 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.971 2019] [7099] rotating index 'mnc_main_index': started
[Fri Apr 12 04:15:22.972 2019] [7099] WARNING: index 'mnc_main_index': prealloc: failed to open file '/var/manticore/indices/mnc_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.972 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.972 2019] [7099] rotating index 'kas_main_index': started
[Fri Apr 12 04:15:22.973 2019] [7099] WARNING: index 'kas_main_index': prealloc: failed to open file '/var/manticore/indices/kas_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.973 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.973 2019] [7099] rotating index 'otk_main_index': started
[Fri Apr 12 04:15:22.974 2019] [7099] WARNING: index 'otk_main_index': prealloc: failed to open file '/var/manticore/indices/otk_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.974 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.974 2019] [7099] rotating index 'aoz_main_index': started
[Fri Apr 12 04:15:22.975 2019] [7099] WARNING: index 'aoz_main_index': prealloc: failed to open file '/var/manticore/indices/aoz_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.975 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.975 2019] [7099] rotating index 'shy_main_index': started
[Fri Apr 12 04:15:22.976 2019] [7099] WARNING: index 'shy_main_index': prealloc: failed to open file '/var/manticore/indices/shy_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.976 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.976 2019] [7099] rotating index 'fkv_main_index': started
[Fri Apr 12 04:15:22.977 2019] [7099] WARNING: index 'fkv_main_index': prealloc: failed to open file '/var/manticore/indices/fkv_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.977 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.977 2019] [7099] rotating index 'rap_main_index': started
[Fri Apr 12 04:15:22.978 2019] [7099] WARNING: index 'rap_main_index': prealloc: failed to open file '/var/manticore/indices/rap_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.978 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.978 2019] [7099] rotating index 'gcf_main_index': started
[Fri Apr 12 04:15:22.979 2019] [7099] WARNING: index 'gcf_main_index': prealloc: failed to open file '/var/manticore/indices/gcf_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.979 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.979 2019] [7099] rotating index 'gos_main_index': started
[Fri Apr 12 04:15:22.981 2019] [7099] WARNING: index 'gos_main_index': prealloc: failed to open file '/var/manticore/indices/gos_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.981 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.981 2019] [7099] rotating index 'lij_main_index': started
[Fri Apr 12 04:15:22.982 2019] [7099] WARNING: index 'lij_main_index': prealloc: failed to open file '/var/manticore/indices/lij_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.982 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.982 2019] [7099] rotating index 'tig_main_index': started
[Fri Apr 12 04:15:22.983 2019] [7099] WARNING: index 'tig_main_index': prealloc: failed to open file '/var/manticore/indices/tig_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.983 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.983 2019] [7099] rotating index 'thv_main_index': started
[Fri Apr 12 04:15:22.984 2019] [7099] WARNING: index 'thv_main_index': prealloc: failed to open file '/var/manticore/indices/thv_main.new.spa': 'Permission denied'; NOT SERVING
[Fri Apr 12 04:15:22.984 2019] [7099] WARNING: (null)
[Fri Apr 12 04:15:22.984 2019] [7099] rotating index: all indexes done
[Fri Apr 12 04:15:23.387 2019] [7099] rotating index 'swh_main_index': started
[Fri Apr 12 04:15:23.390 2019] [7099] rotating index 'swh_main_index': success
[Fri Apr 12 04:15:23.391 2019] [7099] rotating index 'lat_main_index': started
------- FATAL: CRASH DUMP -------
[Fri Apr 12 04:15:23.354 2019] [ 7061]

--- crashed invalid query ---

--- request dump end ---
Manticore 2.8.2 4e81114d@190402 release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 6.3.0
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=stretch -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmariadbclient.so.18 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ODBC=ON -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 -DWITH_ZLIB=ON -DGALERA_SOVERSION=31 -DSYSCONFDIR=/etc/sphinxsearch
Host OS is Linux runner-0277ea0f-project-3858465-concurrent-0 4.19.23-coreos-r1 #1 SMP Mon Feb 25 23:40:01 -00 2019 x86_64 GNU/Linux
Stack bottom = 0x7f62edf16eff, thread stack size = 0x100000
Trying manual backtrace:
[Fri Apr 12 04:15:23.400 2019] [7061] caught SIGHUP (seamless=1, in_rotate=1, need_rotate=0)
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x561c8df149f0)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x561c8df149f0, stack=0x7f62edf20000, stacksize=0x100000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x9b)[0x561c8da0369b]
/usr/bin/searchd(_ZN16SphCrashLogger_c11HandleCrashEi+0x1c0)[0x561c8d844790]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7f6391ea80e0]
/lib/x86_64-linux-gnu/libc.so.6(+0x843b0)[0x7f6390cbd3b0]
/usr/bin/searchd(_ZN9CWordlist7PrereadERK10CSphStringjbRS0_+0x2cf)[0x561c8d921a6f]
/usr/bin/searchd(_ZN13CSphIndex_VLN8PreallocEb+0x172)[0x561c8d95bb82]
/usr/bin/searchd(_Z16PreallocNewIndexR12ServedDesc_tPK17CSphConfigSectionPKc+0x27)[0x561c8d850417]
/usr/bin/searchd(+0x19016d)[0x561c8d86616d]
/usr/bin/searchd(_Z18RotationThreadFuncPv+0x263)[0x561c8d8667d3]
/usr/bin/searchd(_Z20sphThreadProcWrapperPv+0x25)[0x561c8da0ef45]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f6391e9e4a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f6390d21d0f]
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the documentation
(http://docs.manticoresearch.com/latest/html/reporting_bugs.html)
[Fri Apr 12 04:15:23.443 2019] [28705] watchdog: got USR1, performing dump of child's stack
Will run gdb on /usr/bin/searchd, pid 7061
[Fri Apr 12 04:15:23.701 2019] [7061] caught SIGHUP (seamless=1, in_rotate=1, need_rotate=1)
[Fri Apr 12 04:15:25.450 2019] [7061] last message repeated 2 times
--- 2 active threads ---
thd 0, proto sphinxql, state handshake, command SYSTEM
thd 1, proto sphinxql, state handshake, command SYSTEM
------- CRASH DUMP END -------
[Fri Apr 12 04:15:26.458 2019] [7099] FATAL: pthread_mutex_destroy() failed No such file or directory
[Fri Apr 12 04:15:27.283 2019] [28705] watchdog: main process 7061 exited cleanly (exit code 1), shutting down
[Fri Apr 12 04:15:27.284 2019] [28705] watchdog: got error 10, No child processes