segfaults on parallel build (gcc6.3.0)

vespa_parallel_ubuntu.log

In file included from /usr/include/c++/6/map:60:0,
   from /usr/arbor/vespa/vespa/eval/src/vespa/eval/eval/tensor_spec.h:8,
   from /usr/arbor/vespa/vespa/eval/src/apps/make_tensor_binary_format_test_spec/make_tensor_binary_format_test_spec.cpp:6:
   /usr/include/c++/6/bits/stl_tree.h: In member function ‘std::pair<std::_Rb_tree_node_base*, std::_Rb_tree_node_base*> std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_M_get_insert_hint_unique_pos(std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::const_iterator, const key_type&)’:
   /usr/include/c++/6/bits/stl_tree.h:1932:16: internal compiler error: Segmentation fault
   iterator __pos = __position._M_const_cast();
   ^~~~~
   Please submit a full bug report,
   with preprocessed source if appropriate.
   See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
   make[2]: *** [eval/src/apps/make_tensor_binary_format_test_spec/CMakeFiles/eval_make_tensor_binary_format_test_spec_app.dir/make_tensor_binary_format_test_spec.cpp.o] Error 1
   make[1]: *** [eval/src/apps/make_tensor_binary_format_test_spec/CMakeFiles/eval_make_tensor_binary_format_test_spec_app.dir/all] Error 2
   make[1]: *** Waiting for unfinished jobs....



   In file included from /usr/include/c++/6/set:60:0,
   from /usr/arbor/vespa/vespa/searchlib/src/vespa/searchlib/attribute/enumstorebase.h:11,
   from /usr/arbor/vespa/vespa/searchlib/src/vespa/searchlib/attribute/loadedenumvalue.h:6,
   from /usr/arbor/vespa/vespa/searchlib/src/vespa/searchlib/attribute/enumattribute.h:6,
   from /usr/arbor/vespa/vespa/searchlib/src/vespa/searchlib/attribute/singleenumattribute.h:5,
   from /usr/arbor/vespa/vespa/searchlib/src/vespa/searchlib/attribute/singlenumericenumattribute.h:5,
   from /usr/arbor/vespa/vespa/searchlib/src/vespa/searchlib/attribute/singlenumericenumattribute.cpp:3:
   /usr/include/c++/6/bits/stl_tree.h: In member function ‘std::pair<std::_Rb_tree_node_base*, std::_Rb_tree_node_base*> std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_M_get_insert_hint_equal_pos(std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::const_iterator, const key_type&)’:
   /usr/include/c++/6/bits/stl_tree.h:2017:16: internal compiler error: Segmentation fault
   iterator __pos = __position._M_const_cast();
   ^~~~~
   Please submit a full bug report,
   with preprocessed source if appropriate.
   See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
   make[2]: *** [searchlib/src/vespa/searchlib/attribute/CMakeFiles/searchlib_attribute.dir/singlenumericenumattribute.cpp.o] Error 1
   make[1]: *** [searchlib/src/vespa/searchlib/attribute/CMakeFiles/searchlib_attribute.dir/all] Error 2
   make[1]: *** Waiting for unfinished jobs....



   In file included from /usr/include/c++/6/map:60:0,
   from /usr/arbor/vespa/vespa/searchcore/src/vespa/searchcore/proton/matching/ranking_constants.h:6,
   from /usr/arbor/vespa/vespa/searchcore/src/vespa/searchcore/proton/server/documentdbconfig.h:9,
   from /usr/arbor/vespa/vespa/searchcore/src/vespa/searchcore/proton/test/documentdb_config_builder.h:5,
   from /usr/arbor/vespa/vespa/searchcore/src/vespa/searchcore/proton/test/documentdb_config_builder.cpp:3:
   /usr/include/c++/6/bits/stl_tree.h: In member function ‘std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::iterator std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_M_insert_node(std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_Base_ptr, std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_Base_ptr, std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_Link_type)’:
    /usr/include/c++/6/bits/stl_tree.h:2098:12: internal compiler error: Segmentation fault
bool __insert_left = (__x != 0 || __p == _M_end()
        ^~~~~~~~~~~~~

Internal compiler error when building parallel, but not if you build sequentially ?
That does not sound very good. Are you low on memory and gcc has bad handling of OOM situations ?
There should not be anything shared between different invocations of gcc.
Are there any issues with ccache and/or possible incorrect concurrency leading to multiple invocations on the same file. But that normally manifests itself with broken object files that will not link.
Are you using any special options when building. What is your cmake / make commands ?