Fastbit deadlocking code path

fastbit-deadlock.txt

** This is from a modified fork of Fastbit 1.3.8, so the line numbers will be off by a little.

Description
-----------

Deadlock is caused by `ibis::fileManager::getFile` (#7 below) locking the fileManager's mutex, and then the constructor 
at `ibis::fileManager::storage::storage` (#4 below) locking it in the same thread, when it realizes it needs to make
some room in the cache first. There seem to be many similar code paths where this sort of deadlock will arise, all when
the cache size is near or over `maxBytes`.

Simple fix
----------

A simple fix for this, though it does add some overhead, is to use a recursive mutex, which allows locking a mutex
that the current thread already owns. The initialization of the fileManager's mutex would look like this:

// Use a recursive mutex, since several sections of code lock when the thread already has it. 
pthread_mutexattr_t attr;
if (pthread_mutexattr_init(&attr) != 0)
  throw ibis::bad_alloc("pthread_mutexattr_t failed in fileManager ctor");
if (pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE) != 0)
  throw ibis::bad_alloc("pthread_mutexattr_settype failed infileManager ctor");
if (pthread_mutex_init(&mutex, &attr) != 0)
  throw ibis::bad_alloc("pthread_mutex_init failed in fileManager ctor");


Relevant stack trace of a thread deadlocking itself
---------------------------------------------------

#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff5bd14b9 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff5bd12e0 in __GI___pthread_mutex_lock (mutex=0x7ffff78973c0 <ibis::fileManager::instance()::theManager+256>) at ../nptl/pthread_mutex_lock.c:79
#3  0x00007ffff7327618 in mutexLock (m=0x7ffff74d48c7 "fileManager::storage::ctor", lk=<optimized out>, this=0x7ffff41649c0) at util.h:702
#4  ibis::fileManager::storage::storage (this=0x7ffff4164b70, n=63876) at fileManager.cpp:1737
#5  0x00007ffff7327ed1 in ibis::fileManager::storage::enlarge (this=this@entry=0x7fffe4006e60, nelm=nelm@entry=63876) at fileManager.cpp:2005
#6  0x00007ffff7329a6f in ibis::fileManager::roFile::doRead (this=this@entry=0x7fffe4006e60, file=0x7fffe4002bf8 "<*** SOME STUFF ***>") at fileManager.cpp:2455
#7  0x00007ffff732ddf7 in ibis::fileManager::getFile (this=0x7ffff78972c0 <ibis::fileManager::instance()::theManager>, name=name@entry=0x7fffe4002bf8 "<*** SOME STUFF ***>", st=st@entry=0x7fffe4003330, pref=pref@entry=ibis::fileManager::MMAP_LARGE_FILES) at fileManager.cpp:914
#8  0x00007ffff61c70a7 in ibis::part::barrel::open (this=this@entry=0x7ffff4166590, t=0x7fffe4003040, t@entry=0x0) at part.cpp:19578
#9  0x00007ffff61c8744 in ibis::part::calculate (this=this@entry=0x7fffe4003040, trm=..., msk=..., res=...) at part.cpp:6074
#10 0x00007ffff6a09125 in ibis::bord::append (this=this@entry=0x7fffe4029240, sc=..., prt=..., mask=...) at bord.cpp:4568
#11 0x00007ffff7340ada in ibis::filter::sift2 (tms=..., plist=std::vector of length 1, capacity 1 = {...}, cond=...) at filter.cpp:1462
#12 0x00007ffff734b5c6 in ibis::filter::sift (tms=..., plist=std::vector of length 1, capacity 1 = {...}, cond=...) at filter.cpp:573
#13 0x00007ffff734b7dc in ibis::table::select (mylist=std::vector of length 1, capacity 1 = {...}, sel=sel@entry=0x438aa8 "<*** SOME STUFF ***>"..., 
    cond=cond@entry=0x7fffe40088b8 "<*** SOME STUFF ***>") at filter.cpp:2686
#14 0x00007ffff71b2dab in ibis::mensa::select (this=0x7fffe4003880, sel=0x438aa8 "<*** SOME STUFF ***>"..., cond=0x7fffe40088b8 "<*** SOME STUFF ***>") at mensa.cpp:612
# (rest is my stuff)