mdcallag

command lines for 1M keys, 4kb values, 1 thread

./db_bench --benchmarks=overwrite --use_existing_db=1 --sync=0 --db=foobar --wal_dir=/txlogs --disable_data_sync=0 --num=1048576 --num_levels=6 --key_size=20 --value_size=4096 --block_size=32768 --cache_size=34359738368 --cache_numshardbits=6 --compression_type=snappy --min_level_to_compress=3 --compression_ratio=0.5 --level_compaction_dynamic_level_bytes=true --hard_rate_limit=3 --rate_limit_delay_max_milliseconds=1000000 --write_buffer_size=134217728 --max_write_buffer_number=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_grandparent_overlap_factor=8 --max_bytes_for_level_multiplier=8 --statistics=1 --stats_per_interval=1 --stats_interval_seconds=60 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=20480 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=12 --level0_stop_writes_trigger=20 --max_background_compactions=16 --

mdcallag

Sample of stack traces during the shutdown:

     1 arena_avail_tree_insert,arena_avail_insert,arena_run_dalloc,arena_dalloc_bin_run,arena_dalloc_bin_locked_impl,jemalloc_je_arena_dalloc_bin_junked_locked,jemalloc_je_tcache_bin_flush_small,jemalloc_je_tcache_dalloc_small,jemalloc_je_arena_dalloc,jemalloc_je_idalloct,jemalloc_je_iqalloc,ifree,free,operator(),~unique_ptr,~BlockContents,~Block,rocksdb::(anonymous,Free,operator(),ApplyToAllCacheEntries<rocksdb::(anonymous,~HandleTable,~LRUCache,rocksdb::(anonymous,_M_release,~__shared_count,~__shared_ptr,~shared_ptr,~BlockBasedTableOptions,~BlockBasedTableFactory,rocksdb::BlockBasedTableFactory::~BlockBasedTableFactory,_M_release,~__shared_count,~__shared_ptr,~shared_ptr,~ColumnFamilyOptions,~Options,rocksdb::ColumnFamilyData::~ColumnFamilyData,rocksdb::ColumnFamilySet::~ColumnFamilySet,operator(),reset,rocksdb::VersionSet::~VersionSet,operator(),reset,rocksdb::DBImpl::~DBImpl,rocksdb::DBImpl::~DBImpl,rocksdb_done_func,ha_finalize_handlerton,plugin_deinitialize,r

mdcallag

From PMP samples, time occurs in DBImpl dtor which is before the call to DisownData that you added.

RocksDB took 43 seconds with ~26G in the block cache

2015-04-29 09:25:45 2363424 [Note] Shutting down plugin 'ROCKSDB'
2015-04-29 09:26:28 2363424 [Note] Shutting down plugin 'INNODB_FILE_STATUS'

From PMP samples:
      1 arena_avail_comp,arena_avail_tree_insert,arena_avail_insert,arena_run_dalloc,arena_dalloc_bin_run,arena_dalloc_bin_locked_impl,jemalloc_je_arena_dalloc_bin_junked_locked,jemalloc_je_tcache_bin_flush_small,jemalloc_je_tcache_dalloc_small,jemalloc_je_arena_dalloc,jemalloc_je_idalloct,jemalloc_je_iqalloc,ifree,free,operator(),~unique_ptr,~BlockContents,~Block,rocksdb::(anonymous,Free,operator(),ApplyToAllCacheEntries<rocksdb::(anonymous,~HandleTable,~LRUCache,rocksdb::(anonymous,_M_release,~__shared_count,~__shared_ptr,~shared_ptr,~BlockBasedTableOptions,~BlockBasedTableFactory,rocksdb::BlockBasedTableFactory::~BlockBasedTableFactory,_M_release,~__shared_count,~__shared_ptr,~shared_ptr,~Column

mdcallag

ops/sec mb/sec  Size-GB L0_MB   Sum_GB  W-Amp   W-MB/s  usec/op p50     p75     p99     p99.9   p99.99  Uptime  Stall-time      Stall%  Test
6281    196.4   0.1     0.4     0.4     1.0     81.9    159.2   29.2    32.5    48      60      114     5       00:00:4.191     80.3    fillseq.v32768

mdcallag

PosixSequentialFile   : SequentialFile, read only
PoxisRandomAccessFile : RandomAccessFile, read only
PosixMmapReadableFile : RandomAccessFile
PosixMmapFile         : WritableFile
PosixWritableFile     : WritableFile
PosixRandomRWFile     : RandomRWFile 

NewSequentialFile   -> PosixSequentialFile
NewRandomAccessFile -> PosixRandomAccessFile
NewWritableFile     -> PosixWritableFile

mdcallag

** Compaction Stats [default] **
Level    Files   Size(MB) Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt)  KeyIn KeyDrop
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0          0   0.0      0.0     0.0      0.0     865.1    865.1       0.0   0.0      0.0    146.7      6037      7070    0.854       6860       0      0
  L2     43/5       4076   3.8   1526.9   786.8    740.1    1524.9    784.8       0.0   1.9    211.1    210.8      7408       530   13.978       5976   1952M  4798K
  L3    261/44     22372   2.5   3577.7   846.1   2731.6    3566.3    834.7       3.7   4.2     92.7     92.4     39508      6295    6.276       2209   1559M    26M
  L4   1555/68    169266   2.8   5690.3   823.5   4866.9    5606.7    739.8       0.9   6.8     25.0     24.7    232688      6356   36.609       1142   2039M   207

mdcallag

** Compaction Stats [default] **
Level    Files   Size(MB) Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt)  KeyIn KeyDrop
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      3/0        340   0.8      0.0     0.0      0.0       0.4      0.4       0.0   0.0      0.0     47.6        10         4    2.386          0       0      0
  L4      7/4        722   0.9      0.4     0.2      0.2       0.7      0.5       0.0   3.2     30.3     56.1        13         1   12.870          1   1661K    70K
  L5     32/0       1768   0.0      0.0     0.0      0.0       0.0      0.0       0.1   0.0      0.0      0.0         0         0    0.000          0       0      0
 Sum     42/4       2830   0.0      0.4     0.2      0.2       1.1      1.0       0.1   2.6     17.4     52.5        22         5    4.483          1   1661K    70K

mdcallag

2015-05-22 20:05:54 4160248 [Note] RocksDB: Finished filtering dropped index 8612
03:26:36 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=209715200

mdcallag

[mysqld]
innodb_buffer_pool_size=110G
innodb_flush_log_at_trx_commit=2
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
innodb_max_dirty_pages_pct=80

default_storage_engine=MyISAM
innodb_deadlock_detect=0
innodb_file_format=Barracuda

mdcallag

nr=$1
e=$2
eo=$3

rm -f o.res
# for maxn in 1 2 4 8 12 16 20 24 28 32 ; do
for maxn in $( seq 1 64 ); do
  maxr=$(( $nr / $maxn ))

  mysql -uroot -ppw -A -h127.0.0.1 -e 'drop database ib'