Skip to content

Instantly share code, notes, and snippets.

@galexrt

galexrt/rook-ceph-mon50

Last active August 20, 2017 16:04
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save galexrt/6b5196fccbc70dea46bad8131ae4ab0c to your computer and use it in GitHub Desktop.
Save galexrt/6b5196fccbc70dea46bad8131ae4ab0c to your computer and use it in GitHub Desktop.
Download ZIP
rook Ceph mon dying
Raw
rook-ceph-mon50
2017-08-20 15:13:12.638284 I | cephmon: parsing mon endpoints: rook-ceph-mon48=10.3.44.89:6790,rook-ceph-mon49=10.3.150.63:6790,rook-ceph-mon51=10.3.157.215:6790,rook-ceph-mon50=10.3.162.13:6790
2017-08-20 15:13:12.638615 I | cephmon: writing config file /var/lib/rook/rook/rook.config
2017-08-20 15:13:12.638724 I | cephmon: generated admin config in /var/lib/rook/rook
2017-08-20 15:13:12.638823 I | cephmon: writing config file /var/lib/rook/rook-ceph-mon49/rook.config
2017-08-20 15:13:12.638905 I | cephmon: initializing mon
2017-08-20 15:13:12.638913 I | exec: Running command: monmaptool /var/lib/rook/rook-ceph-mon49/monmap --create --clobber --fsid 2c50d7fc-8533-45bd-bf42-970c435e4a3d --add rook-ceph-mon50 10.3.162.13:6790 --add rook-ceph-mon48 10.3.44.89:6790 --add rook-ceph-mon49 10.3.150.63:6790 --add rook-ceph-mon51 10.3.157.215:6790
2017-08-20 15:13:12.657429 I | monmaptool: monmap file /var/lib/rook/rook-ceph-mon49/monmap
2017-08-20 15:13:12.657448 I | monmaptool: set fsid to 2c50d7fc-8533-45bd-bf42-970c435e4a3d
2017-08-20 15:13:12.657452 I | monmaptool: writing epoch 0 to /var/lib/rook/rook-ceph-mon49/monmap (4 monitors)
2017-08-20 15:13:12.669106 I | proc: Running process ceph-mon with args: [--mkfs --name=mon.rook-ceph-mon49 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon49/data --conf=/var/lib/rook/rook-ceph-mon49/rook.config --keyring=/var/lib/rook/rook-ceph-mon49/keyring --monmap=/var/lib/rook/rook-ceph-mon49/monmap]
2017-08-20 15:13:12.669118 I | exec: Running command: ceph-mon --mkfs --name=mon.rook-ceph-mon49 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon49/data --conf=/var/lib/rook/rook-ceph-mon49/rook.config --keyring=/var/lib/rook/rook-ceph-mon49/keyring --monmap=/var/lib/rook/rook-ceph-mon49/monmap
2017-08-20 15:13:12.681819 I | mkfs-rook-ceph-mon49: 2017-08-20 15:13:12.681761 7f884a496f40 -1 '/var/lib/rook/rook-ceph-mon49/data' already exists and is not empty: monitor may already exist
2017-08-20 15:13:12.682895 I | mkfs-rook-ceph-mon49: 2017-08-20 15:13:12.681761 7f884a496f40 -1 '/var/lib/rook/rook-ceph-mon49/data' already exists and is not empty: monitor may already exist
2017-08-20 15:13:12.682968 I | cephmon: starting mon
2017-08-20 15:13:12.682997 I | cephmon: Config file /var/lib/rook/rook-ceph-mon49/rook.config:
[global]
enable experimental unrecoverable data corrupting features =
fsid = 2c50d7fc-8533-45bd-bf42-970c435e4a3d
run dir = /var/lib/rook/rook-ceph-mon49
mon initial members = rook-ceph-mon51 rook-ceph-mon50 rook-ceph-mon48 rook-ceph-mon49
mon host = 10.3.157.215:6790,10.3.162.13:6790,10.3.44.89:6790,10.3.150.63:6790
log file = /dev/stdout
mon cluster log file = /dev/stdout
public addr = 10.3.150.63
cluster addr = 100.64.0.58
mon keyvaluedb = rocksdb
debug default = 0
debug rados = 0
debug mon = 0
debug osd = 0
debug bluestore = 0
debug filestore = 0
debug journal = 0
debug leveldb = 0
filestore_omap_backend = rocksdb
osd pg bits = 11
osd pgp bits = 11
osd pool default size = 1
osd pool default min size = 1
osd pool default pg num = 100
osd pool default pgp num = 100
osd objectstore = filestore
rbd_default_features = 3
fatal signal handlers = false
[client.admin]
keyring = /var/lib/rook/rook-ceph-mon49/keyring
2017-08-20 15:13:12.683014 I | proc: Running process ceph-mon with args: [--foreground --name=mon.rook-ceph-mon49 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon49/data --conf=/var/lib/rook/rook-ceph-mon49/rook.config --keyring=/var/lib/rook/rook-ceph-mon49/keyring --public-addr=10.3.150.63:6790 --public-bind-addr=100.64.0.58:6790]
2017-08-20 15:13:12.683021 I | exec: Running command: ceph-mon --foreground --name=mon.rook-ceph-mon49 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon49/data --conf=/var/lib/rook/rook-ceph-mon49/rook.config --keyring=/var/lib/rook/rook-ceph-mon49/keyring --public-addr=10.3.150.63:6790 --public-bind-addr=100.64.0.58:6790
2017-08-20 15:13:12.697199 I | rook-ceph-mon49: 2017-08-20 15:13:12.697139 7f4642d6ef40 0 ceph version ad3a8b682 (ead3a8b682051c4ab5adf80663c66d4521366930) luminous (rc), process (unknown), pid 24
2017-08-20 15:13:12.697231 I | rook-ceph-mon49: 2017-08-20 15:13:12.697193 7f4642d6ef40 0 pidfile_write: ignore empty --pid-file
2017-08-20 15:13:12.701561 I | rook-ceph-mon49: 2017-08-20 15:13:12.701480 7f4642d6ef40 0 load: jerasure load: lrc load: isa
2017-08-20 15:13:12.701591 I | rook-ceph-mon49: 2017-08-20 15:13:12.701568 7f4642d6ef40 0 set rocksdb option compression = kNoCompression
2017-08-20 15:13:12.701596 I | rook-ceph-mon49: 2017-08-20 15:13:12.701574 7f4642d6ef40 0 set rocksdb option write_buffer_size = 33554432
2017-08-20 15:13:12.701600 I | rook-ceph-mon49: 2017-08-20 15:13:12.701585 7f4642d6ef40 0 set rocksdb option compression = kNoCompression
2017-08-20 15:13:12.701617 I | rook-ceph-mon49: 2017-08-20 15:13:12.701587 7f4642d6ef40 0 set rocksdb option write_buffer_size = 33554432
2017-08-20 15:13:12.701735 I | rook-ceph-mon49: 2017-08-20 15:13:12.701701 7f4642d6ef40 4 rocksdb: RocksDB version: 5.4.0
2017-08-20 15:13:12.701743 I | rook-ceph-mon49:
2017-08-20 15:13:12.701762 I | rook-ceph-mon49: 2017-08-20 15:13:12.701706 7f4642d6ef40 4 rocksdb: Git sha rocksdb_build_git_sha:@e15382c09c87a65eaeca9bda233bab503f1e5772@
2017-08-20 15:13:12.701764 I | rook-ceph-mon49: 2017-08-20 15:13:12.701707 7f4642d6ef40 4 rocksdb: Compile date Aug 11 2017
2017-08-20 15:13:12.701766 I | rook-ceph-mon49: 2017-08-20 15:13:12.701707 7f4642d6ef40 4 rocksdb: DB SUMMARY
2017-08-20 15:13:12.701769 I | rook-ceph-mon49:
2017-08-20 15:13:12.701785 I | rook-ceph-mon49: 2017-08-20 15:13:12.701747 7f4642d6ef40 4 rocksdb: CURRENT file: CURRENT
2017-08-20 15:13:12.701787 I | rook-ceph-mon49:
2017-08-20 15:13:12.701789 I | rook-ceph-mon49: 2017-08-20 15:13:12.701762 7f4642d6ef40 4 rocksdb: IDENTITY file: IDENTITY
2017-08-20 15:13:12.701791 I | rook-ceph-mon49:
2017-08-20 15:13:12.701794 I | rook-ceph-mon49: 2017-08-20 15:13:12.701766 7f4642d6ef40 4 rocksdb: MANIFEST file: MANIFEST-000024 size: 230 Bytes
2017-08-20 15:13:12.701802 I | rook-ceph-mon49:
2017-08-20 15:13:12.701805 I | rook-ceph-mon49: 2017-08-20 15:13:12.701768 7f4642d6ef40 4 rocksdb: SST files in /var/lib/rook/rook-ceph-mon49/data/store.db dir, Total Num: 3, files: 000018.sst 000020.sst 000023.sst
2017-08-20 15:13:12.701808 I | rook-ceph-mon49:
2017-08-20 15:13:12.701810 I | rook-ceph-mon49: 2017-08-20 15:13:12.701770 7f4642d6ef40 4 rocksdb: Write Ahead Log file in /var/lib/rook/rook-ceph-mon49/data/store.db: 000025.log size: 100189 ;
2017-08-20 15:13:12.701812 I | rook-ceph-mon49:
2017-08-20 15:13:12.701814 I | rook-ceph-mon49: 2017-08-20 15:13:12.701785 7f4642d6ef40 4 rocksdb: Options.error_if_exists: 0
2017-08-20 15:13:12.701816 I | rook-ceph-mon49: 2017-08-20 15:13:12.701786 7f4642d6ef40 4 rocksdb: Options.create_if_missing: 0
2017-08-20 15:13:12.701818 I | rook-ceph-mon49: 2017-08-20 15:13:12.701787 7f4642d6ef40 4 rocksdb: Options.paranoid_checks: 1
2017-08-20 15:13:12.701821 I | rook-ceph-mon49: 2017-08-20 15:13:12.701788 7f4642d6ef40 4 rocksdb: Options.env: 0xf5699bc560
2017-08-20 15:13:12.701824 I | rook-ceph-mon49: 2017-08-20 15:13:12.701789 7f4642d6ef40 4 rocksdb: Options.info_log: 0xf56bad0e20
2017-08-20 15:13:12.701825 I | rook-ceph-mon49: 2017-08-20 15:13:12.701789 7f4642d6ef40 4 rocksdb: Options.max_open_files: -1
2017-08-20 15:13:12.701840 I | rook-ceph-mon49: 2017-08-20 15:13:12.701791 7f4642d6ef40 4 rocksdb: Options.max_file_opening_threads: 16
2017-08-20 15:13:12.701842 I | rook-ceph-mon49: 2017-08-20 15:13:12.701792 7f4642d6ef40 4 rocksdb: Options.use_fsync: 0
2017-08-20 15:13:12.701844 I | rook-ceph-mon49: 2017-08-20 15:13:12.701793 7f4642d6ef40 4 rocksdb: Options.max_log_file_size: 0
2017-08-20 15:13:12.701845 I | rook-ceph-mon49: 2017-08-20 15:13:12.701794 7f4642d6ef40 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2017-08-20 15:13:12.701847 I | rook-ceph-mon49: 2017-08-20 15:13:12.701794 7f4642d6ef40 4 rocksdb: Options.log_file_time_to_roll: 0
2017-08-20 15:13:12.701849 I | rook-ceph-mon49: 2017-08-20 15:13:12.701795 7f4642d6ef40 4 rocksdb: Options.keep_log_file_num: 1000
2017-08-20 15:13:12.701852 I | rook-ceph-mon49: 2017-08-20 15:13:12.701796 7f4642d6ef40 4 rocksdb: Options.recycle_log_file_num: 0
2017-08-20 15:13:12.701855 I | rook-ceph-mon49: 2017-08-20 15:13:12.701796 7f4642d6ef40 4 rocksdb: Options.allow_fallocate: 1
2017-08-20 15:13:12.701860 I | rook-ceph-mon49: 2017-08-20 15:13:12.701796 7f4642d6ef40 4 rocksdb: Options.allow_mmap_reads: 0
2017-08-20 15:13:12.701867 I | rook-ceph-mon49: 2017-08-20 15:13:12.701797 7f4642d6ef40 4 rocksdb: Options.allow_mmap_writes: 0
2017-08-20 15:13:12.701872 I | rook-ceph-mon49: 2017-08-20 15:13:12.701797 7f4642d6ef40 4 rocksdb: Options.use_direct_reads: 0
2017-08-20 15:13:12.701876 I | rook-ceph-mon49: 2017-08-20 15:13:12.701797 7f4642d6ef40 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
2017-08-20 15:13:12.701881 I | rook-ceph-mon49: 2017-08-20 15:13:12.701798 7f4642d6ef40 4 rocksdb: Options.create_missing_column_families: 0
2017-08-20 15:13:12.701884 I | rook-ceph-mon49: 2017-08-20 15:13:12.701798 7f4642d6ef40 4 rocksdb: Options.db_log_dir:
2017-08-20 15:13:12.701892 I | rook-ceph-mon49: 2017-08-20 15:13:12.701799 7f4642d6ef40 4 rocksdb: Options.wal_dir: /var/lib/rook/rook-ceph-mon49/data/store.db
2017-08-20 15:13:12.701896 I | rook-ceph-mon49: 2017-08-20 15:13:12.701800 7f4642d6ef40 4 rocksdb: Options.table_cache_numshardbits: 6
2017-08-20 15:13:12.701900 I | rook-ceph-mon49: 2017-08-20 15:13:12.701801 7f4642d6ef40 4 rocksdb: Options.max_subcompactions: 1
2017-08-20 15:13:12.701904 I | rook-ceph-mon49: 2017-08-20 15:13:12.701801 7f4642d6ef40 4 rocksdb: Options.max_background_flushes: 1
2017-08-20 15:13:12.701907 I | rook-ceph-mon49: 2017-08-20 15:13:12.701802 7f4642d6ef40 4 rocksdb: Options.WAL_ttl_seconds: 0
2017-08-20 15:13:12.701910 I | rook-ceph-mon49: 2017-08-20 15:13:12.701802 7f4642d6ef40 4 rocksdb: Options.WAL_size_limit_MB: 0
2017-08-20 15:13:12.701913 I | rook-ceph-mon49: 2017-08-20 15:13:12.701809 7f4642d6ef40 4 rocksdb: Options.manifest_preallocation_size: 4194304
2017-08-20 15:13:12.701917 I | rook-ceph-mon49: 2017-08-20 15:13:12.701810 7f4642d6ef40 4 rocksdb: Options.is_fd_close_on_exec: 1
2017-08-20 15:13:12.701923 I | rook-ceph-mon49: 2017-08-20 15:13:12.701811 7f4642d6ef40 4 rocksdb: Options.advise_random_on_open: 1
2017-08-20 15:13:12.701926 I | rook-ceph-mon49: 2017-08-20 15:13:12.701811 7f4642d6ef40 4 rocksdb: Options.db_write_buffer_size: 0
2017-08-20 15:13:12.701930 I | rook-ceph-mon49: 2017-08-20 15:13:12.701812 7f4642d6ef40 4 rocksdb: Options.access_hint_on_compaction_start: 1
2017-08-20 15:13:12.701933 I | rook-ceph-mon49: 2017-08-20 15:13:12.701813 7f4642d6ef40 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2017-08-20 15:13:12.701937 I | rook-ceph-mon49: 2017-08-20 15:13:12.701813 7f4642d6ef40 4 rocksdb: Options.compaction_readahead_size: 0
2017-08-20 15:13:12.701940 I | rook-ceph-mon49: 2017-08-20 15:13:12.701814 7f4642d6ef40 4 rocksdb: Options.random_access_max_buffer_size: 1048576
2017-08-20 15:13:12.701944 I | rook-ceph-mon49: 2017-08-20 15:13:12.701815 7f4642d6ef40 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
2017-08-20 15:13:12.701947 I | rook-ceph-mon49: 2017-08-20 15:13:12.701816 7f4642d6ef40 4 rocksdb: Options.use_adaptive_mutex: 0
2017-08-20 15:13:12.701951 I | rook-ceph-mon49: 2017-08-20 15:13:12.701817 7f4642d6ef40 4 rocksdb: Options.rate_limiter: (nil)
2017-08-20 15:13:12.701954 I | rook-ceph-mon49: 2017-08-20 15:13:12.701817 7f4642d6ef40 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2017-08-20 15:13:12.701958 I | rook-ceph-mon49: 2017-08-20 15:13:12.701818 7f4642d6ef40 4 rocksdb: Options.bytes_per_sync: 0
2017-08-20 15:13:12.701962 I | rook-ceph-mon49: 2017-08-20 15:13:12.701819 7f4642d6ef40 4 rocksdb: Options.wal_bytes_per_sync: 0
2017-08-20 15:13:12.701966 I | rook-ceph-mon49: 2017-08-20 15:13:12.701820 7f4642d6ef40 4 rocksdb: Options.wal_recovery_mode: 2
2017-08-20 15:13:12.701969 I | rook-ceph-mon49: 2017-08-20 15:13:12.701820 7f4642d6ef40 4 rocksdb: Options.enable_thread_tracking: 0
2017-08-20 15:13:12.701973 I | rook-ceph-mon49: 2017-08-20 15:13:12.701821 7f4642d6ef40 4 rocksdb: Options.allow_concurrent_memtable_write: 1
2017-08-20 15:13:12.701976 I | rook-ceph-mon49: 2017-08-20 15:13:12.701822 7f4642d6ef40 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2017-08-20 15:13:12.701980 I | rook-ceph-mon49: 2017-08-20 15:13:12.701822 7f4642d6ef40 4 rocksdb: Options.write_thread_max_yield_usec: 100
2017-08-20 15:13:12.701983 I | rook-ceph-mon49: 2017-08-20 15:13:12.701823 7f4642d6ef40 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2017-08-20 15:13:12.701987 I | rook-ceph-mon49: 2017-08-20 15:13:12.701823 7f4642d6ef40 4 rocksdb: Options.row_cache: None
2017-08-20 15:13:12.701991 I | rook-ceph-mon49: 2017-08-20 15:13:12.701824 7f4642d6ef40 4 rocksdb: Options.wal_filter: None
2017-08-20 15:13:12.701995 I | rook-ceph-mon49: 2017-08-20 15:13:12.701825 7f4642d6ef40 4 rocksdb: Options.avoid_flush_during_recovery: 0
2017-08-20 15:13:12.701999 I | rook-ceph-mon49: 2017-08-20 15:13:12.701825 7f4642d6ef40 4 rocksdb: Options.base_background_compactions: 1
2017-08-20 15:13:12.702003 I | rook-ceph-mon49: 2017-08-20 15:13:12.701826 7f4642d6ef40 4 rocksdb: Options.max_background_compactions: 1
2017-08-20 15:13:12.702007 I | rook-ceph-mon49: 2017-08-20 15:13:12.701840 7f4642d6ef40 4 rocksdb: Options.avoid_flush_during_shutdown: 0
2017-08-20 15:13:12.702012 I | rook-ceph-mon49: 2017-08-20 15:13:12.701841 7f4642d6ef40 4 rocksdb: Options.delayed_write_rate : 16777216
2017-08-20 15:13:12.702015 I | rook-ceph-mon49: 2017-08-20 15:13:12.701841 7f4642d6ef40 4 rocksdb: Options.max_total_wal_size: 0
2017-08-20 15:13:12.702019 I | rook-ceph-mon49: 2017-08-20 15:13:12.701842 7f4642d6ef40 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2017-08-20 15:13:12.702025 I | rook-ceph-mon49: 2017-08-20 15:13:12.701843 7f4642d6ef40 4 rocksdb: Options.stats_dump_period_sec: 600
2017-08-20 15:13:12.702029 I | rook-ceph-mon49: 2017-08-20 15:13:12.701843 7f4642d6ef40 4 rocksdb: Compression algorithms supported:
2017-08-20 15:13:12.702032 I | rook-ceph-mon49: 2017-08-20 15:13:12.701844 7f4642d6ef40 4 rocksdb: Snappy supported: 0
2017-08-20 15:13:12.702036 I | rook-ceph-mon49: 2017-08-20 15:13:12.701845 7f4642d6ef40 4 rocksdb: Zlib supported: 0
2017-08-20 15:13:12.702039 I | rook-ceph-mon49: 2017-08-20 15:13:12.701845 7f4642d6ef40 4 rocksdb: Bzip supported: 0
2017-08-20 15:13:12.702043 I | rook-ceph-mon49: 2017-08-20 15:13:12.701846 7f4642d6ef40 4 rocksdb: LZ4 supported: 0
2017-08-20 15:13:12.702046 I | rook-ceph-mon49: 2017-08-20 15:13:12.701847 7f4642d6ef40 4 rocksdb: ZSTD supported: 0
2017-08-20 15:13:12.702050 I | rook-ceph-mon49: 2017-08-20 15:13:12.701848 7f4642d6ef40 4 rocksdb: Fast CRC32 supported: 1
2017-08-20 15:13:12.713212 I | rook-ceph-mon49: 2017-08-20 15:13:12.713135 7f4642d6ef40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000024
2017-08-20 15:13:12.713225 I | rook-ceph-mon49:
2017-08-20 15:13:12.713354 I | rook-ceph-mon49: 2017-08-20 15:13:12.713301 7f4642d6ef40 4 rocksdb: [/build/ceph/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
2017-08-20 15:13:12.713360 I | rook-ceph-mon49:
2017-08-20 15:13:12.713376 I | rook-ceph-mon49: 2017-08-20 15:13:12.713305 7f4642d6ef40 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2017-08-20 15:13:12.713379 I | rook-ceph-mon49: 2017-08-20 15:13:12.713306 7f4642d6ef40 4 rocksdb: Options.merge_operator:
2017-08-20 15:13:12.713382 I | rook-ceph-mon49: 2017-08-20 15:13:12.713306 7f4642d6ef40 4 rocksdb: Options.compaction_filter: None
2017-08-20 15:13:12.713385 I | rook-ceph-mon49: 2017-08-20 15:13:12.713307 7f4642d6ef40 4 rocksdb: Options.compaction_filter_factory: None
2017-08-20 15:13:12.713389 I | rook-ceph-mon49: 2017-08-20 15:13:12.713307 7f4642d6ef40 4 rocksdb: Options.memtable_factory: SkipListFactory
2017-08-20 15:13:12.713392 I | rook-ceph-mon49: 2017-08-20 15:13:12.713308 7f4642d6ef40 4 rocksdb: Options.table_factory: BlockBasedTable
2017-08-20 15:13:12.713397 I | rook-ceph-mon49: 2017-08-20 15:13:12.713331 7f4642d6ef40 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0xf56ba56138)
2017-08-20 15:13:12.713401 I | rook-ceph-mon49: cache_index_and_filter_blocks: 1
2017-08-20 15:13:12.713405 I | rook-ceph-mon49: cache_index_and_filter_blocks_with_high_priority: 1
2017-08-20 15:13:12.713409 I | rook-ceph-mon49: pin_l0_filter_and_index_blocks_in_cache: 1
2017-08-20 15:13:12.713416 I | rook-ceph-mon49: index_type: 0
2017-08-20 15:13:12.713419 I | rook-ceph-mon49: hash_index_allow_collision: 1
2017-08-20 15:13:12.713422 I | rook-ceph-mon49: checksum: 1
2017-08-20 15:13:12.713426 I | rook-ceph-mon49: no_block_cache: 0
2017-08-20 15:13:12.713430 I | rook-ceph-mon49: block_cache: 0xf56bd571f0
2017-08-20 15:13:12.713433 I | rook-ceph-mon49: block_cache_name: LRUCache
2017-08-20 15:13:12.713436 I | rook-ceph-mon49: block_cache_options:
2017-08-20 15:13:12.713439 I | rook-ceph-mon49: capacity : 134217728
2017-08-20 15:13:12.713441 I | rook-ceph-mon49: num_shard_bits : 4
2017-08-20 15:13:12.713445 I | rook-ceph-mon49: strict_capacity_limit : 0
2017-08-20 15:13:12.713451 I | rook-ceph-mon49: high_pri_pool_ratio: 0.000
2017-08-20 15:13:12.713453 I | rook-ceph-mon49: block_cache_compressed: (nil)
2017-08-20 15:13:12.713457 I | rook-ceph-mon49: persistent_cache: (nil)
2017-08-20 15:13:12.713459 I | rook-ceph-mon49: block_size: 4096
2017-08-20 15:13:12.713462 I | rook-ceph-mon49: block_size_deviation: 10
2017-08-20 15:13:12.713465 I | rook-ceph-mon49: block_restart_interval: 16
2017-08-20 15:13:12.713468 I | rook-ceph-mon49: index_block_restart_interval: 1
2017-08-20 15:13:12.713471 I | rook-ceph-mon49: filter_policy: rocksdb.BuiltinBloomFilter
2017-08-20 15:13:12.713474 I | rook-ceph-mon49: whole_key_filtering: 1
2017-08-20 15:13:12.713476 I | rook-ceph-mon49: format_version: 2
2017-08-20 15:13:12.713480 I | rook-ceph-mon49:
2017-08-20 15:13:12.713489 I | rook-ceph-mon49: 2017-08-20 15:13:12.713354 7f4642d6ef40 4 rocksdb: Options.write_buffer_size: 33554432
2017-08-20 15:13:12.713491 I | rook-ceph-mon49: 2017-08-20 15:13:12.713355 7f4642d6ef40 4 rocksdb: Options.max_write_buffer_number: 2
2017-08-20 15:13:12.713494 I | rook-ceph-mon49: 2017-08-20 15:13:12.713356 7f4642d6ef40 4 rocksdb: Options.compression: NoCompression
2017-08-20 15:13:12.713498 I | rook-ceph-mon49: 2017-08-20 15:13:12.713357 7f4642d6ef40 4 rocksdb: Options.bottommost_compression: Disabled
2017-08-20 15:13:12.713501 I | rook-ceph-mon49: 2017-08-20 15:13:12.713358 7f4642d6ef40 4 rocksdb: Options.prefix_extractor: nullptr
2017-08-20 15:13:12.713505 I | rook-ceph-mon49: 2017-08-20 15:13:12.713359 7f4642d6ef40 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2017-08-20 15:13:12.713542 I | rook-ceph-mon49: 2017-08-20 15:13:12.713359 7f4642d6ef40 4 rocksdb: Options.num_levels: 7
2017-08-20 15:13:12.713545 I | rook-ceph-mon49: 2017-08-20 15:13:12.713360 7f4642d6ef40 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2017-08-20 15:13:12.713548 I | rook-ceph-mon49: 2017-08-20 15:13:12.713374 7f4642d6ef40 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2017-08-20 15:13:12.713551 I | rook-ceph-mon49: 2017-08-20 15:13:12.713375 7f4642d6ef40 4 rocksdb: Options.compression_opts.window_bits: -14
2017-08-20 15:13:12.713554 I | rook-ceph-mon49: 2017-08-20 15:13:12.713376 7f4642d6ef40 4 rocksdb: Options.compression_opts.level: -1
2017-08-20 15:13:12.713558 I | rook-ceph-mon49: 2017-08-20 15:13:12.713376 7f4642d6ef40 4 rocksdb: Options.compression_opts.strategy: 0
2017-08-20 15:13:12.713560 I | rook-ceph-mon49: 2017-08-20 15:13:12.713377 7f4642d6ef40 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2017-08-20 15:13:12.713564 I | rook-ceph-mon49: 2017-08-20 15:13:12.713378 7f4642d6ef40 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2017-08-20 15:13:12.713567 I | rook-ceph-mon49: 2017-08-20 15:13:12.713378 7f4642d6ef40 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2017-08-20 15:13:12.713570 I | rook-ceph-mon49: 2017-08-20 15:13:12.713379 7f4642d6ef40 4 rocksdb: Options.level0_stop_writes_trigger: 36
2017-08-20 15:13:12.713573 I | rook-ceph-mon49: 2017-08-20 15:13:12.713380 7f4642d6ef40 4 rocksdb: Options.target_file_size_base: 67108864
2017-08-20 15:13:12.713578 I | rook-ceph-mon49: 2017-08-20 15:13:12.713380 7f4642d6ef40 4 rocksdb: Options.target_file_size_multiplier: 1
2017-08-20 15:13:12.713581 I | rook-ceph-mon49: 2017-08-20 15:13:12.713381 7f4642d6ef40 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2017-08-20 15:13:12.713584 I | rook-ceph-mon49: 2017-08-20 15:13:12.713381 7f4642d6ef40 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2017-08-20 15:13:12.713588 I | rook-ceph-mon49: 2017-08-20 15:13:12.713382 7f4642d6ef40 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2017-08-20 15:13:12.713591 I | rook-ceph-mon49: 2017-08-20 15:13:12.713384 7f4642d6ef40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2017-08-20 15:13:12.713594 I | rook-ceph-mon49: 2017-08-20 15:13:12.713385 7f4642d6ef40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2017-08-20 15:13:12.713600 I | rook-ceph-mon49: 2017-08-20 15:13:12.713385 7f4642d6ef40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2017-08-20 15:13:12.713603 I | rook-ceph-mon49: 2017-08-20 15:13:12.713386 7f4642d6ef40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2017-08-20 15:13:12.713606 I | rook-ceph-mon49: 2017-08-20 15:13:12.713387 7f4642d6ef40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2017-08-20 15:13:12.713609 I | rook-ceph-mon49: 2017-08-20 15:13:12.713387 7f4642d6ef40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2017-08-20 15:13:12.713612 I | rook-ceph-mon49: 2017-08-20 15:13:12.713388 7f4642d6ef40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2017-08-20 15:13:12.713615 I | rook-ceph-mon49: 2017-08-20 15:13:12.713388 7f4642d6ef40 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2017-08-20 15:13:12.713618 I | rook-ceph-mon49: 2017-08-20 15:13:12.713389 7f4642d6ef40 4 rocksdb: Options.max_compaction_bytes: 1677721600
2017-08-20 15:13:12.713621 I | rook-ceph-mon49: 2017-08-20 15:13:12.713389 7f4642d6ef40 4 rocksdb: Options.arena_block_size: 4194304
2017-08-20 15:13:12.713623 I | rook-ceph-mon49: 2017-08-20 15:13:12.713390 7f4642d6ef40 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2017-08-20 15:13:12.713626 I | rook-ceph-mon49: 2017-08-20 15:13:12.713391 7f4642d6ef40 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2017-08-20 15:13:12.713629 I | rook-ceph-mon49: 2017-08-20 15:13:12.713392 7f4642d6ef40 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2017-08-20 15:13:12.713632 I | rook-ceph-mon49: 2017-08-20 15:13:12.713392 7f4642d6ef40 4 rocksdb: Options.disable_auto_compactions: 0
2017-08-20 15:13:12.713635 I | rook-ceph-mon49: 2017-08-20 15:13:12.713393 7f4642d6ef40 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2017-08-20 15:13:12.713638 I | rook-ceph-mon49: 2017-08-20 15:13:12.713395 7f4642d6ef40 4 rocksdb: Options.compaction_pri: kByCompensatedSize
2017-08-20 15:13:12.713641 I | rook-ceph-mon49: 2017-08-20 15:13:12.713396 7f4642d6ef40 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2017-08-20 15:13:12.713643 I | rook-ceph-mon49: 2017-08-20 15:13:12.713396 7f4642d6ef40 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2017-08-20 15:13:12.713646 I | rook-ceph-mon49: 2017-08-20 15:13:12.713397 7f4642d6ef40 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2017-08-20 15:13:12.713652 I | rook-ceph-mon49: 2017-08-20 15:13:12.713398 7f4642d6ef40 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2017-08-20 15:13:12.713655 I | rook-ceph-mon49: 2017-08-20 15:13:12.713398 7f4642d6ef40 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2017-08-20 15:13:12.713659 I | rook-ceph-mon49: 2017-08-20 15:13:12.713399 7f4642d6ef40 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2017-08-20 15:13:12.713662 I | rook-ceph-mon49: 2017-08-20 15:13:12.713400 7f4642d6ef40 4 rocksdb: Options.table_properties_collectors:
2017-08-20 15:13:12.713665 I | rook-ceph-mon49: 2017-08-20 15:13:12.713401 7f4642d6ef40 4 rocksdb: Options.inplace_update_support: 0
2017-08-20 15:13:12.713669 I | rook-ceph-mon49: 2017-08-20 15:13:12.713402 7f4642d6ef40 4 rocksdb: Options.inplace_update_num_locks: 10000
2017-08-20 15:13:12.713673 I | rook-ceph-mon49: 2017-08-20 15:13:12.713402 7f4642d6ef40 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2017-08-20 15:13:12.713676 I | rook-ceph-mon49: 2017-08-20 15:13:12.713403 7f4642d6ef40 4 rocksdb: Options.memtable_huge_page_size: 0
2017-08-20 15:13:12.713679 I | rook-ceph-mon49: 2017-08-20 15:13:12.713404 7f4642d6ef40 4 rocksdb: Options.bloom_locality: 0
2017-08-20 15:13:12.713682 I | rook-ceph-mon49: 2017-08-20 15:13:12.713405 7f4642d6ef40 4 rocksdb: Options.max_successive_merges: 0
2017-08-20 15:13:12.713685 I | rook-ceph-mon49: 2017-08-20 15:13:12.713405 7f4642d6ef40 4 rocksdb: Options.optimize_filters_for_hits: 0
2017-08-20 15:13:12.713688 I | rook-ceph-mon49: 2017-08-20 15:13:12.713406 7f4642d6ef40 4 rocksdb: Options.paranoid_file_checks: 0
2017-08-20 15:13:12.713691 I | rook-ceph-mon49: 2017-08-20 15:13:12.713407 7f4642d6ef40 4 rocksdb: Options.force_consistency_checks: 0
2017-08-20 15:13:12.713694 I | rook-ceph-mon49: 2017-08-20 15:13:12.713407 7f4642d6ef40 4 rocksdb: Options.report_bg_io_stats: 0
2017-08-20 15:13:12.713934 I | rook-ceph-mon49: 2017-08-20 15:13:12.713916 7f4642d6ef40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/rook/rook-ceph-mon49/data/store.db/MANIFEST-000024 succeeded,manifest_file_number is 24, next_file_number is 26, last_sequence is 2867, log_number is 0,prev_log_number is 0,max_column_family is 0
2017-08-20 15:13:12.713939 I | rook-ceph-mon49:
2017-08-20 15:13:12.713942 I | rook-ceph-mon49: 2017-08-20 15:13:12.713921 7f4642d6ef40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 23
2017-08-20 15:13:12.713946 I | rook-ceph-mon49:
2017-08-20 15:13:12.713974 I | rook-ceph-mon49: 2017-08-20 15:13:12.713964 7f4642d6ef40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503241992713961, "job": 1, "event": "recovery_started", "log_files": [25]}
2017-08-20 15:13:12.713977 I | rook-ceph-mon49: 2017-08-20 15:13:12.713967 7f4642d6ef40 4 rocksdb: [/build/ceph/src/rocksdb/db/db_impl_open.cc:482] Recovering log #25 mode 2
2017-08-20 15:13:12.768704 I | rook-ceph-mon49: 2017-08-20 15:13:12.768588 7f4642d6ef40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503241992768581, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 26, "file_size": 98958, "table_properties": {"data_size": 97314, "index_size": 383, "filter_size": 344, "raw_key_size": 850, "raw_average_key_size": 22, "raw_value_size": 96356, "raw_average_value_size": 2535, "num_data_blocks": 12, "num_entries": 38, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2017-08-20 15:13:12.768724 I | rook-ceph-mon49: 2017-08-20 15:13:12.768645 7f4642d6ef40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2395] Creating manifest 27
2017-08-20 15:13:12.768740 I | rook-ceph-mon49:
2017-08-20 15:13:12.911595 I | rook-ceph-mon49: 2017-08-20 15:13:12.911535 7f4642d6ef40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503241992911532, "job": 1, "event": "recovery_finished"}
2017-08-20 15:13:13.004464 I | rook-ceph-mon49: 2017-08-20 15:13:13.004381 7f4642d6ef40 4 rocksdb: [/build/ceph/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0xf56be5a000
2017-08-20 15:13:13.005189 I | rook-ceph-mon49: 2017-08-20 15:13:13.005170 7f4642d6ef40 0 starting mon.rook-ceph-mon49 rank 1 at public addr 10.3.150.63:6790/0 at bind addr 100.64.0.58:6790/0 mon_data /var/lib/rook/rook-ceph-mon49/data fsid 2c50d7fc-8533-45bd-bf42-970c435e4a3d
2017-08-20 15:13:13.005324 I | rook-ceph-mon49: 2017-08-20 15:13:13.005311 7f4642d6ef40 0 starting mon.rook-ceph-mon49 rank 1 at 10.3.150.63:6790/0 mon_data /var/lib/rook/rook-ceph-mon49/data fsid 2c50d7fc-8533-45bd-bf42-970c435e4a3d
2017-08-20 15:13:13.005809 I | rook-ceph-mon49: 2017-08-20 15:13:13.005795 7f4642d6ef40 1 mon.rook-ceph-mon49@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
2017-08-20 15:13:13.005875 I | rook-ceph-mon49: 2017-08-20 15:13:13.005847 7f4642d6ef40 0 mon.rook-ceph-mon49@-1(probing).mds e27 print_map
2017-08-20 15:13:13.005878 I | rook-ceph-mon49: e27
2017-08-20 15:13:13.005880 I | rook-ceph-mon49: enable_multiple, ever_enabled_multiple: 0,0
2017-08-20 15:13:13.005884 I | rook-ceph-mon49: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
2017-08-20 15:13:13.005886 I | rook-ceph-mon49: legacy client fscid: 1
2017-08-20 15:13:13.005888 I | rook-ceph-mon49:
2017-08-20 15:13:13.005890 I | rook-ceph-mon49: Filesystem 'cfs-r2-1' (1)
2017-08-20 15:13:13.005892 I | rook-ceph-mon49: fs_name cfs-r2-1
2017-08-20 15:13:13.005894 I | rook-ceph-mon49: epoch 27
2017-08-20 15:13:13.005896 I | rook-ceph-mon49: flags c
2017-08-20 15:13:13.005899 I | rook-ceph-mon49: created 2017-08-18 09:35:11.962291
2017-08-20 15:13:13.005901 I | rook-ceph-mon49: modified 2017-08-18 09:35:11.962291
2017-08-20 15:13:13.005903 I | rook-ceph-mon49: tableserver 0
2017-08-20 15:13:13.005904 I | rook-ceph-mon49: root 0
2017-08-20 15:13:13.005907 I | rook-ceph-mon49: session_timeout 60
2017-08-20 15:13:13.005911 I | rook-ceph-mon49: session_autoclose 300
2017-08-20 15:13:13.005913 I | rook-ceph-mon49: max_file_size 1099511627776
2017-08-20 15:13:13.005915 I | rook-ceph-mon49: last_failure 0
2017-08-20 15:13:13.005918 I | rook-ceph-mon49: last_failure_osd_epoch 174
2017-08-20 15:13:13.005923 I | rook-ceph-mon49: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
2017-08-20 15:13:13.005926 I | rook-ceph-mon49: max_mds 1
2017-08-20 15:13:13.005927 I | rook-ceph-mon49: in 0
2017-08-20 15:13:13.005931 I | rook-ceph-mon49: up {0=228660}
2017-08-20 15:13:13.005933 I | rook-ceph-mon49: failed
2017-08-20 15:13:13.005934 I | rook-ceph-mon49: damaged
2017-08-20 15:13:13.005936 I | rook-ceph-mon49: stopped
2017-08-20 15:13:13.005938 I | rook-ceph-mon49: data_pools [3]
2017-08-20 15:13:13.005939 I | rook-ceph-mon49: metadata_pool 4
2017-08-20 15:13:13.005943 I | rook-ceph-mon49: inline_data disabled
2017-08-20 15:13:13.005944 I | rook-ceph-mon49: balancer
2017-08-20 15:13:13.005946 I | rook-ceph-mon49: standby_count_wanted 0
2017-08-20 15:13:13.005948 I | rook-ceph-mon49: 228660: 100.64.0.48:6800/3988930595 'mds1' mds.0.17 up:active seq 18354
2017-08-20 15:13:13.005950 I | rook-ceph-mon49:
2017-08-20 15:13:13.005951 I | rook-ceph-mon49:
2017-08-20 15:13:13.005953 I | rook-ceph-mon49:
2017-08-20 15:13:13.006015 I | rook-ceph-mon49: 2017-08-20 15:13:13.006001 7f4642d6ef40 0 mon.rook-ceph-mon49@-1(probing).osd e175 crush map has features 2200130813952, adjusting msgr requires
2017-08-20 15:13:13.006021 I | rook-ceph-mon49: 2017-08-20 15:13:13.006006 7f4642d6ef40 0 mon.rook-ceph-mon49@-1(probing).osd e175 crush map has features 2200130813952, adjusting msgr requires
2017-08-20 15:13:13.006029 I | rook-ceph-mon49: 2017-08-20 15:13:13.006008 7f4642d6ef40 0 mon.rook-ceph-mon49@-1(probing).osd e175 crush map has features 720578140510109696, adjusting msgr requires
2017-08-20 15:13:13.006033 I | rook-ceph-mon49: 2017-08-20 15:13:13.006008 7f4642d6ef40 0 mon.rook-ceph-mon49@-1(probing).osd e175 crush map has features 2200130813952, adjusting msgr requires
2017-08-20 15:13:13.006159 I | rook-ceph-mon49: 2017-08-20 15:13:13.006135 7f4642d6ef40 1 mon.rook-ceph-mon49@-1(probing).paxosservice(auth 253..314) refresh upgraded, format 0 -> 2
2017-08-20 15:13:13.006761 I | rook-ceph-mon49: 2017-08-20 15:13:13.006733 7f4642d6ef40 -1 mon.rook-ceph-mon49@-1(probing).mgr e7 Failed to load mgr commands: (2) No such file or directory
2017-08-20 15:13:13.007187 I | rook-ceph-mon49: 2017-08-20 15:13:13.007169 7f4642d6ef40 0 mon.rook-ceph-mon49@-1(probing) e99 my rank is now 1 (was -1)
2017-08-20 15:13:13.008206 I | rook-ceph-mon49: 2017-08-20 15:13:13.008185 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:13.008673 I | rook-ceph-mon49: 2017-08-20 15:13:13.008643 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:13.009716 I | rook-ceph-mon49: 2017-08-20 15:13:13.009688 7f4637334700 0 log_channel(cluster) log [INF] : mon.rook-ceph-mon49 calling new monitor election
2017-08-20 15:13:13.120888 I | rook-ceph-mon49: 2017-08-20 15:13:13.120818 7f4637334700 0 log_channel(cluster) log [INF] : mon.rook-ceph-mon49 calling new monitor election
2017-08-20 15:13:13.210368 I | rook-ceph-mon49: 2017-08-20 15:13:13.210257 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:13.210709 I | rook-ceph-mon49: 2017-08-20 15:13:13.210642 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:13.612924 I | rook-ceph-mon49: 2017-08-20 15:13:13.612777 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:13.613373 I | rook-ceph-mon49: 2017-08-20 15:13:13.613333 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:14.415712 I | rook-ceph-mon49: 2017-08-20 15:13:14.415589 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:14.416053 I | rook-ceph-mon49: 2017-08-20 15:13:14.415942 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:16.019152 I | rook-ceph-mon49: 2017-08-20 15:13:16.019035 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:16.019516 I | rook-ceph-mon49: 2017-08-20 15:13:16.019477 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:18.434842 I | rook-ceph-mon49: 2017-08-20 15:13:18.434766 7f4639b39700 0 log_channel(cluster) log [INF] : mon.rook-ceph-mon49@1 won leader election with quorum 1,2
2017-08-20 15:13:18.460158 I | rook-ceph-mon49: 2017-08-20 15:13:18.460059 7f4639b39700 0 log_channel(cluster) log [WRN] : overall HEALTH_WARN 1/3 mons down, quorum rook-ceph-mon49,rook-ceph-mon50
2017-08-20 15:13:18.610566 I | rook-ceph-mon49: 2017-08-20 15:11:13.481139 mon.rook-ceph-mon50 mon.2 10.3.162.13:6790/0 1 : cluster [INF] mon.rook-ceph-mon50 calling new monitor election
2017-08-20 15:13:18.610594 I | rook-ceph-mon49: 2017-08-20 15:11:13.610815 mon.rook-ceph-mon48 mon.0 10.3.44.89:6790/0 1 : cluster [INF] mon.rook-ceph-mon48 calling new monitor election
2017-08-20 15:13:18.610614 I | rook-ceph-mon49: 2017-08-20 15:11:18.563490 mon.rook-ceph-mon48 mon.0 10.3.44.89:6790/0 2 : cluster [INF] mon.rook-ceph-mon48 calling new monitor election
2017-08-20 15:13:18.610618 I | rook-ceph-mon49: 2017-08-20 15:11:23.641762 mon.rook-ceph-mon48 mon.0 10.3.44.89:6790/0 3 : cluster [INF] mon.rook-ceph-mon48@0 won leader election with quorum 0,2
2017-08-20 15:13:18.610637 I | rook-ceph-mon49: 2017-08-20 15:11:23.666958 mon.rook-ceph-mon48 mon.0 10.3.44.89:6790/0 4 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum rook-ceph-mon49,rook-ceph-mon50
2017-08-20 15:13:18.610641 I | rook-ceph-mon49: 2017-08-20 15:11:23.723400 mon.rook-ceph-mon48 mon.0 10.3.44.89:6790/0 5 : cluster [INF] monmap e99: 3 mons at {rook-ceph-mon48=10.3.44.89:6790/0,rook-ceph-mon49=10.3.150.63:6790/0,rook-ceph-mon50=10.3.162.13:6790/0}
2017-08-20 15:13:18.610751 I | rook-ceph-mon49: 2017-08-20 15:13:18.610673 7f4637334700 0 log_channel(cluster) log [INF] : monmap e99: 3 mons at {rook-ceph-mon48=10.3.44.89:6790/0,rook-ceph-mon49=10.3.150.63:6790/0,rook-ceph-mon50=10.3.162.13:6790/0}
2017-08-20 15:13:18.610786 I | rook-ceph-mon49: 2017-08-20 15:13:18.610746 7f4637334700 0 log_channel(cluster) log [DBG] : fsmap cfs-r2-1-1/1/1 up {0=mds1=up:active}
2017-08-20 15:13:18.610831 I | rook-ceph-mon49: 2017-08-20 15:13:18.610818 7f4637334700 0 log_channel(cluster) log [DBG] : osdmap e175: 16 total, 16 up, 16 in
2017-08-20 15:13:18.610880 I | rook-ceph-mon49: 2017-08-20 15:13:18.610864 7f4637334700 0 log_channel(cluster) log [DBG] : mgrmap e7: rook-ceph-mgr0(active)
2017-08-20 15:13:18.986207 I | rook-ceph-mon49: 2017-08-20 15:13:13.009690 mon.rook-ceph-mon49 mon.1 10.3.150.63:6790/0 1 : cluster [INF] mon.rook-ceph-mon49 calling new monitor election
2017-08-20 15:13:18.986248 I | rook-ceph-mon49: 2017-08-20 15:13:13.120824 mon.rook-ceph-mon49 mon.1 10.3.150.63:6790/0 2 : cluster [INF] mon.rook-ceph-mon49 calling new monitor election
2017-08-20 15:13:18.986252 I | rook-ceph-mon49: 2017-08-20 15:13:18.434780 mon.rook-ceph-mon49 mon.1 10.3.150.63:6790/0 3 : cluster [INF] mon.rook-ceph-mon49@1 won leader election with quorum 1,2
2017-08-20 15:13:18.986273 I | rook-ceph-mon49: 2017-08-20 15:13:18.460062 mon.rook-ceph-mon49 mon.1 10.3.150.63:6790/0 4 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum rook-ceph-mon49,rook-ceph-mon50
2017-08-20 15:13:19.224021 I | rook-ceph-mon49: 2017-08-20 15:13:19.223943 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:19.224395 I | rook-ceph-mon49: 2017-08-20 15:13:19.224371 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:20.145803 I | rook-ceph-mon49: 2017-08-20 15:13:13.009768 mon.rook-ceph-mon50 mon.2 10.3.162.13:6790/0 1 : cluster [INF] mon.rook-ceph-mon50 calling new monitor election
2017-08-20 15:13:20.145831 I | rook-ceph-mon49: 2017-08-20 15:13:18.610676 mon.rook-ceph-mon49 mon.1 10.3.150.63:6790/0 5 : cluster [INF] monmap e99: 3 mons at {rook-ceph-mon48=10.3.44.89:6790/0,rook-ceph-mon49=10.3.150.63:6790/0,rook-ceph-mon50=10.3.162.13:6790/0}
2017-08-20 15:13:24.507399 I | rook-ceph-mon49: 2017-08-20 15:13:24.507317 7f4633b2d700 0 -- 10.3.150.63:6790/0 >> 10.3.162.13:6790/0 conn(0xf56bfa0800 :-1 s=STATE_OPEN pgs=10 cs=1 l=0).fault initiating reconnect
2017-08-20 15:13:25.625521 I | rook-ceph-mon49: 2017-08-20 15:13:25.625465 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:25.625847 I | rook-ceph-mon49: 2017-08-20 15:13:25.625809 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:34.475475 I | rook-ceph-mon49: 2017-08-20 15:13:34.475333 7f4639b39700 1 mon.rook-ceph-mon49@1(leader).paxos(paxos updating c 143072..143663) accept timeout, calling fresh election
2017-08-20 15:13:38.430155 I | rook-ceph-mon49: 2017-08-20 15:13:38.430100 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:38.430498 I | rook-ceph-mon49: 2017-08-20 15:13:38.430479 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:53.431691 I | rook-ceph-mon49: 2017-08-20 15:13:53.431595 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:13:53.432003 I | rook-ceph-mon49: 2017-08-20 15:13:53.431983 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:14:08.434536 I | rook-ceph-mon49: 2017-08-20 15:14:08.434442 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:14:08.434781 I | rook-ceph-mon49: 2017-08-20 15:14:08.434724 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:14:13.007034 I | rook-ceph-mon49: 2017-08-20 15:14:13.006961 7f4639b39700 0 mon.rook-ceph-mon49@1(probing).data_health(2098) update_stats avail 71%!t(MISSING)otal 81965 MB, used 19796 MB, avail 58745 MB
2017-08-20 15:14:23.446283 I | rook-ceph-mon49: 2017-08-20 15:14:23.446179 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:14:23.446613 I | rook-ceph-mon49: 2017-08-20 15:14:23.446560 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:14:38.449633 I | rook-ceph-mon49: 2017-08-20 15:14:38.449512 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:14:38.449913 I | rook-ceph-mon49: 2017-08-20 15:14:38.449876 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:14:51.490582 I | rook-ceph-mon49: 2017-08-20 15:14:51.490519 7f463332c700 0 -- 10.3.150.63:6790/0 >> 10.3.44.89:6790/0 conn(0xf56c09e800 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 15:14:51.492104 I | rook-ceph-mon49: 2017-08-20 15:14:51.492067 7f4637334700 0 log_channel(cluster) log [INF] : mon.rook-ceph-mon49 calling new monitor election
2017-08-20 15:14:53.465639 I | rook-ceph-mon49: 2017-08-20 15:14:53.465527 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:14:53.465925 I | rook-ceph-mon49: 2017-08-20 15:14:53.465887 7f463432e700 0 -- 100.64.0.58:0/24 >> 100.64.6.26:6800/18 conn(0xf56bf9c000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:14:55.522299 I | rook-ceph-mon49: 2017-08-20 15:14:55.522227 7f4633b2d700 0 -- 10.3.150.63:6790/0 >> 10.3.162.13:6790/0 conn(0xf56c09a000 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=2 existing_state=STATE_CONNECTING_RE
2017-08-20 15:14:55.522313 I | rook-ceph-mon49: 2017-08-20 15:14:55.522234 7f4633b2d700 0 -- 10.3.150.63:6790/0 >> 10.3.162.13:6790/0 conn(0xf56c09a000 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept peer reset, then tried to connect to us, replacing
2017-08-20 15:14:57.159788 I | rook-ceph-mon49: 2017-08-20 15:14:57.159694 7f4637334700 0 mon.rook-ceph-mon49@1(peon) e99 handle_command mon_command({"prefix": "osd perf", "format": "json"} v 0) v1
2017-08-20 15:14:57.159818 I | rook-ceph-mon49: 2017-08-20 15:14:57.159753 7f4637334700 0 log_channel(audit) log [DBG] : from='client.242235 100.64.3.35:0/472170703' entity='client.admin' cmd=[{"prefix": "osd perf", "format": "json"}]: dispatch
2017-08-20 15:14:57.367541 I | rook-ceph-mon49: 2017-08-20 15:13:13.006733 7f4642d6ef40 -1 mon.rook-ceph-mon49@-1(probing).mgr e7 Failed to load mgr commands: (2) No such file or directory
failed to run mon. failed to start mon: failed to run ceph-mon: Failed to complete rook-ceph-mon49: signal: aborted (core dumped)
Raw
rook-ceph-mon52-tmslh
2017-08-20 15:24:27.214535 I | cephmon: parsing mon endpoints: rook-ceph-mon52=10.3.179.50:6790,rook-ceph-mon53=10.3.14.3:6790,rook-ceph-mon51=10.3.157.215:6790,rook-ceph-mon54=10.3.202.111:6790
2017-08-20 15:24:27.214916 I | cephmon: writing config file /var/lib/rook/rook/rook.config
2017-08-20 15:24:27.215053 I | cephmon: generated admin config in /var/lib/rook/rook
2017-08-20 15:24:27.215235 I | cephmon: writing config file /var/lib/rook/rook-ceph-mon52/rook.config
2017-08-20 15:24:27.215366 I | cephmon: initializing mon
2017-08-20 15:24:27.215377 I | exec: Running command: monmaptool /var/lib/rook/rook-ceph-mon52/monmap --create --clobber --fsid 2c50d7fc-8533-45bd-bf42-970c435e4a3d --add rook-ceph-mon52 10.3.179.50:6790 --add rook-ceph-mon53 10.3.14.3:6790 --add rook-ceph-mon51 10.3.157.215:6790 --add rook-ceph-mon54 10.3.202.111:6790
2017-08-20 15:24:27.236362 I | monmaptool: monmap file /var/lib/rook/rook-ceph-mon52/monmap
2017-08-20 15:24:27.236375 I | monmaptool: set fsid to 2c50d7fc-8533-45bd-bf42-970c435e4a3d
2017-08-20 15:24:27.236396 I | monmaptool: writing epoch 0 to /var/lib/rook/rook-ceph-mon52/monmap (4 monitors)
2017-08-20 15:24:32.238182 I | proc: Running process ceph-mon with args: [--mkfs --name=mon.rook-ceph-mon52 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon52/data --conf=/var/lib/rook/rook-ceph-mon52/rook.config --keyring=/var/lib/rook/rook-ceph-mon52/keyring --monmap=/var/lib/rook/rook-ceph-mon52/monmap]
2017-08-20 15:24:32.238195 I | exec: Running command: ceph-mon --mkfs --name=mon.rook-ceph-mon52 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon52/data --conf=/var/lib/rook/rook-ceph-mon52/rook.config --keyring=/var/lib/rook/rook-ceph-mon52/keyring --monmap=/var/lib/rook/rook-ceph-mon52/monmap
2017-08-20 15:24:32.251593 I | mkfs-rook-ceph-mon52: 2017-08-20 15:24:32.251537 7f60871baf40 -1 '/var/lib/rook/rook-ceph-mon52/data' already exists and is not empty: monitor may already exist
2017-08-20 15:24:32.252819 I | mkfs-rook-ceph-mon52: 2017-08-20 15:24:32.251537 7f60871baf40 -1 '/var/lib/rook/rook-ceph-mon52/data' already exists and is not empty: monitor may already exist
2017-08-20 15:24:32.252843 I | cephmon: starting mon
2017-08-20 15:24:32.252880 I | cephmon: Config file /var/lib/rook/rook-ceph-mon52/rook.config:
[global]
enable experimental unrecoverable data corrupting features =
fsid = 2c50d7fc-8533-45bd-bf42-970c435e4a3d
run dir = /var/lib/rook/rook-ceph-mon52
mon initial members = rook-ceph-mon52 rook-ceph-mon53 rook-ceph-mon51 rook-ceph-mon54
mon host = 10.3.179.50:6790,10.3.14.3:6790,10.3.157.215:6790,10.3.202.111:6790
log file = /dev/stdout
mon cluster log file = /dev/stdout
public addr = 10.3.179.50
cluster addr = 100.64.1.33
mon keyvaluedb = rocksdb
debug default = 0
debug rados = 0
debug mon = 0
debug osd = 0
debug bluestore = 0
debug filestore = 0
debug journal = 0
debug leveldb = 0
filestore_omap_backend = rocksdb
osd pg bits = 11
osd pgp bits = 11
osd pool default size = 1
osd pool default min size = 1
osd pool default pg num = 100
osd pool default pgp num = 100
osd objectstore = filestore
rbd_default_features = 3
fatal signal handlers = false
[client.admin]
keyring = /var/lib/rook/rook-ceph-mon52/keyring
2017-08-20 15:24:32.252896 I | proc: Running process ceph-mon with args: [--foreground --name=mon.rook-ceph-mon52 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon52/data --conf=/var/lib/rook/rook-ceph-mon52/rook.config --keyring=/var/lib/rook/rook-ceph-mon52/keyring --public-addr=10.3.179.50:6790 --public-bind-addr=100.64.1.33:6790]
2017-08-20 15:24:32.252905 I | exec: Running command: ceph-mon --foreground --name=mon.rook-ceph-mon52 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon52/data --conf=/var/lib/rook/rook-ceph-mon52/rook.config --keyring=/var/lib/rook/rook-ceph-mon52/keyring --public-addr=10.3.179.50:6790 --public-bind-addr=100.64.1.33:6790
2017-08-20 15:24:32.265716 I | rook-ceph-mon52: 2017-08-20 15:24:32.265637 7f57dca18f40 0 ceph version ad3a8b682 (ead3a8b682051c4ab5adf80663c66d4521366930) luminous (rc), process (unknown), pid 24
2017-08-20 15:24:32.265747 I | rook-ceph-mon52: 2017-08-20 15:24:32.265714 7f57dca18f40 0 pidfile_write: ignore empty --pid-file
2017-08-20 15:24:32.270459 I | rook-ceph-mon52: 2017-08-20 15:24:32.270402 7f57dca18f40 0 load: jerasure load: lrc load: isa
2017-08-20 15:24:32.270486 I | rook-ceph-mon52: 2017-08-20 15:24:32.270469 7f57dca18f40 0 set rocksdb option compression = kNoCompression
2017-08-20 15:24:32.270490 I | rook-ceph-mon52: 2017-08-20 15:24:32.270475 7f57dca18f40 0 set rocksdb option write_buffer_size = 33554432
2017-08-20 15:24:32.270494 I | rook-ceph-mon52: 2017-08-20 15:24:32.270485 7f57dca18f40 0 set rocksdb option compression = kNoCompression
2017-08-20 15:24:32.270498 I | rook-ceph-mon52: 2017-08-20 15:24:32.270487 7f57dca18f40 0 set rocksdb option write_buffer_size = 33554432
2017-08-20 15:24:32.270594 I | rook-ceph-mon52: 2017-08-20 15:24:32.270557 7f57dca18f40 4 rocksdb: RocksDB version: 5.4.0
2017-08-20 15:24:32.270614 I | rook-ceph-mon52:
2017-08-20 15:24:32.270618 I | rook-ceph-mon52: 2017-08-20 15:24:32.270559 7f57dca18f40 4 rocksdb: Git sha rocksdb_build_git_sha:@e15382c09c87a65eaeca9bda233bab503f1e5772@
2017-08-20 15:24:32.270621 I | rook-ceph-mon52: 2017-08-20 15:24:32.270560 7f57dca18f40 4 rocksdb: Compile date Aug 11 2017
2017-08-20 15:24:32.270624 I | rook-ceph-mon52: 2017-08-20 15:24:32.270561 7f57dca18f40 4 rocksdb: DB SUMMARY
2017-08-20 15:24:32.270627 I | rook-ceph-mon52:
2017-08-20 15:24:32.270630 I | rook-ceph-mon52: 2017-08-20 15:24:32.270623 7f57dca18f40 4 rocksdb: CURRENT file: CURRENT
2017-08-20 15:24:32.270632 I | rook-ceph-mon52:
2017-08-20 15:24:32.270634 I | rook-ceph-mon52: 2017-08-20 15:24:32.270627 7f57dca18f40 4 rocksdb: IDENTITY file: IDENTITY
2017-08-20 15:24:32.270636 I | rook-ceph-mon52:
2017-08-20 15:24:32.270640 I | rook-ceph-mon52: 2017-08-20 15:24:32.270631 7f57dca18f40 4 rocksdb: MANIFEST file: MANIFEST-000011 size: 217 Bytes
2017-08-20 15:24:32.270645 I | rook-ceph-mon52:
2017-08-20 15:24:32.270649 I | rook-ceph-mon52: 2017-08-20 15:24:32.270634 7f57dca18f40 4 rocksdb: SST files in /var/lib/rook/rook-ceph-mon52/data/store.db dir, Total Num: 3, files: 000004.sst 000007.sst 000010.sst
2017-08-20 15:24:32.270651 I | rook-ceph-mon52:
2017-08-20 15:24:32.270653 I | rook-ceph-mon52: 2017-08-20 15:24:32.270636 7f57dca18f40 4 rocksdb: Write Ahead Log file in /var/lib/rook/rook-ceph-mon52/data/store.db: 000012.log size: 67597 ;
2017-08-20 15:24:32.270655 I | rook-ceph-mon52:
2017-08-20 15:24:32.270657 I | rook-ceph-mon52: 2017-08-20 15:24:32.270637 7f57dca18f40 4 rocksdb: Options.error_if_exists: 0
2017-08-20 15:24:32.270660 I | rook-ceph-mon52: 2017-08-20 15:24:32.270638 7f57dca18f40 4 rocksdb: Options.create_if_missing: 0
2017-08-20 15:24:32.270661 I | rook-ceph-mon52: 2017-08-20 15:24:32.270639 7f57dca18f40 4 rocksdb: Options.paranoid_checks: 1
2017-08-20 15:24:32.270663 I | rook-ceph-mon52: 2017-08-20 15:24:32.270640 7f57dca18f40 4 rocksdb: Options.env: 0xce3b53b560
2017-08-20 15:24:32.270667 I | rook-ceph-mon52: 2017-08-20 15:24:32.270641 7f57dca18f40 4 rocksdb: Options.info_log: 0xce3ca78e20
2017-08-20 15:24:32.270669 I | rook-ceph-mon52: 2017-08-20 15:24:32.270641 7f57dca18f40 4 rocksdb: Options.max_open_files: -1
2017-08-20 15:24:32.270670 I | rook-ceph-mon52: 2017-08-20 15:24:32.270642 7f57dca18f40 4 rocksdb: Options.max_file_opening_threads: 16
2017-08-20 15:24:32.270672 I | rook-ceph-mon52: 2017-08-20 15:24:32.270642 7f57dca18f40 4 rocksdb: Options.use_fsync: 0
2017-08-20 15:24:32.270674 I | rook-ceph-mon52: 2017-08-20 15:24:32.270643 7f57dca18f40 4 rocksdb: Options.max_log_file_size: 0
2017-08-20 15:24:32.270675 I | rook-ceph-mon52: 2017-08-20 15:24:32.270643 7f57dca18f40 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2017-08-20 15:24:32.270677 I | rook-ceph-mon52: 2017-08-20 15:24:32.270644 7f57dca18f40 4 rocksdb: Options.log_file_time_to_roll: 0
2017-08-20 15:24:32.270679 I | rook-ceph-mon52: 2017-08-20 15:24:32.270645 7f57dca18f40 4 rocksdb: Options.keep_log_file_num: 1000
2017-08-20 15:24:32.270681 I | rook-ceph-mon52: 2017-08-20 15:24:32.270645 7f57dca18f40 4 rocksdb: Options.recycle_log_file_num: 0
2017-08-20 15:24:32.270684 I | rook-ceph-mon52: 2017-08-20 15:24:32.270646 7f57dca18f40 4 rocksdb: Options.allow_fallocate: 1
2017-08-20 15:24:32.270686 I | rook-ceph-mon52: 2017-08-20 15:24:32.270647 7f57dca18f40 4 rocksdb: Options.allow_mmap_reads: 0
2017-08-20 15:24:32.270688 I | rook-ceph-mon52: 2017-08-20 15:24:32.270648 7f57dca18f40 4 rocksdb: Options.allow_mmap_writes: 0
2017-08-20 15:24:32.270690 I | rook-ceph-mon52: 2017-08-20 15:24:32.270648 7f57dca18f40 4 rocksdb: Options.use_direct_reads: 0
2017-08-20 15:24:32.270693 I | rook-ceph-mon52: 2017-08-20 15:24:32.270649 7f57dca18f40 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
2017-08-20 15:24:32.270698 I | rook-ceph-mon52: 2017-08-20 15:24:32.270650 7f57dca18f40 4 rocksdb: Options.create_missing_column_families: 0
2017-08-20 15:24:32.270699 I | rook-ceph-mon52: 2017-08-20 15:24:32.270650 7f57dca18f40 4 rocksdb: Options.db_log_dir:
2017-08-20 15:24:32.270702 I | rook-ceph-mon52: 2017-08-20 15:24:32.270651 7f57dca18f40 4 rocksdb: Options.wal_dir: /var/lib/rook/rook-ceph-mon52/data/store.db
2017-08-20 15:24:32.270705 I | rook-ceph-mon52: 2017-08-20 15:24:32.270653 7f57dca18f40 4 rocksdb: Options.table_cache_numshardbits: 6
2017-08-20 15:24:32.270707 I | rook-ceph-mon52: 2017-08-20 15:24:32.270653 7f57dca18f40 4 rocksdb: Options.max_subcompactions: 1
2017-08-20 15:24:32.270708 I | rook-ceph-mon52: 2017-08-20 15:24:32.270654 7f57dca18f40 4 rocksdb: Options.max_background_flushes: 1
2017-08-20 15:24:32.270710 I | rook-ceph-mon52: 2017-08-20 15:24:32.270655 7f57dca18f40 4 rocksdb: Options.WAL_ttl_seconds: 0
2017-08-20 15:24:32.270712 I | rook-ceph-mon52: 2017-08-20 15:24:32.270656 7f57dca18f40 4 rocksdb: Options.WAL_size_limit_MB: 0
2017-08-20 15:24:32.270713 I | rook-ceph-mon52: 2017-08-20 15:24:32.270657 7f57dca18f40 4 rocksdb: Options.manifest_preallocation_size: 4194304
2017-08-20 15:24:32.270715 I | rook-ceph-mon52: 2017-08-20 15:24:32.270658 7f57dca18f40 4 rocksdb: Options.is_fd_close_on_exec: 1
2017-08-20 15:24:32.270719 I | rook-ceph-mon52: 2017-08-20 15:24:32.270659 7f57dca18f40 4 rocksdb: Options.advise_random_on_open: 1
2017-08-20 15:24:32.270721 I | rook-ceph-mon52: 2017-08-20 15:24:32.270659 7f57dca18f40 4 rocksdb: Options.db_write_buffer_size: 0
2017-08-20 15:24:32.270723 I | rook-ceph-mon52: 2017-08-20 15:24:32.270661 7f57dca18f40 4 rocksdb: Options.access_hint_on_compaction_start: 1
2017-08-20 15:24:32.270725 I | rook-ceph-mon52: 2017-08-20 15:24:32.270661 7f57dca18f40 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2017-08-20 15:24:32.270727 I | rook-ceph-mon52: 2017-08-20 15:24:32.270662 7f57dca18f40 4 rocksdb: Options.compaction_readahead_size: 0
2017-08-20 15:24:32.270728 I | rook-ceph-mon52: 2017-08-20 15:24:32.270663 7f57dca18f40 4 rocksdb: Options.random_access_max_buffer_size: 1048576
2017-08-20 15:24:32.270731 I | rook-ceph-mon52: 2017-08-20 15:24:32.270663 7f57dca18f40 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
2017-08-20 15:24:32.270733 I | rook-ceph-mon52: 2017-08-20 15:24:32.270664 7f57dca18f40 4 rocksdb: Options.use_adaptive_mutex: 0
2017-08-20 15:24:32.270735 I | rook-ceph-mon52: 2017-08-20 15:24:32.270665 7f57dca18f40 4 rocksdb: Options.rate_limiter: (nil)
2017-08-20 15:24:32.270736 I | rook-ceph-mon52: 2017-08-20 15:24:32.270666 7f57dca18f40 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2017-08-20 15:24:32.270739 I | rook-ceph-mon52: 2017-08-20 15:24:32.270666 7f57dca18f40 4 rocksdb: Options.bytes_per_sync: 0
2017-08-20 15:24:32.270741 I | rook-ceph-mon52: 2017-08-20 15:24:32.270667 7f57dca18f40 4 rocksdb: Options.wal_bytes_per_sync: 0
2017-08-20 15:24:32.270743 I | rook-ceph-mon52: 2017-08-20 15:24:32.270668 7f57dca18f40 4 rocksdb: Options.wal_recovery_mode: 2
2017-08-20 15:24:32.270745 I | rook-ceph-mon52: 2017-08-20 15:24:32.270668 7f57dca18f40 4 rocksdb: Options.enable_thread_tracking: 0
2017-08-20 15:24:32.270747 I | rook-ceph-mon52: 2017-08-20 15:24:32.270669 7f57dca18f40 4 rocksdb: Options.allow_concurrent_memtable_write: 1
2017-08-20 15:24:32.270749 I | rook-ceph-mon52: 2017-08-20 15:24:32.270670 7f57dca18f40 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2017-08-20 15:24:32.270751 I | rook-ceph-mon52: 2017-08-20 15:24:32.270670 7f57dca18f40 4 rocksdb: Options.write_thread_max_yield_usec: 100
2017-08-20 15:24:32.270753 I | rook-ceph-mon52: 2017-08-20 15:24:32.270671 7f57dca18f40 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2017-08-20 15:24:32.270754 I | rook-ceph-mon52: 2017-08-20 15:24:32.270672 7f57dca18f40 4 rocksdb: Options.row_cache: None
2017-08-20 15:24:32.270756 I | rook-ceph-mon52: 2017-08-20 15:24:32.270672 7f57dca18f40 4 rocksdb: Options.wal_filter: None
2017-08-20 15:24:32.270758 I | rook-ceph-mon52: 2017-08-20 15:24:32.270673 7f57dca18f40 4 rocksdb: Options.avoid_flush_during_recovery: 0
2017-08-20 15:24:32.270759 I | rook-ceph-mon52: 2017-08-20 15:24:32.270674 7f57dca18f40 4 rocksdb: Options.base_background_compactions: 1
2017-08-20 15:24:32.270761 I | rook-ceph-mon52: 2017-08-20 15:24:32.270675 7f57dca18f40 4 rocksdb: Options.max_background_compactions: 1
2017-08-20 15:24:32.270763 I | rook-ceph-mon52: 2017-08-20 15:24:32.270675 7f57dca18f40 4 rocksdb: Options.avoid_flush_during_shutdown: 0
2017-08-20 15:24:32.270767 I | rook-ceph-mon52: 2017-08-20 15:24:32.270676 7f57dca18f40 4 rocksdb: Options.delayed_write_rate : 16777216
2017-08-20 15:24:32.270770 I | rook-ceph-mon52: 2017-08-20 15:24:32.270677 7f57dca18f40 4 rocksdb: Options.max_total_wal_size: 0
2017-08-20 15:24:32.270771 I | rook-ceph-mon52: 2017-08-20 15:24:32.270678 7f57dca18f40 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2017-08-20 15:24:32.270774 I | rook-ceph-mon52: 2017-08-20 15:24:32.270678 7f57dca18f40 4 rocksdb: Options.stats_dump_period_sec: 600
2017-08-20 15:24:32.270776 I | rook-ceph-mon52: 2017-08-20 15:24:32.270679 7f57dca18f40 4 rocksdb: Compression algorithms supported:
2017-08-20 15:24:32.270778 I | rook-ceph-mon52: 2017-08-20 15:24:32.270680 7f57dca18f40 4 rocksdb: Snappy supported: 0
2017-08-20 15:24:32.270779 I | rook-ceph-mon52: 2017-08-20 15:24:32.270681 7f57dca18f40 4 rocksdb: Zlib supported: 0
2017-08-20 15:24:32.270781 I | rook-ceph-mon52: 2017-08-20 15:24:32.270681 7f57dca18f40 4 rocksdb: Bzip supported: 0
2017-08-20 15:24:32.270783 I | rook-ceph-mon52: 2017-08-20 15:24:32.270682 7f57dca18f40 4 rocksdb: LZ4 supported: 0
2017-08-20 15:24:32.270784 I | rook-ceph-mon52: 2017-08-20 15:24:32.270683 7f57dca18f40 4 rocksdb: ZSTD supported: 0
2017-08-20 15:24:32.270786 I | rook-ceph-mon52: 2017-08-20 15:24:32.270683 7f57dca18f40 4 rocksdb: Fast CRC32 supported: 1
2017-08-20 15:24:32.273151 I | rook-ceph-mon52: 2017-08-20 15:24:32.273109 7f57dca18f40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000011
2017-08-20 15:24:32.273155 I | rook-ceph-mon52:
2017-08-20 15:24:32.273256 I | rook-ceph-mon52: 2017-08-20 15:24:32.273217 7f57dca18f40 4 rocksdb: [/build/ceph/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
2017-08-20 15:24:32.273259 I | rook-ceph-mon52:
2017-08-20 15:24:32.273261 I | rook-ceph-mon52: 2017-08-20 15:24:32.273220 7f57dca18f40 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2017-08-20 15:24:32.273262 I | rook-ceph-mon52: 2017-08-20 15:24:32.273221 7f57dca18f40 4 rocksdb: Options.merge_operator:
2017-08-20 15:24:32.273264 I | rook-ceph-mon52: 2017-08-20 15:24:32.273222 7f57dca18f40 4 rocksdb: Options.compaction_filter: None
2017-08-20 15:24:32.273266 I | rook-ceph-mon52: 2017-08-20 15:24:32.273223 7f57dca18f40 4 rocksdb: Options.compaction_filter_factory: None
2017-08-20 15:24:32.273269 I | rook-ceph-mon52: 2017-08-20 15:24:32.273224 7f57dca18f40 4 rocksdb: Options.memtable_factory: SkipListFactory
2017-08-20 15:24:32.273274 I | rook-ceph-mon52: 2017-08-20 15:24:32.273224 7f57dca18f40 4 rocksdb: Options.table_factory: BlockBasedTable
2017-08-20 15:24:32.273276 I | rook-ceph-mon52: 2017-08-20 15:24:32.273234 7f57dca18f40 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0xce3c9fe138)
2017-08-20 15:24:32.273279 I | rook-ceph-mon52: cache_index_and_filter_blocks: 1
2017-08-20 15:24:32.273281 I | rook-ceph-mon52: cache_index_and_filter_blocks_with_high_priority: 1
2017-08-20 15:24:32.273285 I | rook-ceph-mon52: pin_l0_filter_and_index_blocks_in_cache: 1
2017-08-20 15:24:32.273287 I | rook-ceph-mon52: index_type: 0
2017-08-20 15:24:32.273289 I | rook-ceph-mon52: hash_index_allow_collision: 1
2017-08-20 15:24:32.273290 I | rook-ceph-mon52: checksum: 1
2017-08-20 15:24:32.273292 I | rook-ceph-mon52: no_block_cache: 0
2017-08-20 15:24:32.273295 I | rook-ceph-mon52: block_cache: 0xce3ccff1f0
2017-08-20 15:24:32.273297 I | rook-ceph-mon52: block_cache_name: LRUCache
2017-08-20 15:24:32.273300 I | rook-ceph-mon52: block_cache_options:
2017-08-20 15:24:32.273302 I | rook-ceph-mon52: capacity : 134217728
2017-08-20 15:24:32.273304 I | rook-ceph-mon52: num_shard_bits : 4
2017-08-20 15:24:32.273305 I | rook-ceph-mon52: strict_capacity_limit : 0
2017-08-20 15:24:32.273320 I | rook-ceph-mon52: high_pri_pool_ratio: 0.000
2017-08-20 15:24:32.273322 I | rook-ceph-mon52: block_cache_compressed: (nil)
2017-08-20 15:24:32.273325 I | rook-ceph-mon52: persistent_cache: (nil)
2017-08-20 15:24:32.273327 I | rook-ceph-mon52: block_size: 4096
2017-08-20 15:24:32.273328 I | rook-ceph-mon52: block_size_deviation: 10
2017-08-20 15:24:32.273330 I | rook-ceph-mon52: block_restart_interval: 16
2017-08-20 15:24:32.273332 I | rook-ceph-mon52: index_block_restart_interval: 1
2017-08-20 15:24:32.273333 I | rook-ceph-mon52: filter_policy: rocksdb.BuiltinBloomFilter
2017-08-20 15:24:32.273335 I | rook-ceph-mon52: whole_key_filtering: 1
2017-08-20 15:24:32.273337 I | rook-ceph-mon52: format_version: 2
2017-08-20 15:24:32.273338 I | rook-ceph-mon52:
2017-08-20 15:24:32.273341 I | rook-ceph-mon52: 2017-08-20 15:24:32.273237 7f57dca18f40 4 rocksdb: Options.write_buffer_size: 33554432
2017-08-20 15:24:32.273343 I | rook-ceph-mon52: 2017-08-20 15:24:32.273237 7f57dca18f40 4 rocksdb: Options.max_write_buffer_number: 2
2017-08-20 15:24:32.273345 I | rook-ceph-mon52: 2017-08-20 15:24:32.273238 7f57dca18f40 4 rocksdb: Options.compression: NoCompression
2017-08-20 15:24:32.273347 I | rook-ceph-mon52: 2017-08-20 15:24:32.273238 7f57dca18f40 4 rocksdb: Options.bottommost_compression: Disabled
2017-08-20 15:24:32.273348 I | rook-ceph-mon52: 2017-08-20 15:24:32.273252 7f57dca18f40 4 rocksdb: Options.prefix_extractor: nullptr
2017-08-20 15:24:32.273350 I | rook-ceph-mon52: 2017-08-20 15:24:32.273253 7f57dca18f40 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2017-08-20 15:24:32.273360 I | rook-ceph-mon52: 2017-08-20 15:24:32.273253 7f57dca18f40 4 rocksdb: Options.num_levels: 7
2017-08-20 15:24:32.273362 I | rook-ceph-mon52: 2017-08-20 15:24:32.273254 7f57dca18f40 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2017-08-20 15:24:32.273364 I | rook-ceph-mon52: 2017-08-20 15:24:32.273254 7f57dca18f40 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2017-08-20 15:24:32.273365 I | rook-ceph-mon52: 2017-08-20 15:24:32.273255 7f57dca18f40 4 rocksdb: Options.compression_opts.window_bits: -14
2017-08-20 15:24:32.273368 I | rook-ceph-mon52: 2017-08-20 15:24:32.273255 7f57dca18f40 4 rocksdb: Options.compression_opts.level: -1
2017-08-20 15:24:32.273373 I | rook-ceph-mon52: 2017-08-20 15:24:32.273256 7f57dca18f40 4 rocksdb: Options.compression_opts.strategy: 0
2017-08-20 15:24:32.273375 I | rook-ceph-mon52: 2017-08-20 15:24:32.273256 7f57dca18f40 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2017-08-20 15:24:32.273377 I | rook-ceph-mon52: 2017-08-20 15:24:32.273257 7f57dca18f40 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2017-08-20 15:24:32.273380 I | rook-ceph-mon52: 2017-08-20 15:24:32.273257 7f57dca18f40 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2017-08-20 15:24:32.273381 I | rook-ceph-mon52: 2017-08-20 15:24:32.273258 7f57dca18f40 4 rocksdb: Options.level0_stop_writes_trigger: 36
2017-08-20 15:24:32.273383 I | rook-ceph-mon52: 2017-08-20 15:24:32.273258 7f57dca18f40 4 rocksdb: Options.target_file_size_base: 67108864
2017-08-20 15:24:32.273385 I | rook-ceph-mon52: 2017-08-20 15:24:32.273259 7f57dca18f40 4 rocksdb: Options.target_file_size_multiplier: 1
2017-08-20 15:24:32.273386 I | rook-ceph-mon52: 2017-08-20 15:24:32.273259 7f57dca18f40 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2017-08-20 15:24:32.273389 I | rook-ceph-mon52: 2017-08-20 15:24:32.273260 7f57dca18f40 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2017-08-20 15:24:32.273391 I | rook-ceph-mon52: 2017-08-20 15:24:32.273261 7f57dca18f40 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2017-08-20 15:24:32.273393 I | rook-ceph-mon52: 2017-08-20 15:24:32.273262 7f57dca18f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2017-08-20 15:24:32.273395 I | rook-ceph-mon52: 2017-08-20 15:24:32.273263 7f57dca18f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2017-08-20 15:24:32.273398 I | rook-ceph-mon52: 2017-08-20 15:24:32.273264 7f57dca18f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2017-08-20 15:24:32.273401 I | rook-ceph-mon52: 2017-08-20 15:24:32.273265 7f57dca18f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2017-08-20 15:24:32.273402 I | rook-ceph-mon52: 2017-08-20 15:24:32.273266 7f57dca18f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2017-08-20 15:24:32.273404 I | rook-ceph-mon52: 2017-08-20 15:24:32.273267 7f57dca18f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2017-08-20 15:24:32.273407 I | rook-ceph-mon52: 2017-08-20 15:24:32.273268 7f57dca18f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2017-08-20 15:24:32.273408 I | rook-ceph-mon52: 2017-08-20 15:24:32.273269 7f57dca18f40 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2017-08-20 15:24:32.273410 I | rook-ceph-mon52: 2017-08-20 15:24:32.273269 7f57dca18f40 4 rocksdb: Options.max_compaction_bytes: 1677721600
2017-08-20 15:24:32.273412 I | rook-ceph-mon52: 2017-08-20 15:24:32.273270 7f57dca18f40 4 rocksdb: Options.arena_block_size: 4194304
2017-08-20 15:24:32.273413 I | rook-ceph-mon52: 2017-08-20 15:24:32.273275 7f57dca18f40 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2017-08-20 15:24:32.273415 I | rook-ceph-mon52: 2017-08-20 15:24:32.273276 7f57dca18f40 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2017-08-20 15:24:32.273417 I | rook-ceph-mon52: 2017-08-20 15:24:32.273277 7f57dca18f40 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2017-08-20 15:24:32.273418 I | rook-ceph-mon52: 2017-08-20 15:24:32.273278 7f57dca18f40 4 rocksdb: Options.disable_auto_compactions: 0
2017-08-20 15:24:32.273420 I | rook-ceph-mon52: 2017-08-20 15:24:32.273279 7f57dca18f40 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2017-08-20 15:24:32.273424 I | rook-ceph-mon52: 2017-08-20 15:24:32.273280 7f57dca18f40 4 rocksdb: Options.compaction_pri: kByCompensatedSize
2017-08-20 15:24:32.273426 I | rook-ceph-mon52: 2017-08-20 15:24:32.273281 7f57dca18f40 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2017-08-20 15:24:32.273427 I | rook-ceph-mon52: 2017-08-20 15:24:32.273282 7f57dca18f40 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2017-08-20 15:24:32.273430 I | rook-ceph-mon52: 2017-08-20 15:24:32.273282 7f57dca18f40 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2017-08-20 15:24:32.273433 I | rook-ceph-mon52: 2017-08-20 15:24:32.273283 7f57dca18f40 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2017-08-20 15:24:32.273435 I | rook-ceph-mon52: 2017-08-20 15:24:32.273284 7f57dca18f40 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2017-08-20 15:24:32.273437 I | rook-ceph-mon52: 2017-08-20 15:24:32.273284 7f57dca18f40 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2017-08-20 15:24:32.273438 I | rook-ceph-mon52: 2017-08-20 15:24:32.273285 7f57dca18f40 4 rocksdb: Options.table_properties_collectors:
2017-08-20 15:24:32.273440 I | rook-ceph-mon52: 2017-08-20 15:24:32.273286 7f57dca18f40 4 rocksdb: Options.inplace_update_support: 0
2017-08-20 15:24:32.273442 I | rook-ceph-mon52: 2017-08-20 15:24:32.273286 7f57dca18f40 4 rocksdb: Options.inplace_update_num_locks: 10000
2017-08-20 15:24:32.273445 I | rook-ceph-mon52: 2017-08-20 15:24:32.273287 7f57dca18f40 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2017-08-20 15:24:32.273447 I | rook-ceph-mon52: 2017-08-20 15:24:32.273288 7f57dca18f40 4 rocksdb: Options.memtable_huge_page_size: 0
2017-08-20 15:24:32.273449 I | rook-ceph-mon52: 2017-08-20 15:24:32.273289 7f57dca18f40 4 rocksdb: Options.bloom_locality: 0
2017-08-20 15:24:32.273450 I | rook-ceph-mon52: 2017-08-20 15:24:32.273290 7f57dca18f40 4 rocksdb: Options.max_successive_merges: 0
2017-08-20 15:24:32.273454 I | rook-ceph-mon52: 2017-08-20 15:24:32.273291 7f57dca18f40 4 rocksdb: Options.optimize_filters_for_hits: 0
2017-08-20 15:24:32.273456 I | rook-ceph-mon52: 2017-08-20 15:24:32.273292 7f57dca18f40 4 rocksdb: Options.paranoid_file_checks: 0
2017-08-20 15:24:32.273459 I | rook-ceph-mon52: 2017-08-20 15:24:32.273293 7f57dca18f40 4 rocksdb: Options.force_consistency_checks: 0
2017-08-20 15:24:32.273461 I | rook-ceph-mon52: 2017-08-20 15:24:32.273293 7f57dca18f40 4 rocksdb: Options.report_bg_io_stats: 0
2017-08-20 15:24:32.274078 I | rook-ceph-mon52: 2017-08-20 15:24:32.273992 7f57dca18f40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/rook/rook-ceph-mon52/data/store.db/MANIFEST-000011 succeeded,manifest_file_number is 11, next_file_number is 13, last_sequence is 2834, log_number is 0,prev_log_number is 0,max_column_family is 0
2017-08-20 15:24:32.274085 I | rook-ceph-mon52:
2017-08-20 15:24:32.274088 I | rook-ceph-mon52: 2017-08-20 15:24:32.273998 7f57dca18f40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 10
2017-08-20 15:24:32.274090 I | rook-ceph-mon52:
2017-08-20 15:24:32.274183 I | rook-ceph-mon52: 2017-08-20 15:24:32.274065 7f57dca18f40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503242672274061, "job": 1, "event": "recovery_started", "log_files": [12]}
2017-08-20 15:24:32.274205 I | rook-ceph-mon52: 2017-08-20 15:24:32.274080 7f57dca18f40 4 rocksdb: [/build/ceph/src/rocksdb/db/db_impl_open.cc:482] Recovering log #12 mode 2
2017-08-20 15:24:32.304751 I | rook-ceph-mon52: 2017-08-20 15:24:32.304677 7f57dca18f40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503242672304670, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 45683, "table_properties": {"data_size": 44396, "index_size": 189, "filter_size": 181, "raw_key_size": 627, "raw_average_key_size": 23, "raw_value_size": 43718, "raw_average_value_size": 1619, "num_data_blocks": 6, "num_entries": 27, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2017-08-20 15:24:32.304762 I | rook-ceph-mon52: 2017-08-20 15:24:32.304715 7f57dca18f40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2395] Creating manifest 14
2017-08-20 15:24:32.304765 I | rook-ceph-mon52:
2017-08-20 15:24:32.338473 I | rook-ceph-mon52: 2017-08-20 15:24:32.338389 7f57dca18f40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503242672338387, "job": 1, "event": "recovery_finished"}
2017-08-20 15:24:32.372663 I | rook-ceph-mon52: 2017-08-20 15:24:32.372576 7f57dca18f40 4 rocksdb: [/build/ceph/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0xce3ce02000
2017-08-20 15:24:32.372750 I | rook-ceph-mon52: 2017-08-20 15:24:32.372737 7f57cc7d5700 3 rocksdb: [/build/ceph/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
2017-08-20 15:24:32.372758 I | rook-ceph-mon52: 2017-08-20 15:24:32.372741 7f57cc7d5700 3 rocksdb: [/build/ceph/src/rocksdb/db/db_impl.cc:448]
2017-08-20 15:24:32.372766 I | rook-ceph-mon52: ** DB Stats **
2017-08-20 15:24:32.372784 I | rook-ceph-mon52: Uptime(secs): 0.1 total, 0.1 interval
2017-08-20 15:24:32.372789 I | rook-ceph-mon52: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
2017-08-20 15:24:32.372805 I | rook-ceph-mon52: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
2017-08-20 15:24:32.372809 I | rook-ceph-mon52: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
2017-08-20 15:24:32.372813 I | rook-ceph-mon52: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
2017-08-20 15:24:32.372816 I | rook-ceph-mon52: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
2017-08-20 15:24:32.372820 I | rook-ceph-mon52: Interval stall: 00:00:0.000 H:M:S, 0.0 percent
2017-08-20 15:24:32.372824 I | rook-ceph-mon52:
2017-08-20 15:24:32.372827 I | rook-ceph-mon52: ** Compaction Stats [default] **
2017-08-20 15:24:32.372832 I | rook-ceph-mon52: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
2017-08-20 15:24:32.372835 I | rook-ceph-mon52: ----------------------------------------------------------------------------------------------------------------------------------------------------------
2017-08-20 15:24:32.372838 I | rook-ceph-mon52: L0 4/0 11.34 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0 1 0.025 0 0
2017-08-20 15:24:32.372841 I | rook-ceph-mon52: Sum 4/0 11.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0 1 0.025 0 0
2017-08-20 15:24:32.372845 I | rook-ceph-mon52: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0 1 0.025 0 0
2017-08-20 15:24:32.372849 I | rook-ceph-mon52: Uptime(secs): 0.1 total, 0.1 interval
2017-08-20 15:24:32.372866 I | rook-ceph-mon52: Flush(GB): cumulative 0.000, interval 0.000
2017-08-20 15:24:32.372870 I | rook-ceph-mon52: AddFile(GB): cumulative 0.000, interval 0.000
2017-08-20 15:24:32.372874 I | rook-ceph-mon52: AddFile(Total Files): cumulative 0, interval 0
2017-08-20 15:24:32.372877 I | rook-ceph-mon52: AddFile(L0 Files): cumulative 0, interval 0
2017-08-20 15:24:32.372880 I | rook-ceph-mon52: AddFile(Keys): cumulative 0, interval 0
2017-08-20 15:24:32.372884 I | rook-ceph-mon52: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
2017-08-20 15:24:32.372889 I | rook-ceph-mon52: Interval compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
2017-08-20 15:24:32.372895 I | rook-ceph-mon52: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
2017-08-20 15:24:32.372899 I | rook-ceph-mon52:
2017-08-20 15:24:32.372915 I | rook-ceph-mon52: ** File Read Latency Histogram By Level [default] **
2017-08-20 15:24:32.372919 I | rook-ceph-mon52:
2017-08-20 15:24:32.372921 I | rook-ceph-mon52: ** Compaction Stats [default] **
2017-08-20 15:24:32.372929 I | rook-ceph-mon52: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
2017-08-20 15:24:32.372932 I | rook-ceph-mon52: ----------------------------------------------------------------------------------------------------------------------------------------------------------
2017-08-20 15:24:32.372935 I | rook-ceph-mon52: L0 4/0 11.34 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0 1 0.025 0 0
2017-08-20 15:24:32.372941 I | rook-ceph-mon52: Sum 4/0 11.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0 1 0.025 0 0
2017-08-20 15:24:32.372945 I | rook-ceph-mon52: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
2017-08-20 15:24:32.372948 I | rook-ceph-mon52: Uptime(secs): 0.1 total, 0.1 interval
2017-08-20 15:24:32.372953 I | rook-ceph-mon52: Flush(GB): cumulative 0.000, interval 0.000
2017-08-20 15:24:32.372956 I | rook-ceph-mon52: AddFile(GB): cumulative 0.000, interval 0.000
2017-08-20 15:24:32.372960 I | rook-ceph-mon52: AddFile(Total Files): cumulative 0, interval 0
2017-08-20 15:24:32.372962 I | rook-ceph-mon52: AddFile(L0 Files): cumulative 0, interval 0
2017-08-20 15:24:32.372966 I | rook-ceph-mon52: AddFile(Keys): cumulative 0, interval 0
2017-08-20 15:24:32.372968 I | rook-ceph-mon52: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
2017-08-20 15:24:32.372973 I | rook-ceph-mon52: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
2017-08-20 15:24:32.372977 I | rook-ceph-mon52: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
2017-08-20 15:24:32.372979 I | rook-ceph-mon52:
2017-08-20 15:24:32.372984 I | rook-ceph-mon52: ** File Read Latency Histogram By Level [default] **
2017-08-20 15:24:32.372988 I | rook-ceph-mon52:
2017-08-20 15:24:32.372990 I | rook-ceph-mon52: 2017-08-20 15:24:32.372768 7f57cc7d5700 4 rocksdb: [/build/ceph/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 3] Compacting 4@0 files to L1, score 1.00
2017-08-20 15:24:32.372995 I | rook-ceph-mon52: 2017-08-20 15:24:32.372779 7f57cc7d5700 4 rocksdb: [/build/ceph/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2 Base level 0, inputs: [13(44KB) 10(144KB) 7(11MB) 4(2039B)]
2017-08-20 15:24:32.372998 I | rook-ceph-mon52:
2017-08-20 15:24:32.373003 I | rook-ceph-mon52: 2017-08-20 15:24:32.372809 7f57cc7d5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503242672372784, "job": 3, "event": "compaction_started", "files_L0": [13, 10, 7, 4], "score": 1, "input_data_size": 11894169}
2017-08-20 15:24:32.373322 I | rook-ceph-mon52: 2017-08-20 15:24:32.373287 7f57dca18f40 0 starting mon.rook-ceph-mon52 rank 2 at public addr 10.3.179.50:6790/0 at bind addr 100.64.1.33:6790/0 mon_data /var/lib/rook/rook-ceph-mon52/data fsid 2c50d7fc-8533-45bd-bf42-970c435e4a3d
2017-08-20 15:24:32.373430 I | rook-ceph-mon52: 2017-08-20 15:24:32.373418 7f57dca18f40 0 starting mon.rook-ceph-mon52 rank 2 at 10.3.179.50:6790/0 mon_data /var/lib/rook/rook-ceph-mon52/data fsid 2c50d7fc-8533-45bd-bf42-970c435e4a3d
2017-08-20 15:24:32.373855 I | rook-ceph-mon52: 2017-08-20 15:24:32.373840 7f57dca18f40 1 mon.rook-ceph-mon52@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
2017-08-20 15:24:32.373935 I | rook-ceph-mon52: 2017-08-20 15:24:32.373907 7f57dca18f40 0 mon.rook-ceph-mon52@-1(probing).mds e27 print_map
2017-08-20 15:24:32.373956 I | rook-ceph-mon52: e27
2017-08-20 15:24:32.373973 I | rook-ceph-mon52: enable_multiple, ever_enabled_multiple: 0,0
2017-08-20 15:24:32.373977 I | rook-ceph-mon52: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
2017-08-20 15:24:32.373981 I | rook-ceph-mon52: legacy client fscid: 1
2017-08-20 15:24:32.373984 I | rook-ceph-mon52:
2017-08-20 15:24:32.373989 I | rook-ceph-mon52: Filesystem 'cfs-r2-1' (1)
2017-08-20 15:24:32.373992 I | rook-ceph-mon52: fs_name cfs-r2-1
2017-08-20 15:24:32.373995 I | rook-ceph-mon52: epoch 27
2017-08-20 15:24:32.373999 I | rook-ceph-mon52: flags c
2017-08-20 15:24:32.374016 I | rook-ceph-mon52: created 2017-08-18 09:35:11.962291
2017-08-20 15:24:32.374020 I | rook-ceph-mon52: modified 2017-08-18 09:35:11.962291
2017-08-20 15:24:32.374037 I | rook-ceph-mon52: tableserver 0
2017-08-20 15:24:32.374040 I | rook-ceph-mon52: root 0
2017-08-20 15:24:32.374044 I | rook-ceph-mon52: session_timeout 60
2017-08-20 15:24:32.374048 I | rook-ceph-mon52: session_autoclose 300
2017-08-20 15:24:32.374050 I | rook-ceph-mon52: max_file_size 1099511627776
2017-08-20 15:24:32.374053 I | rook-ceph-mon52: last_failure 0
2017-08-20 15:24:32.374058 I | rook-ceph-mon52: last_failure_osd_epoch 174
2017-08-20 15:24:32.374061 I | rook-ceph-mon52: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
2017-08-20 15:24:32.374064 I | rook-ceph-mon52: max_mds 1
2017-08-20 15:24:32.374068 I | rook-ceph-mon52: in 0
2017-08-20 15:24:32.374072 I | rook-ceph-mon52: up {0=228660}
2017-08-20 15:24:32.374075 I | rook-ceph-mon52: failed
2017-08-20 15:24:32.374079 I | rook-ceph-mon52: damaged
2017-08-20 15:24:32.374082 I | rook-ceph-mon52: stopped
2017-08-20 15:24:32.374086 I | rook-ceph-mon52: data_pools [3]
2017-08-20 15:24:32.374089 I | rook-ceph-mon52: metadata_pool 4
2017-08-20 15:24:32.374092 I | rook-ceph-mon52: inline_data disabled
2017-08-20 15:24:32.374096 I | rook-ceph-mon52: balancer
2017-08-20 15:24:32.374099 I | rook-ceph-mon52: standby_count_wanted 0
2017-08-20 15:24:32.374104 I | rook-ceph-mon52: 228660: 100.64.0.48:6800/3988930595 'mds1' mds.0.17 up:active seq 18354
2017-08-20 15:24:32.374107 I | rook-ceph-mon52:
2017-08-20 15:24:32.374109 I | rook-ceph-mon52:
2017-08-20 15:24:32.374112 I | rook-ceph-mon52:
2017-08-20 15:24:32.374141 I | rook-ceph-mon52: 2017-08-20 15:24:32.374130 7f57dca18f40 0 mon.rook-ceph-mon52@-1(probing).osd e175 crush map has features 2200130813952, adjusting msgr requires
2017-08-20 15:24:32.374147 I | rook-ceph-mon52: 2017-08-20 15:24:32.374134 7f57dca18f40 0 mon.rook-ceph-mon52@-1(probing).osd e175 crush map has features 2200130813952, adjusting msgr requires
2017-08-20 15:24:32.374152 I | rook-ceph-mon52: 2017-08-20 15:24:32.374136 7f57dca18f40 0 mon.rook-ceph-mon52@-1(probing).osd e175 crush map has features 720578140510109696, adjusting msgr requires
2017-08-20 15:24:32.374156 I | rook-ceph-mon52: 2017-08-20 15:24:32.374136 7f57dca18f40 0 mon.rook-ceph-mon52@-1(probing).osd e175 crush map has features 2200130813952, adjusting msgr requires
2017-08-20 15:24:32.374254 I | rook-ceph-mon52: 2017-08-20 15:23:47.205584 mon.rook-ceph-mon53 mon.0 10.3.14.3:6790/0 27 : cluster [WRN] Health check failed: 1/3 mons down, quorum rook-ceph-mon53,rook-ceph-mon52 (MON_DOWN)
2017-08-20 15:24:32.374284 I | rook-ceph-mon52: 2017-08-20 15:24:32.374274 7f57dca18f40 1 mon.rook-ceph-mon52@-1(probing).paxosservice(auth 253..329) refresh upgraded, format 0 -> 2
2017-08-20 15:24:32.374861 I | rook-ceph-mon52: 2017-08-20 15:24:32.374828 7f57dca18f40 -1 mon.rook-ceph-mon52@-1(probing).mgr e7 Failed to load mgr commands: (2) No such file or directory
2017-08-20 15:24:32.375436 I | rook-ceph-mon52: 2017-08-20 15:24:32.375329 7f57dca18f40 0 mon.rook-ceph-mon52@-1(probing) e106 my rank is now 2 (was -1)
2017-08-20 15:24:32.376387 I | rook-ceph-mon52: 2017-08-20 15:24:32.376365 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:32.376891 I | rook-ceph-mon52: 2017-08-20 15:24:32.376824 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:32.378026 I | rook-ceph-mon52: 2017-08-20 15:24:32.377962 7f57d3fe4700 0 -- 10.3.179.50:6790/0 >> 10.3.202.111:6790/0 conn(0xce3cf72800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=26 cs=1 l=0).process missed message? skipped from seq 0 to 18319820
2017-08-20 15:24:32.378154 I | rook-ceph-mon52: 2017-08-20 15:24:32.378138 7f57cd7d7700 0 -- 10.3.179.50:6790/0 >> 10.3.14.3:6790/0 conn(0xce3cf6f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=46 cs=1 l=0).process missed message? skipped from seq 0 to 421879741
2017-08-20 15:24:32.378243 I | rook-ceph-mon52: 2017-08-20 15:24:32.378193 7f57d07dd700 0 log_channel(cluster) log [INF] : mon.rook-ceph-mon52 calling new monitor election
2017-08-20 15:24:32.498821 I | rook-ceph-mon52: 2017-08-20 15:24:32.498737 7f57cc7d5700 4 rocksdb: [/build/ceph/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #18: 2679 keys, 11885974 bytes
2017-08-20 15:24:32.498886 I | rook-ceph-mon52: 2017-08-20 15:24:32.498770 7f57cc7d5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503242672498762, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 18, "file_size": 11885974, "table_properties": {"data_size": 11802944, "index_size": 45411, "filter_size": 36692, "raw_key_size": 53812, "raw_average_key_size": 20, "raw_value_size": 11733343, "raw_average_value_size": 4379, "num_data_blocks": 1323, "num_entries": 2679, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2017-08-20 15:24:32.523110 I | rook-ceph-mon52: 2017-08-20 15:24:32.523026 7f57cc7d5700 4 rocksdb: [/build/ceph/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 3] Compacted 4@0 files to L1 => 11885974 bytes
2017-08-20 15:24:32.539915 I | rook-ceph-mon52: 2017-08-20 15:24:32.539845 7f57cc7d5700 4 rocksdb: (Original Log Time 2017/08/20-15:24:32.539776) [/build/ceph/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.04, MB/sec: 79.2 rd, 79.1 wr, level 1, files in(4, 0) out(1) MB in(11.3, 0.0) out(11.3), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 2715, records dropped: 36
2017-08-20 15:24:32.539956 I | rook-ceph-mon52:
2017-08-20 15:24:32.539961 I | rook-ceph-mon52: 2017-08-20 15:24:32.539854 7f57cc7d5700 4 rocksdb: (Original Log Time 2017/08/20-15:24:32.539794) EVENT_LOG_v1 {"time_micros": 1503242672539788, "job": 3, "event": "compaction_finished", "compaction_time_micros": 150198, "output_level": 1, "num_output_files": 1, "total_output_size": 11885974, "num_input_records": 2715, "num_output_records": 2679, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
2017-08-20 15:24:32.539980 I | rook-ceph-mon52: 2017-08-20 15:24:32.539966 7f57cc7d5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503242672539963, "job": 3, "event": "table_file_deletion", "file_number": 13}
2017-08-20 15:24:32.540086 I | rook-ceph-mon52: 2017-08-20 15:24:32.540045 7f57cc7d5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503242672540043, "job": 3, "event": "table_file_deletion", "file_number": 10}
2017-08-20 15:24:32.541713 I | rook-ceph-mon52: 2017-08-20 15:24:32.541699 7f57cc7d5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503242672541696, "job": 3, "event": "table_file_deletion", "file_number": 7}
2017-08-20 15:24:32.541746 I | rook-ceph-mon52: 2017-08-20 15:24:32.541740 7f57cc7d5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503242672541739, "job": 3, "event": "table_file_deletion", "file_number": 4}
2017-08-20 15:24:32.578278 I | rook-ceph-mon52: 2017-08-20 15:24:32.578175 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:32.578561 I | rook-ceph-mon52: 2017-08-20 15:24:32.578541 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:32.980894 I | rook-ceph-mon52: 2017-08-20 15:24:32.980747 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:32.981356 I | rook-ceph-mon52: 2017-08-20 15:24:32.981318 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:33.781445 I | rook-ceph-mon52: 2017-08-20 15:24:33.781300 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:33.781738 I | rook-ceph-mon52: 2017-08-20 15:24:33.781681 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:35.385346 I | rook-ceph-mon52: 2017-08-20 15:24:35.385227 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:35.385789 I | rook-ceph-mon52: 2017-08-20 15:24:35.385738 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:38.589031 I | rook-ceph-mon52: 2017-08-20 15:24:38.588927 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:38.589327 I | rook-ceph-mon52: 2017-08-20 15:24:38.589305 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:39.420936 I | rook-ceph-mon52: 2017-08-20 15:24:39.420883 7f57d07dd700 0 mon.rook-ceph-mon52@2(peon) e107 my rank is now 1 (was 2)
2017-08-20 15:24:39.423036 I | rook-ceph-mon52: 2017-08-20 15:24:39.423017 7f57cd7d7700 0 -- 10.3.179.50:6790/0 >> 10.3.14.3:6790/0 conn(0xce3ca5f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=50 cs=1 l=0).process missed message? skipped from seq 0 to 483073035
2017-08-20 15:24:39.423081 I | rook-ceph-mon52: 2017-08-20 15:24:39.423071 7f57d07dd700 0 log_channel(cluster) log [INF] : mon.rook-ceph-mon52 calling new monitor election
2017-08-20 15:24:39.438176 I | rook-ceph-mon52: 2017-08-20 15:24:39.438115 7f57cd7d7700 0 -- 10.3.179.50:6790/0 >> 10.3.202.111:6790/0 conn(0xce3d042800 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=1 existing_state=STATE_STANDBY
2017-08-20 15:24:39.438198 I | rook-ceph-mon52: 2017-08-20 15:24:39.438121 7f57cd7d7700 0 -- 10.3.179.50:6790/0 >> 10.3.202.111:6790/0 conn(0xce3d042800 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept peer reset, then tried to connect to us, replacing
2017-08-20 15:24:41.706258 I | rook-ceph-mon52: 2017-08-20 15:24:08.422670 mon.rook-ceph-mon54 mon.3 10.3.202.111:6790/0 2 : cluster [INF] mon.rook-ceph-mon54 calling new monitor election
2017-08-20 15:24:41.706283 I | rook-ceph-mon52: 2017-08-20 15:24:32.378195 mon.rook-ceph-mon52 mon.2 10.3.179.50:6790/0 1 : cluster [INF] mon.rook-ceph-mon52 calling new monitor election
2017-08-20 15:24:41.706285 I | rook-ceph-mon52: 2017-08-20 15:24:39.423074 mon.rook-ceph-mon52 mon.1 10.3.179.50:6790/0 2 : cluster [INF] mon.rook-ceph-mon52 calling new monitor election
2017-08-20 15:24:41.706287 I | rook-ceph-mon52: 2017-08-20 15:24:39.439252 mon.rook-ceph-mon54 mon.2 10.3.202.111:6790/0 3 : cluster [INF] mon.rook-ceph-mon54 calling new monitor election
2017-08-20 15:24:41.706289 I | rook-ceph-mon52: 2017-08-20 15:24:41.402191 mon.rook-ceph-mon53 mon.0 10.3.14.3:6790/0 46 : cluster [INF] mon.rook-ceph-mon53 calling new monitor election
2017-08-20 15:24:41.706292 I | rook-ceph-mon52: 2017-08-20 15:24:41.485849 mon.rook-ceph-mon53 mon.0 10.3.14.3:6790/0 47 : cluster [INF] mon.rook-ceph-mon53@0 won leader election with quorum 0,1,2
2017-08-20 15:24:41.706294 I | rook-ceph-mon52: 2017-08-20 15:24:41.529191 mon.rook-ceph-mon53 mon.0 10.3.14.3:6790/0 48 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum rook-ceph-mon53,rook-ceph-mon52
2017-08-20 15:24:41.706318 I | rook-ceph-mon52: 2017-08-20 15:24:41.549498 mon.rook-ceph-mon53 mon.0 10.3.14.3:6790/0 49 : cluster [INF] monmap e107: 3 mons at {rook-ceph-mon52=10.3.179.50:6790/0,rook-ceph-mon53=10.3.14.3:6790/0,rook-ceph-mon54=10.3.202.111:6790/0}
2017-08-20 15:24:42.840398 I | rook-ceph-mon52: 2017-08-20 15:24:42.276647 mon.rook-ceph-mon53 mon.0 10.3.14.3:6790/0 53 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum rook-ceph-mon53,rook-ceph-mon52)
2017-08-20 15:24:42.840410 I | rook-ceph-mon52: 2017-08-20 15:24:42.276724 mon.rook-ceph-mon53 mon.0 10.3.14.3:6790/0 54 : cluster [INF] Cluster is now healthy
2017-08-20 15:24:44.991142 I | rook-ceph-mon52: 2017-08-20 15:24:44.991034 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:44.991546 I | rook-ceph-mon52: 2017-08-20 15:24:44.991494 7f57d3fe4700 0 -- 100.64.1.33:0/24 >> 100.64.6.26:6800/18 conn(0xce3d016800 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:24:54.316908 I | rook-ceph-mon52: 2017-08-20 15:24:54.316834 7f57d07dd700 0 mon.rook-ceph-mon52@1(peon) e107 handle_command mon_command({"prefix": "osd perf", "format": "json"} v 0) v1
2017-08-20 15:24:54.316946 I | rook-ceph-mon52: 2017-08-20 15:24:54.316883 7f57d07dd700 0 log_channel(audit) log [DBG] : from='client.242235 100.64.3.35:0/472170703' entity='client.admin' cmd=[{"prefix": "osd perf", "format": "json"}]: dispatch
2017-08-20 15:24:54.513785 I | rook-ceph-mon52: 2017-08-20 15:24:32.374828 7f57dca18f40 -1 mon.rook-ceph-mon52@-1(probing).mgr e7 Failed to load mgr commands: (2) No such file or directory
failed to run mon. failed to start mon: failed to run ceph-mon: Failed to complete rook-ceph-mon52: signal: aborted (core dumped)
Raw
rook-ceph-mon62
2017-08-20 15:57:07.785080 I | cephmon: parsing mon endpoints: rook-ceph-mon59=10.3.169.122:6790,rook-ceph-mon62=10.3.149.229:6790,rook-ceph-mon58=10.3.192.19:6790
2017-08-20 15:57:07.785432 I | cephmon: writing config file /var/lib/rook/rook/rook.config
2017-08-20 15:57:07.785627 I | cephmon: generated admin config in /var/lib/rook/rook
2017-08-20 15:57:07.785774 I | cephmon: writing config file /var/lib/rook/rook-ceph-mon62/rook.config
2017-08-20 15:57:07.785926 I | cephmon: initializing mon
2017-08-20 15:57:07.785935 I | exec: Running command: monmaptool /var/lib/rook/rook-ceph-mon62/monmap --create --clobber --fsid 2c50d7fc-8533-45bd-bf42-970c435e4a3d --add rook-ceph-mon59 10.3.169.122:6790 --add rook-ceph-mon62 10.3.149.229:6790 --add rook-ceph-mon58 10.3.192.19:6790
2017-08-20 15:57:07.812393 I | monmaptool: monmap file /var/lib/rook/rook-ceph-mon62/monmap
2017-08-20 15:57:07.812410 I | monmaptool: set fsid to 2c50d7fc-8533-45bd-bf42-970c435e4a3d
2017-08-20 15:57:07.812415 I | monmaptool: writing epoch 0 to /var/lib/rook/rook-ceph-mon62/monmap (3 monitors)
2017-08-20 15:57:07.814369 I | proc: Running process ceph-mon with args: [--mkfs --name=mon.rook-ceph-mon62 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon62/data --conf=/var/lib/rook/rook-ceph-mon62/rook.config --keyring=/var/lib/rook/rook-ceph-mon62/keyring --monmap=/var/lib/rook/rook-ceph-mon62/monmap]
2017-08-20 15:57:07.814382 I | exec: Running command: ceph-mon --mkfs --name=mon.rook-ceph-mon62 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon62/data --conf=/var/lib/rook/rook-ceph-mon62/rook.config --keyring=/var/lib/rook/rook-ceph-mon62/keyring --monmap=/var/lib/rook/rook-ceph-mon62/monmap
2017-08-20 15:57:07.833005 I | mkfs-rook-ceph-mon62: 2017-08-20 15:57:07.832849 7fb0cd174f40 -1 '/var/lib/rook/rook-ceph-mon62/data' already exists and is not empty: monitor may already exist
2017-08-20 15:57:07.834474 I | mkfs-rook-ceph-mon62: 2017-08-20 15:57:07.832849 7fb0cd174f40 -1 '/var/lib/rook/rook-ceph-mon62/data' already exists and is not empty: monitor may already exist
2017-08-20 15:57:07.834527 I | cephmon: starting mon
2017-08-20 15:57:07.834560 I | cephmon: Config file /var/lib/rook/rook-ceph-mon62/rook.config:
[global]
enable experimental unrecoverable data corrupting features =
fsid = 2c50d7fc-8533-45bd-bf42-970c435e4a3d
run dir = /var/lib/rook/rook-ceph-mon62
mon initial members = rook-ceph-mon62 rook-ceph-mon58 rook-ceph-mon59
mon host = 10.3.149.229:6790,10.3.192.19:6790,10.3.169.122:6790
log file = /dev/stdout
mon cluster log file = /dev/stdout
public addr = 10.3.149.229
cluster addr = 100.64.2.65
mon keyvaluedb = rocksdb
debug default = 0
debug rados = 0
debug mon = 0
debug osd = 0
debug bluestore = 0
debug filestore = 0
debug journal = 0
debug leveldb = 0
filestore_omap_backend = rocksdb
osd pg bits = 11
osd pgp bits = 11
osd pool default size = 1
osd pool default min size = 1
osd pool default pg num = 100
osd pool default pgp num = 100
osd objectstore = filestore
rbd_default_features = 3
fatal signal handlers = false
[client.admin]
keyring = /var/lib/rook/rook-ceph-mon62/keyring
2017-08-20 15:57:07.834589 I | proc: Running process ceph-mon with args: [--foreground --name=mon.rook-ceph-mon62 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon62/data --conf=/var/lib/rook/rook-ceph-mon62/rook.config --keyring=/var/lib/rook/rook-ceph-mon62/keyring --public-addr=10.3.149.229:6790 --public-bind-addr=100.64.2.65:6790]
2017-08-20 15:57:07.834595 I | exec: Running command: ceph-mon --foreground --name=mon.rook-ceph-mon62 --cluster=rook --mon-data=/var/lib/rook/rook-ceph-mon62/data --conf=/var/lib/rook/rook-ceph-mon62/rook.config --keyring=/var/lib/rook/rook-ceph-mon62/keyring --public-addr=10.3.149.229:6790 --public-bind-addr=100.64.2.65:6790
2017-08-20 15:57:07.851751 I | rook-ceph-mon62: 2017-08-20 15:57:07.851679 7f21039e8f40 0 ceph version ad3a8b682 (ead3a8b682051c4ab5adf80663c66d4521366930) luminous (rc), process (unknown), pid 22
2017-08-20 15:57:07.851766 I | rook-ceph-mon62: 2017-08-20 15:57:07.851750 7f21039e8f40 0 pidfile_write: ignore empty --pid-file
2017-08-20 15:57:07.857390 I | rook-ceph-mon62: 2017-08-20 15:57:07.857341 7f21039e8f40 0 load: jerasure load: lrc load: isa
2017-08-20 15:57:07.857496 I | rook-ceph-mon62: 2017-08-20 15:57:07.857421 7f21039e8f40 0 set rocksdb option compression = kNoCompression
2017-08-20 15:57:07.857503 I | rook-ceph-mon62: 2017-08-20 15:57:07.857428 7f21039e8f40 0 set rocksdb option write_buffer_size = 33554432
2017-08-20 15:57:07.857506 I | rook-ceph-mon62: 2017-08-20 15:57:07.857471 7f21039e8f40 0 set rocksdb option compression = kNoCompression
2017-08-20 15:57:07.857508 I | rook-ceph-mon62: 2017-08-20 15:57:07.857490 7f21039e8f40 0 set rocksdb option write_buffer_size = 33554432
2017-08-20 15:57:07.857650 I | rook-ceph-mon62: 2017-08-20 15:57:07.857590 7f21039e8f40 4 rocksdb: RocksDB version: 5.4.0
2017-08-20 15:57:07.857656 I | rook-ceph-mon62:
2017-08-20 15:57:07.857661 I | rook-ceph-mon62: 2017-08-20 15:57:07.857594 7f21039e8f40 4 rocksdb: Git sha rocksdb_build_git_sha:@e15382c09c87a65eaeca9bda233bab503f1e5772@
2017-08-20 15:57:07.857664 I | rook-ceph-mon62: 2017-08-20 15:57:07.857595 7f21039e8f40 4 rocksdb: Compile date Aug 11 2017
2017-08-20 15:57:07.857687 I | rook-ceph-mon62: 2017-08-20 15:57:07.857596 7f21039e8f40 4 rocksdb: DB SUMMARY
2017-08-20 15:57:07.857689 I | rook-ceph-mon62:
2017-08-20 15:57:07.857693 I | rook-ceph-mon62: 2017-08-20 15:57:07.857650 7f21039e8f40 4 rocksdb: CURRENT file: CURRENT
2017-08-20 15:57:07.857697 I | rook-ceph-mon62:
2017-08-20 15:57:07.857699 I | rook-ceph-mon62: 2017-08-20 15:57:07.857653 7f21039e8f40 4 rocksdb: IDENTITY file: IDENTITY
2017-08-20 15:57:07.857701 I | rook-ceph-mon62:
2017-08-20 15:57:07.857704 I | rook-ceph-mon62: 2017-08-20 15:57:07.857657 7f21039e8f40 4 rocksdb: MANIFEST file: MANIFEST-000008 size: 166 Bytes
2017-08-20 15:57:07.857706 I | rook-ceph-mon62:
2017-08-20 15:57:07.857709 I | rook-ceph-mon62: 2017-08-20 15:57:07.857660 7f21039e8f40 4 rocksdb: SST files in /var/lib/rook/rook-ceph-mon62/data/store.db dir, Total Num: 2, files: 000004.sst 000007.sst
2017-08-20 15:57:07.857710 I | rook-ceph-mon62:
2017-08-20 15:57:07.857726 I | rook-ceph-mon62: 2017-08-20 15:57:07.857661 7f21039e8f40 4 rocksdb: Write Ahead Log file in /var/lib/rook/rook-ceph-mon62/data/store.db: 000009.log size: 83174 ;
2017-08-20 15:57:07.857728 I | rook-ceph-mon62:
2017-08-20 15:57:07.857730 I | rook-ceph-mon62: 2017-08-20 15:57:07.857663 7f21039e8f40 4 rocksdb: Options.error_if_exists: 0
2017-08-20 15:57:07.857733 I | rook-ceph-mon62: 2017-08-20 15:57:07.857663 7f21039e8f40 4 rocksdb: Options.create_if_missing: 0
2017-08-20 15:57:07.857735 I | rook-ceph-mon62: 2017-08-20 15:57:07.857664 7f21039e8f40 4 rocksdb: Options.paranoid_checks: 1
2017-08-20 15:57:07.857738 I | rook-ceph-mon62: 2017-08-20 15:57:07.857665 7f21039e8f40 4 rocksdb: Options.env: 0xaf84851560
2017-08-20 15:57:07.857739 I | rook-ceph-mon62: 2017-08-20 15:57:07.857665 7f21039e8f40 4 rocksdb: Options.info_log: 0xaf851fce20
2017-08-20 15:57:07.857744 I | rook-ceph-mon62: 2017-08-20 15:57:07.857666 7f21039e8f40 4 rocksdb: Options.max_open_files: -1
2017-08-20 15:57:07.857746 I | rook-ceph-mon62: 2017-08-20 15:57:07.857667 7f21039e8f40 4 rocksdb: Options.max_file_opening_threads: 16
2017-08-20 15:57:07.857748 I | rook-ceph-mon62: 2017-08-20 15:57:07.857668 7f21039e8f40 4 rocksdb: Options.use_fsync: 0
2017-08-20 15:57:07.857751 I | rook-ceph-mon62: 2017-08-20 15:57:07.857668 7f21039e8f40 4 rocksdb: Options.max_log_file_size: 0
2017-08-20 15:57:07.857755 I | rook-ceph-mon62: 2017-08-20 15:57:07.857682 7f21039e8f40 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2017-08-20 15:57:07.857759 I | rook-ceph-mon62: 2017-08-20 15:57:07.857683 7f21039e8f40 4 rocksdb: Options.log_file_time_to_roll: 0
2017-08-20 15:57:07.857762 I | rook-ceph-mon62: 2017-08-20 15:57:07.857684 7f21039e8f40 4 rocksdb: Options.keep_log_file_num: 1000
2017-08-20 15:57:07.857764 I | rook-ceph-mon62: 2017-08-20 15:57:07.857684 7f21039e8f40 4 rocksdb: Options.recycle_log_file_num: 0
2017-08-20 15:57:07.857766 I | rook-ceph-mon62: 2017-08-20 15:57:07.857685 7f21039e8f40 4 rocksdb: Options.allow_fallocate: 1
2017-08-20 15:57:07.857768 I | rook-ceph-mon62: 2017-08-20 15:57:07.857686 7f21039e8f40 4 rocksdb: Options.allow_mmap_reads: 0
2017-08-20 15:57:07.857769 I | rook-ceph-mon62: 2017-08-20 15:57:07.857687 7f21039e8f40 4 rocksdb: Options.allow_mmap_writes: 0
2017-08-20 15:57:07.857772 I | rook-ceph-mon62: 2017-08-20 15:57:07.857688 7f21039e8f40 4 rocksdb: Options.use_direct_reads: 0
2017-08-20 15:57:07.857774 I | rook-ceph-mon62: 2017-08-20 15:57:07.857688 7f21039e8f40 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
2017-08-20 15:57:07.857777 I | rook-ceph-mon62: 2017-08-20 15:57:07.857690 7f21039e8f40 4 rocksdb: Options.create_missing_column_families: 0
2017-08-20 15:57:07.857779 I | rook-ceph-mon62: 2017-08-20 15:57:07.857690 7f21039e8f40 4 rocksdb: Options.db_log_dir:
2017-08-20 15:57:07.857783 I | rook-ceph-mon62: 2017-08-20 15:57:07.857691 7f21039e8f40 4 rocksdb: Options.wal_dir: /var/lib/rook/rook-ceph-mon62/data/store.db
2017-08-20 15:57:07.857786 I | rook-ceph-mon62: 2017-08-20 15:57:07.857692 7f21039e8f40 4 rocksdb: Options.table_cache_numshardbits: 6
2017-08-20 15:57:07.857787 I | rook-ceph-mon62: 2017-08-20 15:57:07.857693 7f21039e8f40 4 rocksdb: Options.max_subcompactions: 1
2017-08-20 15:57:07.857789 I | rook-ceph-mon62: 2017-08-20 15:57:07.857693 7f21039e8f40 4 rocksdb: Options.max_background_flushes: 1
2017-08-20 15:57:07.857794 I | rook-ceph-mon62: 2017-08-20 15:57:07.857695 7f21039e8f40 4 rocksdb: Options.WAL_ttl_seconds: 0
2017-08-20 15:57:07.857796 I | rook-ceph-mon62: 2017-08-20 15:57:07.857695 7f21039e8f40 4 rocksdb: Options.WAL_size_limit_MB: 0
2017-08-20 15:57:07.857798 I | rook-ceph-mon62: 2017-08-20 15:57:07.857696 7f21039e8f40 4 rocksdb: Options.manifest_preallocation_size: 4194304
2017-08-20 15:57:07.857800 I | rook-ceph-mon62: 2017-08-20 15:57:07.857697 7f21039e8f40 4 rocksdb: Options.is_fd_close_on_exec: 1
2017-08-20 15:57:07.857802 I | rook-ceph-mon62: 2017-08-20 15:57:07.857697 7f21039e8f40 4 rocksdb: Options.advise_random_on_open: 1
2017-08-20 15:57:07.857804 I | rook-ceph-mon62: 2017-08-20 15:57:07.857703 7f21039e8f40 4 rocksdb: Options.db_write_buffer_size: 0
2017-08-20 15:57:07.857806 I | rook-ceph-mon62: 2017-08-20 15:57:07.857704 7f21039e8f40 4 rocksdb: Options.access_hint_on_compaction_start: 1
2017-08-20 15:57:07.857808 I | rook-ceph-mon62: 2017-08-20 15:57:07.857704 7f21039e8f40 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2017-08-20 15:57:07.857810 I | rook-ceph-mon62: 2017-08-20 15:57:07.857705 7f21039e8f40 4 rocksdb: Options.compaction_readahead_size: 0
2017-08-20 15:57:07.857812 I | rook-ceph-mon62: 2017-08-20 15:57:07.857705 7f21039e8f40 4 rocksdb: Options.random_access_max_buffer_size: 1048576
2017-08-20 15:57:07.857814 I | rook-ceph-mon62: 2017-08-20 15:57:07.857706 7f21039e8f40 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
2017-08-20 15:57:07.857816 I | rook-ceph-mon62: 2017-08-20 15:57:07.857707 7f21039e8f40 4 rocksdb: Options.use_adaptive_mutex: 0
2017-08-20 15:57:07.857819 I | rook-ceph-mon62: 2017-08-20 15:57:07.857708 7f21039e8f40 4 rocksdb: Options.rate_limiter: (nil)
2017-08-20 15:57:07.857821 I | rook-ceph-mon62: 2017-08-20 15:57:07.857708 7f21039e8f40 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2017-08-20 15:57:07.857822 I | rook-ceph-mon62: 2017-08-20 15:57:07.857709 7f21039e8f40 4 rocksdb: Options.bytes_per_sync: 0
2017-08-20 15:57:07.857825 I | rook-ceph-mon62: 2017-08-20 15:57:07.857710 7f21039e8f40 4 rocksdb: Options.wal_bytes_per_sync: 0
2017-08-20 15:57:07.857826 I | rook-ceph-mon62: 2017-08-20 15:57:07.857710 7f21039e8f40 4 rocksdb: Options.wal_recovery_mode: 2
2017-08-20 15:57:07.857828 I | rook-ceph-mon62: 2017-08-20 15:57:07.857711 7f21039e8f40 4 rocksdb: Options.enable_thread_tracking: 0
2017-08-20 15:57:07.857830 I | rook-ceph-mon62: 2017-08-20 15:57:07.857725 7f21039e8f40 4 rocksdb: Options.allow_concurrent_memtable_write: 1
2017-08-20 15:57:07.857832 I | rook-ceph-mon62: 2017-08-20 15:57:07.857726 7f21039e8f40 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2017-08-20 15:57:07.857835 I | rook-ceph-mon62: 2017-08-20 15:57:07.857727 7f21039e8f40 4 rocksdb: Options.write_thread_max_yield_usec: 100
2017-08-20 15:57:07.857836 I | rook-ceph-mon62: 2017-08-20 15:57:07.857727 7f21039e8f40 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2017-08-20 15:57:07.857839 I | rook-ceph-mon62: 2017-08-20 15:57:07.857728 7f21039e8f40 4 rocksdb: Options.row_cache: None
2017-08-20 15:57:07.857842 I | rook-ceph-mon62: 2017-08-20 15:57:07.857729 7f21039e8f40 4 rocksdb: Options.wal_filter: None
2017-08-20 15:57:07.857844 I | rook-ceph-mon62: 2017-08-20 15:57:07.857729 7f21039e8f40 4 rocksdb: Options.avoid_flush_during_recovery: 0
2017-08-20 15:57:07.857848 I | rook-ceph-mon62: 2017-08-20 15:57:07.857731 7f21039e8f40 4 rocksdb: Options.base_background_compactions: 1
2017-08-20 15:57:07.857851 I | rook-ceph-mon62: 2017-08-20 15:57:07.857731 7f21039e8f40 4 rocksdb: Options.max_background_compactions: 1
2017-08-20 15:57:07.857859 I | rook-ceph-mon62: 2017-08-20 15:57:07.857732 7f21039e8f40 4 rocksdb: Options.avoid_flush_during_shutdown: 0
2017-08-20 15:57:07.857864 I | rook-ceph-mon62: 2017-08-20 15:57:07.857732 7f21039e8f40 4 rocksdb: Options.delayed_write_rate : 16777216
2017-08-20 15:57:07.857866 I | rook-ceph-mon62: 2017-08-20 15:57:07.857733 7f21039e8f40 4 rocksdb: Options.max_total_wal_size: 0
2017-08-20 15:57:07.857868 I | rook-ceph-mon62: 2017-08-20 15:57:07.857734 7f21039e8f40 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2017-08-20 15:57:07.857870 I | rook-ceph-mon62: 2017-08-20 15:57:07.857735 7f21039e8f40 4 rocksdb: Options.stats_dump_period_sec: 600
2017-08-20 15:57:07.857872 I | rook-ceph-mon62: 2017-08-20 15:57:07.857735 7f21039e8f40 4 rocksdb: Compression algorithms supported:
2017-08-20 15:57:07.857874 I | rook-ceph-mon62: 2017-08-20 15:57:07.857736 7f21039e8f40 4 rocksdb: Snappy supported: 0
2017-08-20 15:57:07.857877 I | rook-ceph-mon62: 2017-08-20 15:57:07.857737 7f21039e8f40 4 rocksdb: Zlib supported: 0
2017-08-20 15:57:07.857878 I | rook-ceph-mon62: 2017-08-20 15:57:07.857738 7f21039e8f40 4 rocksdb: Bzip supported: 0
2017-08-20 15:57:07.857881 I | rook-ceph-mon62: 2017-08-20 15:57:07.857738 7f21039e8f40 4 rocksdb: LZ4 supported: 0
2017-08-20 15:57:07.857882 I | rook-ceph-mon62: 2017-08-20 15:57:07.857739 7f21039e8f40 4 rocksdb: ZSTD supported: 0
2017-08-20 15:57:07.857885 I | rook-ceph-mon62: 2017-08-20 15:57:07.857740 7f21039e8f40 4 rocksdb: Fast CRC32 supported: 1
2017-08-20 15:57:07.865779 I | rook-ceph-mon62: 2017-08-20 15:57:07.865686 7f21039e8f40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000008
2017-08-20 15:57:07.865790 I | rook-ceph-mon62:
2017-08-20 15:57:07.865992 I | rook-ceph-mon62: 2017-08-20 15:57:07.865884 7f21039e8f40 4 rocksdb: [/build/ceph/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
2017-08-20 15:57:07.865996 I | rook-ceph-mon62:
2017-08-20 15:57:07.865998 I | rook-ceph-mon62: 2017-08-20 15:57:07.865892 7f21039e8f40 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2017-08-20 15:57:07.866000 I | rook-ceph-mon62: 2017-08-20 15:57:07.865893 7f21039e8f40 4 rocksdb: Options.merge_operator:
2017-08-20 15:57:07.866002 I | rook-ceph-mon62: 2017-08-20 15:57:07.865894 7f21039e8f40 4 rocksdb: Options.compaction_filter: None
2017-08-20 15:57:07.866005 I | rook-ceph-mon62: 2017-08-20 15:57:07.865895 7f21039e8f40 4 rocksdb: Options.compaction_filter_factory: None
2017-08-20 15:57:07.866009 I | rook-ceph-mon62: 2017-08-20 15:57:07.865895 7f21039e8f40 4 rocksdb: Options.memtable_factory: SkipListFactory
2017-08-20 15:57:07.866012 I | rook-ceph-mon62: 2017-08-20 15:57:07.865896 7f21039e8f40 4 rocksdb: Options.table_factory: BlockBasedTable
2017-08-20 15:57:07.866015 I | rook-ceph-mon62: 2017-08-20 15:57:07.865938 7f21039e8f40 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0xaf8517e138)
2017-08-20 15:57:07.866017 I | rook-ceph-mon62: cache_index_and_filter_blocks: 1
2017-08-20 15:57:07.866019 I | rook-ceph-mon62: cache_index_and_filter_blocks_with_high_priority: 1
2017-08-20 15:57:07.866022 I | rook-ceph-mon62: pin_l0_filter_and_index_blocks_in_cache: 1
2017-08-20 15:57:07.866024 I | rook-ceph-mon62: index_type: 0
2017-08-20 15:57:07.866026 I | rook-ceph-mon62: hash_index_allow_collision: 1
2017-08-20 15:57:07.866028 I | rook-ceph-mon62: checksum: 1
2017-08-20 15:57:07.866029 I | rook-ceph-mon62: no_block_cache: 0
2017-08-20 15:57:07.866048 I | rook-ceph-mon62: block_cache: 0xaf8547f1f0
2017-08-20 15:57:07.866050 I | rook-ceph-mon62: block_cache_name: LRUCache
2017-08-20 15:57:07.866052 I | rook-ceph-mon62: block_cache_options:
2017-08-20 15:57:07.866054 I | rook-ceph-mon62: capacity : 134217728
2017-08-20 15:57:07.866056 I | rook-ceph-mon62: num_shard_bits : 4
2017-08-20 15:57:07.866058 I | rook-ceph-mon62: strict_capacity_limit : 0
2017-08-20 15:57:07.866060 I | rook-ceph-mon62: high_pri_pool_ratio: 0.000
2017-08-20 15:57:07.866061 I | rook-ceph-mon62: block_cache_compressed: (nil)
2017-08-20 15:57:07.866063 I | rook-ceph-mon62: persistent_cache: (nil)
2017-08-20 15:57:07.866066 I | rook-ceph-mon62: block_size: 4096
2017-08-20 15:57:07.866067 I | rook-ceph-mon62: block_size_deviation: 10
2017-08-20 15:57:07.866070 I | rook-ceph-mon62: block_restart_interval: 16
2017-08-20 15:57:07.866073 I | rook-ceph-mon62: index_block_restart_interval: 1
2017-08-20 15:57:07.866077 I | rook-ceph-mon62: filter_policy: rocksdb.BuiltinBloomFilter
2017-08-20 15:57:07.866081 I | rook-ceph-mon62: whole_key_filtering: 1
2017-08-20 15:57:07.866085 I | rook-ceph-mon62: format_version: 2
2017-08-20 15:57:07.866087 I | rook-ceph-mon62:
2017-08-20 15:57:07.866089 I | rook-ceph-mon62: 2017-08-20 15:57:07.865941 7f21039e8f40 4 rocksdb: Options.write_buffer_size: 33554432
2017-08-20 15:57:07.866091 I | rook-ceph-mon62: 2017-08-20 15:57:07.865942 7f21039e8f40 4 rocksdb: Options.max_write_buffer_number: 2
2017-08-20 15:57:07.866093 I | rook-ceph-mon62: 2017-08-20 15:57:07.865944 7f21039e8f40 4 rocksdb: Options.compression: NoCompression
2017-08-20 15:57:07.866097 I | rook-ceph-mon62: 2017-08-20 15:57:07.865965 7f21039e8f40 4 rocksdb: Options.bottommost_compression: Disabled
2017-08-20 15:57:07.866100 I | rook-ceph-mon62: 2017-08-20 15:57:07.865967 7f21039e8f40 4 rocksdb: Options.prefix_extractor: nullptr
2017-08-20 15:57:07.866103 I | rook-ceph-mon62: 2017-08-20 15:57:07.865968 7f21039e8f40 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2017-08-20 15:57:07.866105 I | rook-ceph-mon62: 2017-08-20 15:57:07.865969 7f21039e8f40 4 rocksdb: Options.num_levels: 7
2017-08-20 15:57:07.866107 I | rook-ceph-mon62: 2017-08-20 15:57:07.865970 7f21039e8f40 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2017-08-20 15:57:07.866108 I | rook-ceph-mon62: 2017-08-20 15:57:07.865970 7f21039e8f40 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2017-08-20 15:57:07.866110 I | rook-ceph-mon62: 2017-08-20 15:57:07.865972 7f21039e8f40 4 rocksdb: Options.compression_opts.window_bits: -14
2017-08-20 15:57:07.866112 I | rook-ceph-mon62: 2017-08-20 15:57:07.865972 7f21039e8f40 4 rocksdb: Options.compression_opts.level: -1
2017-08-20 15:57:07.866115 I | rook-ceph-mon62: 2017-08-20 15:57:07.865986 7f21039e8f40 4 rocksdb: Options.compression_opts.strategy: 0
2017-08-20 15:57:07.866116 I | rook-ceph-mon62: 2017-08-20 15:57:07.865987 7f21039e8f40 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2017-08-20 15:57:07.866118 I | rook-ceph-mon62: 2017-08-20 15:57:07.865988 7f21039e8f40 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2017-08-20 15:57:07.866120 I | rook-ceph-mon62: 2017-08-20 15:57:07.865988 7f21039e8f40 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2017-08-20 15:57:07.866121 I | rook-ceph-mon62: 2017-08-20 15:57:07.865989 7f21039e8f40 4 rocksdb: Options.level0_stop_writes_trigger: 36
2017-08-20 15:57:07.866123 I | rook-ceph-mon62: 2017-08-20 15:57:07.865990 7f21039e8f40 4 rocksdb: Options.target_file_size_base: 67108864
2017-08-20 15:57:07.866127 I | rook-ceph-mon62: 2017-08-20 15:57:07.865991 7f21039e8f40 4 rocksdb: Options.target_file_size_multiplier: 1
2017-08-20 15:57:07.866129 I | rook-ceph-mon62: 2017-08-20 15:57:07.865992 7f21039e8f40 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2017-08-20 15:57:07.866134 I | rook-ceph-mon62: 2017-08-20 15:57:07.865992 7f21039e8f40 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2017-08-20 15:57:07.866136 I | rook-ceph-mon62: 2017-08-20 15:57:07.865993 7f21039e8f40 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2017-08-20 15:57:07.866138 I | rook-ceph-mon62: 2017-08-20 15:57:07.865995 7f21039e8f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2017-08-20 15:57:07.866139 I | rook-ceph-mon62: 2017-08-20 15:57:07.865996 7f21039e8f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2017-08-20 15:57:07.866141 I | rook-ceph-mon62: 2017-08-20 15:57:07.865997 7f21039e8f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2017-08-20 15:57:07.866143 I | rook-ceph-mon62: 2017-08-20 15:57:07.865997 7f21039e8f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2017-08-20 15:57:07.866146 I | rook-ceph-mon62: 2017-08-20 15:57:07.865998 7f21039e8f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2017-08-20 15:57:07.866148 I | rook-ceph-mon62: 2017-08-20 15:57:07.865999 7f21039e8f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2017-08-20 15:57:07.866149 I | rook-ceph-mon62: 2017-08-20 15:57:07.865999 7f21039e8f40 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2017-08-20 15:57:07.866152 I | rook-ceph-mon62: 2017-08-20 15:57:07.866000 7f21039e8f40 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2017-08-20 15:57:07.866154 I | rook-ceph-mon62: 2017-08-20 15:57:07.866001 7f21039e8f40 4 rocksdb: Options.max_compaction_bytes: 1677721600
2017-08-20 15:57:07.866156 I | rook-ceph-mon62: 2017-08-20 15:57:07.866002 7f21039e8f40 4 rocksdb: Options.arena_block_size: 4194304
2017-08-20 15:57:07.866157 I | rook-ceph-mon62: 2017-08-20 15:57:07.866002 7f21039e8f40 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2017-08-20 15:57:07.866159 I | rook-ceph-mon62: 2017-08-20 15:57:07.866003 7f21039e8f40 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2017-08-20 15:57:07.866165 I | rook-ceph-mon62: 2017-08-20 15:57:07.866004 7f21039e8f40 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2017-08-20 15:57:07.866169 I | rook-ceph-mon62: 2017-08-20 15:57:07.866004 7f21039e8f40 4 rocksdb: Options.disable_auto_compactions: 0
2017-08-20 15:57:07.866174 I | rook-ceph-mon62: 2017-08-20 15:57:07.866006 7f21039e8f40 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2017-08-20 15:57:07.866178 I | rook-ceph-mon62: 2017-08-20 15:57:07.866006 7f21039e8f40 4 rocksdb: Options.compaction_pri: kByCompensatedSize
2017-08-20 15:57:07.866182 I | rook-ceph-mon62: 2017-08-20 15:57:07.866007 7f21039e8f40 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2017-08-20 15:57:07.866185 I | rook-ceph-mon62: 2017-08-20 15:57:07.866008 7f21039e8f40 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2017-08-20 15:57:07.866187 I | rook-ceph-mon62: 2017-08-20 15:57:07.866009 7f21039e8f40 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2017-08-20 15:57:07.866191 I | rook-ceph-mon62: 2017-08-20 15:57:07.866009 7f21039e8f40 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2017-08-20 15:57:07.866196 I | rook-ceph-mon62: 2017-08-20 15:57:07.866010 7f21039e8f40 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2017-08-20 15:57:07.866199 I | rook-ceph-mon62: 2017-08-20 15:57:07.866011 7f21039e8f40 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2017-08-20 15:57:07.866204 I | rook-ceph-mon62: 2017-08-20 15:57:07.866011 7f21039e8f40 4 rocksdb: Options.table_properties_collectors:
2017-08-20 15:57:07.866208 I | rook-ceph-mon62: 2017-08-20 15:57:07.866012 7f21039e8f40 4 rocksdb: Options.inplace_update_support: 0
2017-08-20 15:57:07.866212 I | rook-ceph-mon62: 2017-08-20 15:57:07.866013 7f21039e8f40 4 rocksdb: Options.inplace_update_num_locks: 10000
2017-08-20 15:57:07.866216 I | rook-ceph-mon62: 2017-08-20 15:57:07.866014 7f21039e8f40 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2017-08-20 15:57:07.866221 I | rook-ceph-mon62: 2017-08-20 15:57:07.866015 7f21039e8f40 4 rocksdb: Options.memtable_huge_page_size: 0
2017-08-20 15:57:07.866225 I | rook-ceph-mon62: 2017-08-20 15:57:07.866015 7f21039e8f40 4 rocksdb: Options.bloom_locality: 0
2017-08-20 15:57:07.866229 I | rook-ceph-mon62: 2017-08-20 15:57:07.866016 7f21039e8f40 4 rocksdb: Options.max_successive_merges: 0
2017-08-20 15:57:07.866233 I | rook-ceph-mon62: 2017-08-20 15:57:07.866017 7f21039e8f40 4 rocksdb: Options.optimize_filters_for_hits: 0
2017-08-20 15:57:07.866240 I | rook-ceph-mon62: 2017-08-20 15:57:07.866017 7f21039e8f40 4 rocksdb: Options.paranoid_file_checks: 0
2017-08-20 15:57:07.866244 I | rook-ceph-mon62: 2017-08-20 15:57:07.866018 7f21039e8f40 4 rocksdb: Options.force_consistency_checks: 0
2017-08-20 15:57:07.866248 I | rook-ceph-mon62: 2017-08-20 15:57:07.866018 7f21039e8f40 4 rocksdb: Options.report_bg_io_stats: 0
2017-08-20 15:57:07.866511 I | rook-ceph-mon62: 2017-08-20 15:57:07.866469 7f21039e8f40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/rook/rook-ceph-mon62/data/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 10, last_sequence is 2889, log_number is 0,prev_log_number is 0,max_column_family is 0
2017-08-20 15:57:07.866518 I | rook-ceph-mon62:
2017-08-20 15:57:07.866521 I | rook-ceph-mon62: 2017-08-20 15:57:07.866476 7f21039e8f40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 7
2017-08-20 15:57:07.866523 I | rook-ceph-mon62:
2017-08-20 15:57:07.866580 I | rook-ceph-mon62: 2017-08-20 15:57:07.866532 7f21039e8f40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503244627866528, "job": 1, "event": "recovery_started", "log_files": [9]}
2017-08-20 15:57:07.866586 I | rook-ceph-mon62: 2017-08-20 15:57:07.866536 7f21039e8f40 4 rocksdb: [/build/ceph/src/rocksdb/db/db_impl_open.cc:482] Recovering log #9 mode 2
2017-08-20 15:57:07.907155 I | rook-ceph-mon62: 2017-08-20 15:57:07.907101 7f21039e8f40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503244627907093, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 10, "file_size": 83899, "table_properties": {"data_size": 82341, "index_size": 341, "filter_size": 300, "raw_key_size": 729, "raw_average_key_size": 22, "raw_value_size": 81482, "raw_average_value_size": 2546, "num_data_blocks": 11, "num_entries": 32, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2017-08-20 15:57:07.907194 I | rook-ceph-mon62: 2017-08-20 15:57:07.907178 7f21039e8f40 4 rocksdb: [/build/ceph/src/rocksdb/db/version_set.cc:2395] Creating manifest 11
2017-08-20 15:57:07.907199 I | rook-ceph-mon62:
2017-08-20 15:57:07.948921 I | rook-ceph-mon62: 2017-08-20 15:57:07.948847 7f21039e8f40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503244627948843, "job": 1, "event": "recovery_finished"}
2017-08-20 15:57:07.992232 I | rook-ceph-mon62: 2017-08-20 15:57:07.992147 7f21039e8f40 4 rocksdb: [/build/ceph/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0xaf85582000
2017-08-20 15:57:07.992829 I | rook-ceph-mon62: 2017-08-20 15:57:07.992792 7f21039e8f40 0 starting mon.rook-ceph-mon62 rank 0 at public addr 10.3.149.229:6790/0 at bind addr 100.64.2.65:6790/0 mon_data /var/lib/rook/rook-ceph-mon62/data fsid 2c50d7fc-8533-45bd-bf42-970c435e4a3d
2017-08-20 15:57:07.992953 I | rook-ceph-mon62: 2017-08-20 15:57:07.992940 7f21039e8f40 0 starting mon.rook-ceph-mon62 rank 0 at 10.3.149.229:6790/0 mon_data /var/lib/rook/rook-ceph-mon62/data fsid 2c50d7fc-8533-45bd-bf42-970c435e4a3d
2017-08-20 15:57:07.993334 I | rook-ceph-mon62: 2017-08-20 15:57:07.993320 7f21039e8f40 1 mon.rook-ceph-mon62@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
2017-08-20 15:57:07.993408 I | rook-ceph-mon62: 2017-08-20 15:57:07.993369 7f21039e8f40 0 mon.rook-ceph-mon62@-1(probing).mds e33 print_map
2017-08-20 15:57:07.993412 I | rook-ceph-mon62: e33
2017-08-20 15:57:07.993414 I | rook-ceph-mon62: enable_multiple, ever_enabled_multiple: 0,0
2017-08-20 15:57:07.993417 I | rook-ceph-mon62: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
2017-08-20 15:57:07.993420 I | rook-ceph-mon62: legacy client fscid: 1
2017-08-20 15:57:07.993421 I | rook-ceph-mon62:
2017-08-20 15:57:07.993424 I | rook-ceph-mon62: Filesystem 'cfs-r2-1' (1)
2017-08-20 15:57:07.993425 I | rook-ceph-mon62: fs_name cfs-r2-1
2017-08-20 15:57:07.993427 I | rook-ceph-mon62: epoch 33
2017-08-20 15:57:07.993431 I | rook-ceph-mon62: flags c
2017-08-20 15:57:07.993433 I | rook-ceph-mon62: created 2017-08-18 09:35:11.962291
2017-08-20 15:57:07.993435 I | rook-ceph-mon62: modified 2017-08-18 09:35:11.962291
2017-08-20 15:57:07.993437 I | rook-ceph-mon62: tableserver 0
2017-08-20 15:57:07.993439 I | rook-ceph-mon62: root 0
2017-08-20 15:57:07.993442 I | rook-ceph-mon62: session_timeout 60
2017-08-20 15:57:07.993457 I | rook-ceph-mon62: session_autoclose 300
2017-08-20 15:57:07.993461 I | rook-ceph-mon62: max_file_size 1099511627776
2017-08-20 15:57:07.993463 I | rook-ceph-mon62: last_failure 0
2017-08-20 15:57:07.993465 I | rook-ceph-mon62: last_failure_osd_epoch 174
2017-08-20 15:57:07.993467 I | rook-ceph-mon62: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
2017-08-20 15:57:07.993470 I | rook-ceph-mon62: max_mds 1
2017-08-20 15:57:07.993474 I | rook-ceph-mon62: in 0
2017-08-20 15:57:07.993476 I | rook-ceph-mon62: up {0=228660}
2017-08-20 15:57:07.993478 I | rook-ceph-mon62: failed
2017-08-20 15:57:07.993481 I | rook-ceph-mon62: damaged
2017-08-20 15:57:07.993484 I | rook-ceph-mon62: stopped
2017-08-20 15:57:07.993487 I | rook-ceph-mon62: data_pools [3]
2017-08-20 15:57:07.993489 I | rook-ceph-mon62: metadata_pool 4
2017-08-20 15:57:07.993492 I | rook-ceph-mon62: inline_data disabled
2017-08-20 15:57:07.993493 I | rook-ceph-mon62: balancer
2017-08-20 15:57:07.993496 I | rook-ceph-mon62: standby_count_wanted 0
2017-08-20 15:57:07.993498 I | rook-ceph-mon62: 228660: 100.64.0.48:6800/3988930595 'mds1' mds.0.17 up:active seq 22380
2017-08-20 15:57:07.993500 I | rook-ceph-mon62:
2017-08-20 15:57:07.993502 I | rook-ceph-mon62:
2017-08-20 15:57:07.993504 I | rook-ceph-mon62:
2017-08-20 15:57:07.993533 I | rook-ceph-mon62: 2017-08-20 15:57:07.993520 7f21039e8f40 0 mon.rook-ceph-mon62@-1(probing).osd e175 crush map has features 2200130813952, adjusting msgr requires
2017-08-20 15:57:07.993536 I | rook-ceph-mon62: 2017-08-20 15:57:07.993524 7f21039e8f40 0 mon.rook-ceph-mon62@-1(probing).osd e175 crush map has features 2200130813952, adjusting msgr requires
2017-08-20 15:57:07.993538 I | rook-ceph-mon62: 2017-08-20 15:57:07.993526 7f21039e8f40 0 mon.rook-ceph-mon62@-1(probing).osd e175 crush map has features 720578140510109696, adjusting msgr requires
2017-08-20 15:57:07.993540 I | rook-ceph-mon62: 2017-08-20 15:57:07.993528 7f21039e8f40 0 mon.rook-ceph-mon62@-1(probing).osd e175 crush map has features 2200130813952, adjusting msgr requires
2017-08-20 15:57:07.993652 I | rook-ceph-mon62: 2017-08-20 15:57:07.993641 7f21039e8f40 1 mon.rook-ceph-mon62@-1(probing).paxosservice(auth 253..361) refresh upgraded, format 0 -> 2
2017-08-20 15:57:07.994393 I | rook-ceph-mon62: 2017-08-20 15:57:07.994368 7f21039e8f40 -1 mon.rook-ceph-mon62@-1(probing).mgr e7 Failed to load mgr commands: (2) No such file or directory
2017-08-20 15:57:07.994831 I | rook-ceph-mon62: 2017-08-20 15:57:07.994815 7f21039e8f40 0 mon.rook-ceph-mon62@-1(probing) e123 my rank is now 0 (was -1)
2017-08-20 15:57:07.996051 I | rook-ceph-mon62: 2017-08-20 15:57:07.996032 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:07.996563 I | rook-ceph-mon62: 2017-08-20 15:57:07.996544 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:08.198306 I | rook-ceph-mon62: 2017-08-20 15:57:08.198193 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:08.198583 I | rook-ceph-mon62: 2017-08-20 15:57:08.198544 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:08.600958 I | rook-ceph-mon62: 2017-08-20 15:57:08.600808 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:08.601423 I | rook-ceph-mon62: 2017-08-20 15:57:08.601360 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:09.403451 I | rook-ceph-mon62: 2017-08-20 15:57:09.403362 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:09.403852 I | rook-ceph-mon62: 2017-08-20 15:57:09.403769 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:11.006505 I | rook-ceph-mon62: 2017-08-20 15:57:11.006385 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:11.006908 I | rook-ceph-mon62: 2017-08-20 15:57:11.006890 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:14.209134 I | rook-ceph-mon62: 2017-08-20 15:57:14.209000 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:14.209393 I | rook-ceph-mon62: 2017-08-20 15:57:14.209356 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:20.614256 I | rook-ceph-mon62: 2017-08-20 15:57:20.614071 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:20.614615 I | rook-ceph-mon62: 2017-08-20 15:57:20.614569 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:33.416577 I | rook-ceph-mon62: 2017-08-20 15:57:33.416503 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:33.417004 I | rook-ceph-mon62: 2017-08-20 15:57:33.416971 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:48.419830 I | rook-ceph-mon62: 2017-08-20 15:57:48.419700 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:57:48.420341 I | rook-ceph-mon62: 2017-08-20 15:57:48.420273 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:58:03.422456 I | rook-ceph-mon62: 2017-08-20 15:58:03.422351 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:58:03.422741 I | rook-ceph-mon62: 2017-08-20 15:58:03.422719 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:58:07.994741 I | rook-ceph-mon62: 2017-08-20 15:58:07.994603 7f20fa7b3700 0 mon.rook-ceph-mon62@0(probing).data_health(0) update_stats avail 83%!t(MISSING)otal 81965 MB, used 10400 MB, avail 68142 MB
2017-08-20 15:58:18.425353 I | rook-ceph-mon62: 2017-08-20 15:58:18.425281 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:58:18.425775 I | rook-ceph-mon62: 2017-08-20 15:58:18.425716 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:58:33.427213 I | rook-ceph-mon62: 2017-08-20 15:58:33.427082 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:58:33.427570 I | rook-ceph-mon62: 2017-08-20 15:58:33.427517 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:58:48.429046 I | rook-ceph-mon62: 2017-08-20 15:58:48.428968 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:58:48.429390 I | rook-ceph-mon62: 2017-08-20 15:58:48.429370 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:59:03.431522 I | rook-ceph-mon62: 2017-08-20 15:59:03.431402 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:59:03.431807 I | rook-ceph-mon62: 2017-08-20 15:59:03.431762 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:59:07.994998 I | rook-ceph-mon62: 2017-08-20 15:59:07.994937 7f20fa7b3700 0 mon.rook-ceph-mon62@0(probing).data_health(0) update_stats avail 83%!t(MISSING)otal 81965 MB, used 10408 MB, avail 68133 MB
2017-08-20 15:59:18.438151 I | rook-ceph-mon62: 2017-08-20 15:59:18.438055 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:59:18.438595 I | rook-ceph-mon62: 2017-08-20 15:59:18.438533 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:59:33.440406 I | rook-ceph-mon62: 2017-08-20 15:59:33.440319 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:59:33.440848 I | rook-ceph-mon62: 2017-08-20 15:59:33.440815 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:59:48.443290 I | rook-ceph-mon62: 2017-08-20 15:59:48.443215 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 15:59:48.443623 I | rook-ceph-mon62: 2017-08-20 15:59:48.443602 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:00:03.445337 I | rook-ceph-mon62: 2017-08-20 16:00:03.445219 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:00:03.445680 I | rook-ceph-mon62: 2017-08-20 16:00:03.445646 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:00:07.995151 I | rook-ceph-mon62: 2017-08-20 16:00:07.995110 7f20fa7b3700 0 mon.rook-ceph-mon62@0(probing).data_health(0) update_stats avail 83%!t(MISSING)otal 81965 MB, used 10409 MB, avail 68132 MB
2017-08-20 16:00:18.450965 I | rook-ceph-mon62: 2017-08-20 16:00:18.450842 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:00:18.451294 I | rook-ceph-mon62: 2017-08-20 16:00:18.451237 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:00:33.463826 I | rook-ceph-mon62: 2017-08-20 16:00:33.463700 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:00:33.464128 I | rook-ceph-mon62: 2017-08-20 16:00:33.464110 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:00:48.469473 I | rook-ceph-mon62: 2017-08-20 16:00:48.469365 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:00:48.469775 I | rook-ceph-mon62: 2017-08-20 16:00:48.469757 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:01:03.484930 I | rook-ceph-mon62: 2017-08-20 16:01:03.484785 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:01:03.485300 I | rook-ceph-mon62: 2017-08-20 16:01:03.485236 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:01:07.995482 I | rook-ceph-mon62: 2017-08-20 16:01:07.995380 7f20fa7b3700 0 mon.rook-ceph-mon62@0(probing).data_health(0) update_stats avail 83%!t(MISSING)otal 81965 MB, used 10410 MB, avail 68131 MB
2017-08-20 16:01:18.487368 I | rook-ceph-mon62: 2017-08-20 16:01:18.487257 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:01:18.487679 I | rook-ceph-mon62: 2017-08-20 16:01:18.487633 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:01:33.489249 I | rook-ceph-mon62: 2017-08-20 16:01:33.489128 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:01:33.489528 I | rook-ceph-mon62: 2017-08-20 16:01:33.489509 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:01:48.491750 I | rook-ceph-mon62: 2017-08-20 16:01:48.491665 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:01:48.492153 I | rook-ceph-mon62: 2017-08-20 16:01:48.492118 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:02:00.461108 I | rook-ceph-mon62: 2017-08-20 16:02:00.461048 7f20f47a7700 0 -- 10.3.149.229:6790/0 >> 10.3.169.122:6790/0 conn(0xaf857e3800 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:02:03.496318 I | rook-ceph-mon62: 2017-08-20 16:02:03.496129 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:02:03.496780 I | rook-ceph-mon62: 2017-08-20 16:02:03.496757 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:02:05.000422 I | rook-ceph-mon62: 2017-08-20 16:02:05.000324 7f20f3fa6700 0 -- 10.3.149.229:6790/0 >> 10.3.192.19:6790/0 conn(0xaf857a4800 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:02:07.995755 I | rook-ceph-mon62: 2017-08-20 16:02:07.995657 7f20fa7b3700 0 mon.rook-ceph-mon62@0(probing).data_health(0) update_stats avail 83%!t(MISSING)otal 81965 MB, used 10411 MB, avail 68131 MB
2017-08-20 16:02:15.462188 I | rook-ceph-mon62: 2017-08-20 16:02:15.462098 7f20f47a7700 0 -- 10.3.149.229:6790/0 >> 10.3.169.122:6790/0 conn(0xaf85725000 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:02:18.498159 I | rook-ceph-mon62: 2017-08-20 16:02:18.498091 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:02:18.498575 I | rook-ceph-mon62: 2017-08-20 16:02:18.498523 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:02:20.002718 I | rook-ceph-mon62: 2017-08-20 16:02:20.002650 7f20f3fa6700 0 -- 10.3.149.229:6790/0 >> 10.3.192.19:6790/0 conn(0xaf857e3800 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:02:30.463544 I | rook-ceph-mon62: 2017-08-20 16:02:30.463434 7f20f4fa8700 0 -- 10.3.149.229:6790/0 >> 10.3.169.122:6790/0 conn(0xaf857e2000 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:02:33.500555 I | rook-ceph-mon62: 2017-08-20 16:02:33.500432 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:02:33.500818 I | rook-ceph-mon62: 2017-08-20 16:02:33.500802 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:02:35.003603 I | rook-ceph-mon62: 2017-08-20 16:02:35.003479 7f20f4fa8700 0 -- 10.3.149.229:6790/0 >> 10.3.192.19:6790/0 conn(0xaf857e2000 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:02:45.464845 I | rook-ceph-mon62: 2017-08-20 16:02:45.464743 7f20f47a7700 0 -- 10.3.149.229:6790/0 >> 10.3.169.122:6790/0 conn(0xaf85885800 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:02:48.502290 I | rook-ceph-mon62: 2017-08-20 16:02:48.502153 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:02:48.502656 I | rook-ceph-mon62: 2017-08-20 16:02:48.502562 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:02:50.004711 I | rook-ceph-mon62: 2017-08-20 16:02:50.004592 7f20f4fa8700 0 -- 10.3.149.229:6790/0 >> 10.3.192.19:6790/0 conn(0xaf85887000 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:03:00.466195 I | rook-ceph-mon62: 2017-08-20 16:03:00.466067 7f20f4fa8700 0 -- 10.3.149.229:6790/0 >> 10.3.169.122:6790/0 conn(0xaf85722000 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:03:03.504413 I | rook-ceph-mon62: 2017-08-20 16:03:03.504301 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:03:03.504747 I | rook-ceph-mon62: 2017-08-20 16:03:03.504716 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:03:05.005310 I | rook-ceph-mon62: 2017-08-20 16:03:05.005252 7f20f47a7700 0 -- 10.3.149.229:6790/0 >> 10.3.192.19:6790/0 conn(0xaf857a4800 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:03:07.995944 I | rook-ceph-mon62: 2017-08-20 16:03:07.995845 7f20fa7b3700 0 mon.rook-ceph-mon62@0(probing).data_health(0) update_stats avail 83%!t(MISSING)otal 81965 MB, used 10411 MB, avail 68130 MB
2017-08-20 16:03:15.467012 I | rook-ceph-mon62: 2017-08-20 16:03:15.466972 7f20f47a7700 0 -- 10.3.149.229:6790/0 >> 10.3.169.122:6790/0 conn(0xaf85884000 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:03:18.508807 I | rook-ceph-mon62: 2017-08-20 16:03:18.508745 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:03:18.509234 I | rook-ceph-mon62: 2017-08-20 16:03:18.509198 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:03:20.006574 I | rook-ceph-mon62: 2017-08-20 16:03:20.006414 7f20f47a7700 0 -- 10.3.149.229:6790/0 >> 10.3.192.19:6790/0 conn(0xaf85884000 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:03:30.468061 I | rook-ceph-mon62: 2017-08-20 16:03:30.467942 7f20f4fa8700 0 -- 10.3.149.229:6790/0 >> 10.3.169.122:6790/0 conn(0xaf85723800 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:03:33.511738 I | rook-ceph-mon62: 2017-08-20 16:03:33.511598 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:03:33.512197 I | rook-ceph-mon62: 2017-08-20 16:03:33.512148 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:03:35.007644 I | rook-ceph-mon62: 2017-08-20 16:03:35.007530 7f20f4fa8700 0 -- 10.3.149.229:6790/0 >> 10.3.192.19:6790/0 conn(0xaf85726800 :6790 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_RE
2017-08-20 16:03:43.065286 I | rook-ceph-mon62: 2017-08-20 16:03:43.065229 7f20f7fae700 0 log_channel(cluster) log [INF] : mon.rook-ceph-mon62 calling new monitor election
2017-08-20 16:03:43.127120 I | rook-ceph-mon62: 2017-08-20 16:03:43.127015 7f20f7fae700 0 log_channel(cluster) log [INF] : mon.rook-ceph-mon62@0 won leader election with quorum 0,1,2
2017-08-20 16:03:43.152112 I | rook-ceph-mon62: 2017-08-20 16:03:43.152082 7f20f7fae700 0 log_channel(cluster) log [WRN] : overall HEALTH_WARN 1 MDSs report slow requests
2017-08-20 16:03:43.179428 I | rook-ceph-mon62: 2017-08-20 16:03:43.179312 7f20f7fae700 0 log_channel(cluster) log [INF] : monmap e123: 3 mons at {rook-ceph-mon58=10.3.192.19:6790/0,rook-ceph-mon59=10.3.169.122:6790/0,rook-ceph-mon62=10.3.149.229:6790/0}
2017-08-20 16:03:43.179549 I | rook-ceph-mon62: 2017-08-20 16:03:43.179384 7f20f7fae700 0 log_channel(cluster) log [DBG] : fsmap cfs-r2-1-1/1/1 up {0=mds1=up:active}
2017-08-20 16:03:43.179572 I | rook-ceph-mon62: 2017-08-20 16:03:43.179518 7f20f7fae700 0 log_channel(cluster) log [DBG] : osdmap e175: 16 total, 16 up, 16 in
2017-08-20 16:03:43.179755 I | rook-ceph-mon62: 2017-08-20 16:03:43.179624 7f20f7fae700 0 log_channel(cluster) log [DBG] : mgrmap e7: rook-ceph-mgr0(active)
2017-08-20 16:03:43.337500 I | rook-ceph-mon62: 2017-08-20 16:03:43.065094 mon.rook-ceph-mon59 mon.1 10.3.169.122:6790/0 1 : cluster [INF] mon.rook-ceph-mon59 calling new monitor election
2017-08-20 16:03:43.337527 I | rook-ceph-mon62: 2017-08-20 16:03:43.065236 mon.rook-ceph-mon58 mon.2 10.3.192.19:6790/0 1 : cluster [INF] mon.rook-ceph-mon58 calling new monitor election
2017-08-20 16:03:43.337530 I | rook-ceph-mon62: 2017-08-20 16:03:43.065237 mon.rook-ceph-mon62 mon.0 10.3.149.229:6790/0 1 : cluster [INF] mon.rook-ceph-mon62 calling new monitor election
2017-08-20 16:03:43.337546 I | rook-ceph-mon62: 2017-08-20 16:03:43.127018 mon.rook-ceph-mon62 mon.0 10.3.149.229:6790/0 2 : cluster [INF] mon.rook-ceph-mon62@0 won leader election with quorum 0,1,2
2017-08-20 16:03:43.337549 I | rook-ceph-mon62: 2017-08-20 16:03:43.152085 mon.rook-ceph-mon62 mon.0 10.3.149.229:6790/0 3 : cluster [WRN] overall HEALTH_WARN 1 MDSs report slow requests
2017-08-20 16:03:43.337552 I | rook-ceph-mon62: 2017-08-20 16:03:43.179328 mon.rook-ceph-mon62 mon.0 10.3.149.229:6790/0 4 : cluster [INF] monmap e123: 3 mons at {rook-ceph-mon58=10.3.192.19:6790/0,rook-ceph-mon59=10.3.169.122:6790/0,rook-ceph-mon62=10.3.149.229:6790/0}
2017-08-20 16:03:44.412258 I | rook-ceph-mon62: 2017-08-20 15:56:33.402303 mds.mds1 mds.0 100.64.0.48:6800/3988930595 26 : cluster [WRN] slow request 1240.020809 seconds old, received at 2017-08-20 15:35:53.381468: client_request(client.325076:138 lookup #0x10000000002/docker 2017-08-20 15:35:53.380211 caller_uid=0, caller_gid=0{}) currently initiated
2017-08-20 16:03:46.648405 I | rook-ceph-mon62: 2017-08-20 16:03:46.648280 7f20f7fae700 0 log_channel(cluster) log [WRN] : MDS health message (mds.0): 1 slow requests are blocked > 30 sec
2017-08-20 16:03:46.648437 I | rook-ceph-mon62: 2017-08-20 16:03:46.648313 7f20f7fae700 0 log_channel(cluster) log [INF] : daemon mds.mds1 is now active in filesystem cfs-r2-1 as rank 0
2017-08-20 16:03:46.788562 I | rook-ceph-mon62: 2017-08-20 16:03:46.788479 7f20f37a5700 0 mon.rook-ceph-mon62@0(leader).mds e34 print_map
2017-08-20 16:03:46.788588 I | rook-ceph-mon62: e34
2017-08-20 16:03:46.788607 I | rook-ceph-mon62: enable_multiple, ever_enabled_multiple: 0,0
2017-08-20 16:03:46.788609 I | rook-ceph-mon62: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
2017-08-20 16:03:46.788611 I | rook-ceph-mon62: legacy client fscid: 1
2017-08-20 16:03:46.788612 I | rook-ceph-mon62:
2017-08-20 16:03:46.788629 I | rook-ceph-mon62: Filesystem 'cfs-r2-1' (1)
2017-08-20 16:03:46.788631 I | rook-ceph-mon62: fs_name cfs-r2-1
2017-08-20 16:03:46.788633 I | rook-ceph-mon62: epoch 34
2017-08-20 16:03:46.788634 I | rook-ceph-mon62: flags c
2017-08-20 16:03:46.788636 I | rook-ceph-mon62: created 2017-08-18 09:35:11.962291
2017-08-20 16:03:46.788639 I | rook-ceph-mon62: modified 2017-08-18 09:35:11.962291
2017-08-20 16:03:46.788654 I | rook-ceph-mon62: tableserver 0
2017-08-20 16:03:46.788657 I | rook-ceph-mon62: root 0
2017-08-20 16:03:46.788659 I | rook-ceph-mon62: session_timeout 60
2017-08-20 16:03:46.788662 I | rook-ceph-mon62: session_autoclose 300
2017-08-20 16:03:46.788665 I | rook-ceph-mon62: max_file_size 1099511627776
2017-08-20 16:03:46.788667 I | rook-ceph-mon62: last_failure 0
2017-08-20 16:03:46.788669 I | rook-ceph-mon62: last_failure_osd_epoch 174
2017-08-20 16:03:46.788671 I | rook-ceph-mon62: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
2017-08-20 16:03:46.788674 I | rook-ceph-mon62: max_mds 1
2017-08-20 16:03:46.788676 I | rook-ceph-mon62: in 0
2017-08-20 16:03:46.788678 I | rook-ceph-mon62: up {0=228660}
2017-08-20 16:03:46.788680 I | rook-ceph-mon62: failed
2017-08-20 16:03:46.788683 I | rook-ceph-mon62: damaged
2017-08-20 16:03:46.788686 I | rook-ceph-mon62: stopped
2017-08-20 16:03:46.788689 I | rook-ceph-mon62: data_pools [3]
2017-08-20 16:03:46.788692 I | rook-ceph-mon62: metadata_pool 4
2017-08-20 16:03:46.788696 I | rook-ceph-mon62: inline_data disabled
2017-08-20 16:03:46.788699 I | rook-ceph-mon62: balancer
2017-08-20 16:03:46.788702 I | rook-ceph-mon62: standby_count_wanted 0
2017-08-20 16:03:46.788706 I | rook-ceph-mon62: 228660: 100.64.0.48:6800/3988930595 'mds1' mds.0.17 up:active seq 22489
2017-08-20 16:03:46.788709 I | rook-ceph-mon62:
2017-08-20 16:03:46.788712 I | rook-ceph-mon62:
2017-08-20 16:03:46.788716 I | rook-ceph-mon62:
2017-08-20 16:03:46.788722 I | rook-ceph-mon62: 2017-08-20 16:03:46.788711 7f20f37a5700 0 log_channel(cluster) log [DBG] : mds.0 100.64.0.48:6800/3988930595 up:active
2017-08-20 16:03:46.788826 I | rook-ceph-mon62: 2017-08-20 16:03:46.788792 7f20f37a5700 0 log_channel(cluster) log [DBG] : fsmap cfs-r2-1-1/1/1 up {0=mds1=up:active}
2017-08-20 16:03:46.855502 I | rook-ceph-mon62: 2017-08-20 16:03:46.648281 mon.rook-ceph-mon62 mon.0 10.3.149.229:6790/0 8 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec
2017-08-20 16:03:46.855532 I | rook-ceph-mon62: 2017-08-20 16:03:46.648314 mon.rook-ceph-mon62 mon.0 10.3.149.229:6790/0 9 : cluster [INF] daemon mds.mds1 is now active in filesystem cfs-r2-1 as rank 0
2017-08-20 16:03:48.515666 I | rook-ceph-mon62: 2017-08-20 16:03:48.515533 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:03:48.515966 I | rook-ceph-mon62: 2017-08-20 16:03:48.515949 7f20f4fa8700 0 -- 100.64.2.65:0/22 >> 100.64.6.26:6800/18 conn(0xaf856de000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-08-20 16:03:54.738728 I | rook-ceph-mon62: 2017-08-20 16:03:54.738609 7f20f7fae700 0 mon.rook-ceph-mon62@0(leader) e123 handle_command mon_command({"prefix": "osd perf", "format": "json"} v 0) v1
2017-08-20 16:03:54.738758 I | rook-ceph-mon62: 2017-08-20 16:03:54.738663 7f20f7fae700 0 log_channel(audit) log [DBG] : from='client.242235 100.64.3.35:0/472170703' entity='client.admin' cmd=[{"prefix": "osd perf", "format": "json"}]: dispatch
2017-08-20 16:03:54.933157 I | rook-ceph-mon62: 2017-08-20 15:57:07.994368 7f21039e8f40 -1 mon.rook-ceph-mon62@-1(probing).mgr e7 Failed to load mgr commands: (2) No such file or directory
failed to run mon. failed to start mon: failed to run ceph-mon: Failed to complete rook-ceph-mon62: signal: aborted (core dumped)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment