Created
July 26, 2016 11:12
-
-
Save anonymous/42a1856cc819de5b110d9f887e9859d2 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2016-07-26 10:34:32.730707 7f95212107a0 0 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403), process ceph-mon, pid 42114 | |
| 2016-07-26 10:34:32.742473 7f95212107a0 10 load: jerasure load: lrc load: isa | |
| 2016-07-26 10:34:32.837564 7f95212107a0 10 obtain_monmap | |
| 2016-07-26 10:34:32.837620 7f95212107a0 10 obtain_monmap read last committed monmap ver 5 | |
| 2016-07-26 10:34:32.837681 7f95212107a0 0 starting mon.615yl02 rank 4 at 10.98.2.216:6789/0 mon_data /var/lib/ceph/mon/ceph-615yl02 fsid 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af | |
| 2016-07-26 10:34:32.837735 7f95212107a0 1 -- 10.98.2.216:6789/0 learned my addr 10.98.2.216:6789/0 | |
| 2016-07-26 10:34:32.837741 7f95212107a0 1 accepter.accepter.bind my_inst.addr is 10.98.2.216:6789/0 need_addr=0 | |
| 2016-07-26 10:34:32.837894 7f95212107a0 1 mon.615yl02@-1(probing) e5 preinit fsid 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af | |
| 2016-07-26 10:34:32.837942 7f95212107a0 10 mon.615yl02@-1(probing) e5 check_fsid cluster_uuid contains '2f51a247-3155-4bcf-9aee-c6f6b2c5e2af' | |
| 2016-07-26 10:34:32.837969 7f95212107a0 10 mon.615yl02@-1(probing) e5 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?),4=support erasure code pools,5=new-style osdmap encoding,6=support isa/lrc erasure code} | |
| 2016-07-26 10:34:32.837976 7f95212107a0 10 mon.615yl02@-1(probing) e5 apply_compatset_features_to_quorum_requirements required_features 18416819765248 | |
| 2016-07-26 10:34:32.837979 7f95212107a0 10 mon.615yl02@-1(probing) e5 required_features 18416819765248 | |
| 2016-07-26 10:34:32.837994 7f95212107a0 10 mon.615yl02@-1(probing) e5 has_ever_joined = 1 | |
| 2016-07-26 10:34:32.838024 7f95212107a0 10 mon.615yl02@-1(probing) e5 sync_last_committed_floor 0 | |
| 2016-07-26 10:34:32.838029 7f95212107a0 10 mon.615yl02@-1(probing) e5 init_paxos | |
| 2016-07-26 10:34:32.838109 7f95212107a0 10 mon.615yl02@-1(probing).paxos(paxos recovering c 1318755..1319323) init last_pn: 0 accepted_pn: 469401 last_committed: 1319323 first_committed: 1318755 | |
| 2016-07-26 10:34:32.838117 7f95212107a0 10 mon.615yl02@-1(probing).paxos(paxos recovering c 1318755..1319323) init | |
| 2016-07-26 10:34:32.838139 7f95212107a0 10 mon.615yl02@-1(probing) e5 refresh_from_paxos | |
| 2016-07-26 10:34:32.838190 7f95212107a0 1 mon.615yl02@-1(probing).paxosservice(pgmap 538057..538795) refresh upgraded, format 0 -> 1 | |
| 2016-07-26 10:34:32.838198 7f95212107a0 1 mon.615yl02@-1(probing).pg v0 on_upgrade discarding in-core PGMap | |
| 2016-07-26 10:34:32.838217 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(pgmap 538057..538795) refresh | |
| 2016-07-26 10:34:32.838224 7f95212107a0 10 mon.615yl02@-1(probing).pg v0 update_from_paxos v0, read_full | |
| 2016-07-26 10:34:32.838226 7f95212107a0 10 mon.615yl02@-1(probing).pg v0 read_pgmap_meta | |
| 2016-07-26 10:34:32.849738 7f95212107a0 10 mon.615yl02@-1(probing).pg v538795 update_logger | |
| 2016-07-26 10:34:32.849776 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(mdsmap 1..1) refresh | |
| 2016-07-26 10:34:32.849780 7f95212107a0 10 mon.615yl02@-1(probing).mds e0 update_from_paxos version 1, my e 0 | |
| 2016-07-26 10:34:32.849805 7f95212107a0 10 mon.615yl02@-1(probing).mds e0 update_from_paxos got 1 | |
| 2016-07-26 10:34:32.849812 7f95212107a0 4 mon.615yl02@-1(probing).mds e1 new map | |
| 2016-07-26 10:34:32.849814 7f95212107a0 0 mon.615yl02@-1(probing).mds e1 print_map | |
| epoch 1 | |
| flags 0 | |
| created 0.000000 | |
| modified 2016-04-25 21:12:07.303333 | |
| tableserver 0 | |
| root 0 | |
| session_timeout 0 | |
| session_autoclose 0 | |
| max_file_size 0 | |
| last_failure 0 | |
| last_failure_osd_epoch 0 | |
| compat compat={},rocompat={},incompat={} | |
| max_mds 0 | |
| in | |
| up {} | |
| failed | |
| stopped | |
| data_pools | |
| metadata_pool 0 | |
| inline_data disabled | |
| 2016-07-26 10:34:32.849831 7f95212107a0 10 mon.615yl02@-1(probing).mds e1 update_logger | |
| 2016-07-26 10:34:32.849912 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(osdmap 1769..2433) refresh | |
| 2016-07-26 10:34:32.849970 7f95212107a0 7 mon.615yl02@-1(probing).osd e0 update_from_paxos loading latest full map e2433 | |
| 2016-07-26 10:34:32.850134 7f95212107a0 10 mon.615yl02@-1(probing).osd e2433 adding osd.48 to down_pending_out map | |
| 2016-07-26 10:34:32.850138 7f95212107a0 10 mon.615yl02@-1(probing).osd e2433 adding osd.49 to down_pending_out map | |
| 2016-07-26 10:34:32.850140 7f95212107a0 10 mon.615yl02@-1(probing).osd e2433 check_subs | |
| 2016-07-26 10:34:32.850142 7f95212107a0 10 mon.615yl02@-1(probing).osd e2433 share_map_with_random_osd no up osd on our session map | |
| 2016-07-26 10:34:32.850145 7f95212107a0 10 mon.615yl02@-1(probing).osd e2433 update_logger | |
| 2016-07-26 10:34:32.850153 7f95212107a0 0 mon.615yl02@-1(probing).osd e2433 crush map has features 1107558400, adjusting msgr requires | |
| 2016-07-26 10:34:32.850155 7f95212107a0 0 mon.615yl02@-1(probing).osd e2433 crush map has features 1107558400, adjusting msgr requires | |
| 2016-07-26 10:34:32.850157 7f95212107a0 0 mon.615yl02@-1(probing).osd e2433 crush map has features 1107558400, adjusting msgr requires | |
| 2016-07-26 10:34:32.850158 7f95212107a0 0 mon.615yl02@-1(probing).osd e2433 crush map has features 1107558400, adjusting msgr requires | |
| 2016-07-26 10:34:32.850230 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(logm 765975..766716) refresh | |
| 2016-07-26 10:34:32.850235 7f95212107a0 10 mon.615yl02@-1(probing).log v766716 update_from_paxos | |
| 2016-07-26 10:34:32.850238 7f95212107a0 10 mon.615yl02@-1(probing).log v766716 update_from_paxos version 766716 summary v 0 | |
| 2016-07-26 10:34:32.850249 7f95212107a0 10 mon.615yl02@-1(probing).log v766716 update_from_paxos latest full 766715 | |
| 2016-07-26 10:34:32.850274 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos loading summary e766715 | |
| 2016-07-26 10:34:32.850316 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos loaded summary e766715 | |
| 2016-07-26 10:34:32.850336 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:36:56.473836 mon.4 10.98.2.216:6789/0 1 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:34:32.850356 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:36:56.473912 mon.4 10.98.2.216:6789/0 2 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:34:32.850363 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:36:56.474080 mon.4 10.98.2.216:6789/0 3 : cluster [INF] mon.615yl02 calling new monitor election | |
| 2016-07-26 10:34:32.850377 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:24.253944 mon.4 10.98.2.216:6789/0 4 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:34:32.850383 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:24.254011 mon.4 10.98.2.216:6789/0 5 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:34:32.850389 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:24.254115 mon.1 10.98.2.167:6789/0 289 : cluster [INF] mon.60zxl02 calling new monitor election | |
| 2016-07-26 10:34:32.850402 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:24.254224 mon.2 10.98.2.173:6789/0 316 : cluster [INF] mon.610wl02 calling new monitor election | |
| 2016-07-26 10:34:32.850411 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:24.265758 mon.4 10.98.2.216:6789/0 6 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:34:32.850416 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:24.265820 mon.4 10.98.2.216:6789/0 7 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:34:32.850424 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:25.333663 mon.4 10.98.2.216:6789/0 8 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:34:32.850430 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:25.333722 mon.4 10.98.2.216:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:34:32.850435 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:26.412490 mon.4 10.98.2.216:6789/0 10 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:34:32.850440 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:26.412555 mon.4 10.98.2.216:6789/0 11 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:34:32.850446 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:27.490889 mon.4 10.98.2.216:6789/0 12 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:34:32.850451 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:27.490944 mon.4 10.98.2.216:6789/0 13 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:34:32.850456 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:28.569356 mon.4 10.98.2.216:6789/0 14 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:34:32.850462 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:28.569415 mon.4 10.98.2.216:6789/0 15 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:34:32.850467 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:29.647699 mon.4 10.98.2.216:6789/0 16 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:34:32.850473 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:29.647758 mon.4 10.98.2.216:6789/0 17 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:34:32.850478 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:39.348432 mon.1 10.98.2.167:6789/0 290 : cluster [INF] mon.60zxl02 calling new monitor election | |
| 2016-07-26 10:34:32.850486 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:39.348552 mon.2 10.98.2.173:6789/0 317 : cluster [INF] mon.610wl02 calling new monitor election | |
| 2016-07-26 10:34:32.850495 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:44.395483 mon.1 10.98.2.167:6789/0 291 : cluster [INF] mon.60zxl02@1 won leader election with quorum 1,2,4 | |
| 2016-07-26 10:34:32.850511 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:44.402186 mon.1 10.98.2.167:6789/0 292 : cluster [INF] HEALTH_WARN; 17 pgs degraded; 4 pgs down; 883 pgs peering; 1516 pgs stale; 17 pgs stuck degraded; 900 pgs stuck inactive; 1516 pgs stuck stale; 900 pgs stuck unclean; 17 pgs stuck undersized; 17 pgs undersized; recovery 14/2367 objects degraded (0.591%); too many PGs per OSD (316 > max 300); 2/19 in osds are down; 2 mons down, quorum 1,2,4 60zxl02,610wl02,615yl02 | |
| 2016-07-26 10:34:32.850522 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:44.435794 mon.1 10.98.2.167:6789/0 293 : cluster [INF] monmap e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:34:32.850536 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:44.435879 mon.1 10.98.2.167:6789/0 294 : cluster [INF] pgmap v538795: 2120 pgs: 296 stale+peering, 1220 stale+active+clean, 583 peering, 17 undersized+degraded+peered, 4 down+peering; 4715 MB data, 6612 MB used, 50009 GB / 50015 GB avail; 14/2367 objects degraded (0.591%) | |
| 2016-07-26 10:34:32.850546 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:44.435935 mon.1 10.98.2.167:6789/0 295 : cluster [INF] mdsmap e1: 0/0/0 up | |
| 2016-07-26 10:34:32.850555 7f95212107a0 7 mon.615yl02@-1(probing).log v766716 update_from_paxos applying incremental log 766716 2016-07-25 12:37:44.436630 mon.1 10.98.2.167:6789/0 296 : cluster [INF] osdmap e2433: 65 osds: 17 up, 19 in | |
| 2016-07-26 10:34:32.850587 7f95212107a0 10 mon.615yl02@-1(probing).log v766716 check_subs | |
| 2016-07-26 10:34:32.850631 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(monmap 1..5) refresh | |
| 2016-07-26 10:34:32.850706 7f95212107a0 1 mon.615yl02@-1(probing).paxosservice(auth 2251..2417) refresh upgraded, format 0 -> 1 | |
| 2016-07-26 10:34:32.850709 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(auth 2251..2417) refresh | |
| 2016-07-26 10:34:32.850720 7f95212107a0 10 mon.615yl02@-1(probing).auth v2417 update_from_paxos | |
| 2016-07-26 10:34:32.850736 7f95212107a0 10 mon.615yl02@-1(probing).auth v2417 update_from_paxos version 2417 keys ver 0 latest 2401 | |
| 2016-07-26 10:34:32.850753 7f95212107a0 7 mon.615yl02@-1(probing).auth v2417 update_from_paxos loading summary e 2401 | |
| 2016-07-26 10:34:32.850755 7f95212107a0 7 mon.615yl02@-1(probing).auth v2417 update_from_paxos latest length 8187 | |
| 2016-07-26 10:34:32.850956 7f95212107a0 10 mon.615yl02@-1(probing).auth v2417 update_from_paxos key server version 2401 | |
| 2016-07-26 10:34:32.851229 7f95212107a0 10 mon.615yl02@-1(probing).auth v2417 update_from_paxos() last_allocated_id=544096 max_global_id=544096 format_version 1 | |
| 2016-07-26 10:34:32.851232 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(pgmap 538057..538795) post_refresh | |
| 2016-07-26 10:34:32.851236 7f95212107a0 10 mon.615yl02@-1(probing).pg v538795 map_pg_creates to 0 pgs osdmap epoch 2433 | |
| 2016-07-26 10:34:32.851238 7f95212107a0 10 mon.615yl02@-1(probing).pg v538795 send_pg_creates to 0 pgs | |
| 2016-07-26 10:34:32.851240 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(mdsmap 1..1) post_refresh | |
| 2016-07-26 10:34:32.851241 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(osdmap 1769..2433) post_refresh | |
| 2016-07-26 10:34:32.851242 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(logm 765975..766716) post_refresh | |
| 2016-07-26 10:34:32.851242 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(monmap 1..5) post_refresh | |
| 2016-07-26 10:34:32.851243 7f95212107a0 10 mon.615yl02@-1(probing).paxosservice(auth 2251..2417) post_refresh | |
| 2016-07-26 10:34:32.851246 7f95212107a0 10 mon.615yl02@-1(probing).health(0) init | |
| 2016-07-26 10:37:30.747768 7f95212107a0 1 -- 10.98.2.216:6789/0 messenger.start | |
| 2016-07-26 10:37:30.747805 7f95212107a0 2 mon.615yl02@-1(probing) e5 init | |
| 2016-07-26 10:37:30.747863 7f95212107a0 1 accepter.accepter.start | |
| 2016-07-26 10:37:30.747885 7f95212107a0 10 mon.615yl02@-1(probing) e5 bootstrap | |
| 2016-07-26 10:37:30.747891 7f95212107a0 10 mon.615yl02@-1(probing) e5 sync_reset_requester | |
| 2016-07-26 10:37:30.747892 7f95212107a0 10 mon.615yl02@-1(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:37:30.747894 7f95212107a0 10 mon.615yl02@-1(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:37:30.747895 7f95212107a0 0 mon.615yl02@-1(probing) e5 my rank is now 4 (was -1) | |
| 2016-07-26 10:37:30.747897 7f95212107a0 1 -- 10.98.2.216:6789/0 mark_down_all | |
| 2016-07-26 10:37:30.747975 7f95212107a0 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:37:30.747979 7f95212107a0 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:37:30.747981 7f95212107a0 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:37:30.747983 7f95212107a0 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:37:30.747987 7f95212107a0 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:37:30.747982 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f2520 :/0 | |
| 2016-07-26 10:37:30.747992 7f95212107a0 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:37:30.748005 7f95212107a0 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:37:30.748009 7f95212107a0 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:37:30.748010 7f95212107a0 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:37:30.748011 7f95212107a0 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:37:30.748012 7f95212107a0 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:37:30.748015 7f95212107a0 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:37:30.748018 7f95212107a0 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3486f30 after 2 seconds | |
| 2016-07-26 10:37:30.748035 7f95212107a0 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:37:30.748041 7f95212107a0 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34f9180 | |
| 2016-07-26 10:37:30.748064 7f951646c700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37e9000 sd=15 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2680).accept sd=15 10.98.3.15:56799/0 | |
| 2016-07-26 10:37:30.748094 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f2680 :/0 | |
| 2016-07-26 10:37:30.748100 7f951636b700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37e4800 sd=16 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2aa0).accept sd=16 10.98.3.15:56811/0 | |
| 2016-07-26 10:37:30.748131 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f2aa0 :/0 | |
| 2016-07-26 10:37:30.748128 7f95212107a0 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34f8f00 | |
| 2016-07-26 10:37:30.748178 7f951656d700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37f2000 sd=17 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2ec0).accept sd=17 10.98.3.15:56813/0 | |
| 2016-07-26 10:37:30.748202 7f95212107a0 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34f8c80 | |
| 2016-07-26 10:37:30.748209 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f2ec0 :/0 | |
| 2016-07-26 10:37:30.748247 7f951636b700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37ed800 sd=15 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2d60).accept sd=15 10.98.3.15:56816/0 | |
| 2016-07-26 10:37:30.748264 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f2d60 :/0 | |
| 2016-07-26 10:37:30.748297 7f9516169700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37e9000 sd=16 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2520).accept sd=16 10.98.3.15:56828/0 | |
| 2016-07-26 10:37:30.748308 7f95212107a0 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34f9b80 | |
| 2016-07-26 10:37:30.748329 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f2520 :/0 | |
| 2016-07-26 10:37:30.748366 7f951656d700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x3725800 sd=19 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f3440).accept sd=19 10.98.3.15:56836/0 | |
| 2016-07-26 10:37:30.748378 7f951646c700 10 mon.615yl02@4(probing) e5 ms_get_authorizer for mon | |
| 2016-07-26 10:37:30.748386 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f3440 :/0 | |
| 2016-07-26 10:37:30.748392 7f951626a700 10 mon.615yl02@4(probing) e5 ms_get_authorizer for mon | |
| 2016-07-26 10:37:30.748394 7f9515f67700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x3721000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f32e0).accept sd=21 10.98.3.15:56870/0 | |
| 2016-07-26 10:37:30.748431 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f32e0 :/0 | |
| 2016-07-26 10:37:30.748500 7f9516068700 10 mon.615yl02@4(probing) e5 ms_get_authorizer for mon | |
| 2016-07-26 10:37:30.748530 7f9515f67700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37e4800 sd=22 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f3020).accept sd=22 10.98.3.23:52925/0 | |
| 2016-07-26 10:37:30.748551 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f3020 :/0 | |
| 2016-07-26 10:37:30.748549 7f9516169700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37ed800 sd=15 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2aa0).accept sd=15 10.98.3.15:56881/0 | |
| 2016-07-26 10:37:30.748566 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f2aa0 :/0 | |
| 2016-07-26 10:37:30.748570 7f9518d71700 10 mon.615yl02@4(probing).data_health(0) service_tick | |
| 2016-07-26 10:37:30.748590 7f9518d71700 0 mon.615yl02@4(probing).data_health(0) update_stats avail 94% total 2015 GB, used 4665 MB, avail 1908 GB | |
| 2016-07-26 10:37:30.748596 7f951656d700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37e9000 sd=16 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f40a0).accept sd=16 10.98.3.15:56886/0 | |
| 2016-07-26 10:37:30.748616 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f40a0 :/0 | |
| 2016-07-26 10:37:30.748636 7f951636b700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x3721000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2ec0).accept sd=21 10.98.2.167:38065/0 | |
| 2016-07-26 10:37:30.748650 7f9515f67700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37f2000 sd=22 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2d60).accept sd=22 10.98.2.173:60445/0 | |
| 2016-07-26 10:37:30.748801 7f951636b700 10 mon.615yl02@4(probing) e5 ms_verify_authorizer 10.98.2.167:6789/0 mon protocol 2 | |
| 2016-07-26 10:37:30.748843 7f9515f67700 10 mon.615yl02@4(probing) e5 ms_verify_authorizer 10.98.2.173:6789/0 mon protocol 2 | |
| 2016-07-26 10:37:30.748861 7f951636b700 0 -- 10.98.2.216:6789/0 >> 10.98.2.167:6789/0 pipe(0x3721000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2ec0).accept connect_seq 0 vs existing 0 state wait | |
| 2016-07-26 10:37:30.748882 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f2ec0 10.98.2.167:6789/0 | |
| 2016-07-26 10:37:30.748900 7f9515f67700 0 -- 10.98.2.216:6789/0 >> 10.98.2.173:6789/0 pipe(0x37f2000 sd=22 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2d60).accept connect_seq 0 vs existing 0 state wait | |
| 2016-07-26 10:37:30.748923 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f2d60 10.98.2.173:6789/0 | |
| 2016-07-26 10:37:30.788408 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 1 ==== mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02) v6 ==== 64+0+0 (3448912970 0 0) 0x34f9e00 con 0x33f1e40 | |
| 2016-07-26 10:37:30.788422 7f9518370700 10 mon.615yl02@4(probing) e5 do not have session, making new one | |
| 2016-07-26 10:37:30.788425 7f9518370700 10 mon.615yl02@4(probing) e5 ms_dispatch new session MonSession: mon.2 10.98.2.173:6789/0 is open for mon.2 10.98.2.173:6789/0 | |
| 2016-07-26 10:37:30.788432 7f9518370700 5 mon.615yl02@4(probing) e5 setting monitor caps on this connection | |
| 2016-07-26 10:37:30.788439 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02) v6 | |
| 2016-07-26 10:37:30.788443 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_probe mon.2 10.98.2.173:6789/0mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02) v6 features 37154696925806591 | |
| 2016-07-26 10:37:30.788459 7f9518370700 1 -- 10.98.2.216:6789/0 --> 10.98.2.173:6789/0 -- mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02 paxos( fc 1318755 lc 1319323 )) v6 -- ?+0 0x34fad00 con 0x33f1e40 | |
| 2016-07-26 10:37:30.788475 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.1 10.98.2.167:6789/0 1 ==== mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02) v6 ==== 64+0+0 (3585442509 0 0) 0x34fa080 con 0x33f2100 | |
| 2016-07-26 10:37:30.788482 7f9518370700 10 mon.615yl02@4(probing) e5 do not have session, making new one | |
| 2016-07-26 10:37:30.788485 7f9518370700 10 mon.615yl02@4(probing) e5 ms_dispatch new session MonSession: mon.1 10.98.2.167:6789/0 is open for mon.1 10.98.2.167:6789/0 | |
| 2016-07-26 10:37:30.788488 7f9518370700 5 mon.615yl02@4(probing) e5 setting monitor caps on this connection | |
| 2016-07-26 10:37:30.788490 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02) v6 | |
| 2016-07-26 10:37:30.788493 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_probe mon.1 10.98.2.167:6789/0mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02) v6 features 37154696925806591 | |
| 2016-07-26 10:37:30.788501 7f9518370700 1 -- 10.98.2.216:6789/0 --> 10.98.2.167:6789/0 -- mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02 paxos( fc 1318755 lc 1319323 )) v6 -- ?+0 0x34f9e00 con 0x33f2100 | |
| 2016-07-26 10:37:30.788511 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 2 ==== mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02) v6 ==== 64+0+0 (3448912970 0 0) 0x34fa580 con 0x33f1e40 | |
| 2016-07-26 10:37:30.788518 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02) v6 | |
| 2016-07-26 10:37:30.788520 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_probe mon.2 10.98.2.173:6789/0mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02) v6 features 37154696925806591 | |
| 2016-07-26 10:37:30.788527 7f9518370700 1 -- 10.98.2.216:6789/0 --> 10.98.2.173:6789/0 -- mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02 paxos( fc 1318755 lc 1319323 )) v6 -- ?+0 0x34fa080 con 0x33f1e40 | |
| 2016-07-26 10:37:30.788536 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.1 10.98.2.167:6789/0 2 ==== mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02) v6 ==== 64+0+0 (3585442509 0 0) 0x34faa80 con 0x33f2100 | |
| 2016-07-26 10:37:30.788543 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02) v6 | |
| 2016-07-26 10:37:30.788545 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_probe mon.1 10.98.2.167:6789/0mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02) v6 features 37154696925806591 | |
| 2016-07-26 10:37:30.788552 7f9518370700 1 -- 10.98.2.216:6789/0 --> 10.98.2.167:6789/0 -- mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02 paxos( fc 1318755 lc 1319323 )) v6 -- ?+0 0x34fa580 con 0x33f2100 | |
| 2016-07-26 10:37:30.788568 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.1 10.98.2.167:6789/0 3 ==== mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02) v6 ==== 64+0+0 (3585442509 0 0) 0x34fa800 con 0x33f2100 | |
| 2016-07-26 10:37:30.788575 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02) v6 | |
| 2016-07-26 10:37:30.788578 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_probe mon.1 10.98.2.167:6789/0mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02) v6 features 37154696925806591 | |
| 2016-07-26 10:37:30.788590 7f9518370700 1 -- 10.98.2.216:6789/0 --> 10.98.2.167:6789/0 -- mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02 paxos( fc 1318755 lc 1319323 )) v6 -- ?+0 0x34faa80 con 0x33f2100 | |
| 2016-07-26 10:37:30.788597 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 3 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34f8c80 con 0x33f1e40 | |
| 2016-07-26 10:37:30.788607 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:37:30.788612 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:37:30.788617 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:37:30.788629 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:37:30.788631 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:37:30.788633 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:37:30.788635 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:37:30.788651 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.1 10.98.2.167:6789/0 4 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (109461966 0 0) 0x34f8f00 con 0x33f2100 | |
| 2016-07-26 10:37:30.788658 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:37:30.788661 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.1 10.98.2.167:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 60zxl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:37:30.788665 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:37:30.788674 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 60zxl02 | |
| 2016-07-26 10:37:30.788675 7f9518370700 10 mon.615yl02@4(probing) e5 mon.60zxl02 is outside the quorum | |
| 2016-07-26 10:37:30.788676 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 60zxl02,610wl02,615yl02, need 3 | |
| 2016-07-26 10:37:30.788678 7f9518370700 10 mon.615yl02@4(probing) e5 that's enough to form a new quorum, calling election | |
| 2016-07-26 10:37:30.788679 7f9518370700 10 mon.615yl02@4(probing) e5 start_election | |
| 2016-07-26 10:37:30.788681 7f9518370700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:37:30.788682 7f9518370700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout 0x3486f30 | |
| 2016-07-26 10:37:30.788684 7f9518370700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:37:30.788686 7f9518370700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:37:30.788688 7f9518370700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:37:30.788691 7f9518370700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:37:30.788694 7f9518370700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:37:30.788695 7f9518370700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:37:30.788697 7f9518370700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:37:30.788698 7f9518370700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:37:30.788699 7f9518370700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:37:30.788702 7f9518370700 10 mon.615yl02@4(electing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:37:30.788719 7f9518370700 0 log_channel(cluster) log [INF] : mon.615yl02 calling new monitor election | |
| 2016-07-26 10:37:30.788721 7f9518370700 0 log_channel(cluster) do_log log to syslog | |
| 2016-07-26 10:37:32.805346 7f951656d700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37e4800 sd=16 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2520).accept sd=16 10.98.3.15:57039/0 | |
| 2016-07-26 10:37:32.805441 7f951656d700 10 mon.615yl02@4(electing) e5 ms_verify_authorizer 10.98.3.15:0/4187059021 client protocol 0 | |
| 2016-07-26 10:37:33.748543 7f9515e66700 0 -- 10.98.2.216:6789/0 >> 10.98.2.214:6789/0 pipe(0x372a000 sd=25 :0 s=1 pgs=0 cs=0 l=0 c=0x33f1b80).fault | |
| 2016-07-26 10:37:45.384038 7f951656d700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x3725800 sd=18 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f32e0).accept sd=18 10.98.3.23:53054/0 | |
| 2016-07-26 10:37:45.384156 7f951656d700 10 mon.615yl02@4(electing) e5 ms_verify_authorizer 10.98.3.23:0/1011397 client protocol 0 | |
| 2016-07-26 10:37:45.748191 7f9515364700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37e4800 sd=20 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f3440).accept sd=20 10.98.2.166:39205/0 | |
| 2016-07-26 10:37:53.807208 7f9515263700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x371c800 sd=23 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f4200).accept sd=23 10.98.3.15:57066/0 | |
| 2016-07-26 10:37:53.807309 7f9515263700 10 mon.615yl02@4(electing) e5 ms_verify_authorizer 10.98.3.15:0/4187059021 client protocol 0 | |
| 2016-07-26 10:38:02.808008 7f9515263700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x3718000 sd=23 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2aa0).accept sd=23 10.98.3.15:57077/0 | |
| 2016-07-26 10:38:02.808100 7f9515263700 10 mon.615yl02@4(electing) e5 ms_verify_authorizer 10.98.3.15:0/4187059021 client protocol 0 | |
| 2016-07-26 10:38:04.857483 7f9515061700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37ed800 sd=25 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f40a0).accept sd=25 10.98.3.18:41077/0 | |
| 2016-07-26 10:38:04.857601 7f9515061700 10 mon.615yl02@4(electing) e5 ms_verify_authorizer 10.98.3.18:0/1012640 client protocol 0 | |
| 2016-07-26 10:38:06.750858 7f9518370700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a0d80 con 0x33f11e0 | |
| 2016-07-26 10:38:06.750870 7f9518370700 5 mon.615yl02@4(electing).elector(0) start -- can i be leader? | |
| 2016-07-26 10:38:06.750932 7f9518370700 1 mon.615yl02@4(electing).elector(11769) init, last seen epoch 11769 | |
| 2016-07-26 10:38:06.814577 7f9518370700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af propose 11769) v5 -- ?+0 0x34e23c0 | |
| 2016-07-26 10:38:06.814593 7f9518370700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af propose 11769) v5 -- ?+0 0x34e2940 | |
| 2016-07-26 10:38:06.814605 7f9518370700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af propose 11769) v5 -- ?+0 0x34e2680 | |
| 2016-07-26 10:38:06.814615 7f9518370700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af propose 11769) v5 -- ?+0 0x34e3180 | |
| 2016-07-26 10:38:06.814633 7f9518370700 10 mon.615yl02@4(electing) e5 ms_handle_reset 0x33f2520 10.98.3.15:0/4187059021 | |
| 2016-07-26 10:38:06.814638 7f9518370700 10 mon.615yl02@4(electing) e5 ms_handle_reset 0x33f4200 10.98.3.15:0/4187059021 | |
| 2016-07-26 10:38:06.814642 7f9518370700 10 mon.615yl02@4(electing) e5 ms_handle_reset 0x33f2aa0 10.98.3.15:0/4187059021 | |
| 2016-07-26 10:38:06.814645 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:38:06.814648 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:06.814645 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.1 10.98.2.167:6789/0 5 ==== election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af propose 11771) v5 ==== 841+0+0 (3418337519 0 0) 0x34e18c0 con 0x33f2100 | |
| 2016-07-26 10:38:06.814666 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a0fc0 con 0x33f11e0 | |
| 2016-07-26 10:38:06.814710 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:38:06.814712 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:06.814722 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a1d40 con 0x33f11e0 | |
| 2016-07-26 10:38:06.814770 7f9518370700 5 mon.615yl02@4(electing).elector(11769) handle_propose from mon.1 | |
| 2016-07-26 10:38:06.814774 7f9518370700 10 mon.615yl02@4(electing).elector(11769) handle_propose required features 18416819765248, peer features 37154696925806591 | |
| 2016-07-26 10:38:06.814778 7f9518370700 10 mon.615yl02@4(electing).elector(11769) bump_epoch 11769 to 11771 | |
| 2016-07-26 10:38:06.827402 7f9518370700 10 mon.615yl02@4(electing) e5 join_election | |
| 2016-07-26 10:38:06.827407 7f9518370700 10 mon.615yl02@4(electing) e5 _reset | |
| 2016-07-26 10:38:06.827409 7f9518370700 10 mon.615yl02@4(electing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:06.827410 7f9518370700 10 mon.615yl02@4(electing) e5 timecheck_finish | |
| 2016-07-26 10:38:06.827412 7f9518370700 10 mon.615yl02@4(electing) e5 scrub_reset | |
| 2016-07-26 10:38:06.827413 7f9518370700 10 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:06.827417 7f9518370700 10 mon.615yl02@4(electing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:06.827419 7f9518370700 10 mon.615yl02@4(electing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:06.827420 7f9518370700 10 mon.615yl02@4(electing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:06.827421 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:06.827422 7f9518370700 10 mon.615yl02@4(electing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:06.827423 7f9518370700 10 mon.615yl02@4(electing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:06.827426 7f9518370700 5 mon.615yl02@4(electing).elector(11771) defer to 1 | |
| 2016-07-26 10:38:06.827433 7f9518370700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af ack 11771) v5 -- ?+0 0x34e2ec0 | |
| 2016-07-26 10:38:06.827451 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 4 ==== election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af propose 11771) v5 ==== 841+0+0 (3418337519 0 0) 0x34e1b80 con 0x33f1e40 | |
| 2016-07-26 10:38:06.827469 7f9518370700 5 mon.615yl02@4(electing).elector(11771) handle_propose from mon.2 | |
| 2016-07-26 10:38:06.827471 7f9518370700 10 mon.615yl02@4(electing).elector(11771) handle_propose required features 18416819765248, peer features 37154696925806591 | |
| 2016-07-26 10:38:06.827473 7f9518370700 5 mon.615yl02@4(electing).elector(11771) no, we already acked 1 | |
| 2016-07-26 10:38:06.827476 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a0d80 con 0x33f11e0 | |
| 2016-07-26 10:38:06.827481 7f9518370700 10 mon.615yl02@4(electing) e5 do not have session, making new one | |
| 2016-07-26 10:38:06.827486 7f9518370700 10 mon.615yl02@4(electing) e5 ms_dispatch new session MonSession: mon.4 10.98.2.216:6789/0 is open for mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:06.827491 7f9518370700 5 mon.615yl02@4(electing) e5 setting monitor caps on this connection | |
| 2016-07-26 10:38:06.827503 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:06.827513 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:06.827514 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:06.827525 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:06.827527 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:06.827527 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:06.827558 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.1 10.98.2.167:6789/0 6 ==== election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af propose 11771) v5 ==== 841+0+0 (3418337519 0 0) 0x34e2100 con 0x33f2100 | |
| 2016-07-26 10:38:06.827578 7f9518370700 5 mon.615yl02@4(electing).elector(11771) handle_propose from mon.1 | |
| 2016-07-26 10:38:06.827580 7f9518370700 10 mon.615yl02@4(electing).elector(11771) handle_propose required features 18416819765248, peer features 37154696925806591 | |
| 2016-07-26 10:38:06.827595 7f9518370700 5 mon.615yl02@4(electing).elector(11771) defer to 1 | |
| 2016-07-26 10:38:06.827602 7f9518370700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af ack 11771) v5 -- ?+0 0x34e1b80 | |
| 2016-07-26 10:38:06.827615 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a0fc0 con 0x33f11e0 | |
| 2016-07-26 10:38:06.827620 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:06.827631 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:06.827631 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:06.827635 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:06.827637 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:06.827637 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:06.827641 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a1d40 con 0x33f11e0 | |
| 2016-07-26 10:38:06.827646 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:06.827651 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:06.827652 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:06.827655 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:06.827656 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:06.827657 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:06.827660 7f9518370700 1 -- 10.98.2.216:6789/0 <== client.69334 10.98.3.23:0/1011397 1 ==== auth(proto 0 35 bytes epoch 5) v1 ==== 65+0+0 (4078390826 0 0) 0x37a1680 con 0x33f32e0 | |
| 2016-07-26 10:38:06.827675 7f9518370700 5 mon.615yl02@4(electing) e5 discarding message auth(proto 0 35 bytes epoch 5) v1 and sending client elsewhere | |
| 2016-07-26 10:38:06.827678 7f9518370700 1 -- 10.98.2.216:6789/0 mark_down 0x33f32e0 -- 0x3725800 | |
| 2016-07-26 10:38:06.827705 7f9518370700 1 -- 10.98.2.216:6789/0 <== client.216235 10.98.3.18:0/1012640 1 ==== auth(proto 0 35 bytes epoch 5) v1 ==== 65+0+0 (234851919 0 0) 0x37a1b00 con 0x33f40a0 | |
| 2016-07-26 10:38:06.827733 7f9518370700 5 mon.615yl02@4(electing) e5 waitlisting message auth(proto 0 35 bytes epoch 5) v1 | |
| 2016-07-26 10:38:07.894317 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:38:07.894319 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:07.894331 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a1200 con 0x33f11e0 | |
| 2016-07-26 10:38:07.894351 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a1200 con 0x33f11e0 | |
| 2016-07-26 10:38:07.894367 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:38:07.894369 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:07.894378 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a0900 con 0x33f11e0 | |
| 2016-07-26 10:38:07.894387 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:07.894393 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:07.894393 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:07.894398 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:07.894400 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:07.894400 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:07.894405 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a0900 con 0x33f11e0 | |
| 2016-07-26 10:38:07.894409 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:07.894412 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:07.894413 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:07.894419 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:07.894421 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:07.894421 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:08.973346 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:38:08.973348 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:08.973368 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a18c0 con 0x33f11e0 | |
| 2016-07-26 10:38:08.973392 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a18c0 con 0x33f11e0 | |
| 2016-07-26 10:38:08.973412 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:38:08.973415 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:08.973424 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a2400 con 0x33f11e0 | |
| 2016-07-26 10:38:08.973435 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:08.973441 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:08.973441 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:08.973447 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:08.973449 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:08.973450 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:08.973454 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a2400 con 0x33f11e0 | |
| 2016-07-26 10:38:08.973459 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:08.973462 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:08.973464 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:08.973466 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:08.973467 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:08.973468 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:10.052163 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:38:10.052165 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:10.052181 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a2640 con 0x33f11e0 | |
| 2016-07-26 10:38:10.052202 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a2640 con 0x33f11e0 | |
| 2016-07-26 10:38:10.052219 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:38:10.052221 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:10.052242 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a1f80 con 0x33f11e0 | |
| 2016-07-26 10:38:10.052251 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:10.052256 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:10.052257 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:10.052263 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:10.052265 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:10.052265 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:10.052271 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a1f80 con 0x33f11e0 | |
| 2016-07-26 10:38:10.052275 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:10.052279 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:10.052279 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:10.052282 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:10.052283 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:10.052283 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:11.130507 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:38:11.130509 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:11.130526 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a21c0 con 0x33f11e0 | |
| 2016-07-26 10:38:11.130551 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a21c0 con 0x33f11e0 | |
| 2016-07-26 10:38:11.130566 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:38:11.130568 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:11.130577 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a2f40 con 0x33f11e0 | |
| 2016-07-26 10:38:11.130588 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:11.130594 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:11.130594 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:11.130600 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:11.130602 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:11.130602 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:11.130607 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a2f40 con 0x33f11e0 | |
| 2016-07-26 10:38:11.130613 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:11.130616 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:11.130616 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:11.130619 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:11.130621 7f9518370700 5 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:11.130622 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:11.803155 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.1 10.98.2.167:6789/0 7 ==== election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af victory 11772) v5 ==== 26487+0+0 (1879616163 0 0) 0x34e1b80 con 0x33f2100 | |
| 2016-07-26 10:38:11.803176 7f9518370700 5 mon.615yl02@4(electing).elector(11771) handle_victory from mon.1 quorum_features 37154696925806591 | |
| 2016-07-26 10:38:11.803179 7f9518370700 10 mon.615yl02@4(electing).elector(11771) bump_epoch 11771 to 11772 | |
| 2016-07-26 10:38:11.938219 7f9518370700 10 mon.615yl02@4(electing) e5 join_election | |
| 2016-07-26 10:38:11.938224 7f9518370700 10 mon.615yl02@4(electing) e5 _reset | |
| 2016-07-26 10:38:11.938225 7f9518370700 10 mon.615yl02@4(electing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:11.938226 7f9518370700 10 mon.615yl02@4(electing) e5 timecheck_finish | |
| 2016-07-26 10:38:11.938228 7f9518370700 10 mon.615yl02@4(electing) e5 scrub_reset | |
| 2016-07-26 10:38:11.938229 7f9518370700 10 mon.615yl02@4(electing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:11.938232 7f9518370700 10 mon.615yl02@4(electing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:11.938234 7f9518370700 10 mon.615yl02@4(electing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:11.938235 7f9518370700 10 mon.615yl02@4(electing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:11.938237 7f9518370700 10 mon.615yl02@4(electing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:11.938238 7f9518370700 10 mon.615yl02@4(electing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:11.938239 7f9518370700 10 mon.615yl02@4(electing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:11.938247 7f9518370700 10 mon.615yl02@4(peon) e5 lose_election, epoch 11772 leader is mon1 quorum is 1,2,4 features are 37154696925806591 | |
| 2016-07-26 10:38:11.938249 7f9518370700 10 mon.615yl02@4(peon).paxos(paxos recovering c 1318755..1319323) peon_init -- i am a peon | |
| 2016-07-26 10:38:11.938253 7f9518370700 10 mon.615yl02@4(peon).paxosservice(pgmap 538057..538795) election_finished | |
| 2016-07-26 10:38:11.938254 7f9518370700 10 mon.615yl02@4(peon).paxosservice(pgmap 538057..538795) _active - not active | |
| 2016-07-26 10:38:11.938256 7f9518370700 10 mon.615yl02@4(peon).paxosservice(mdsmap 1..1) election_finished | |
| 2016-07-26 10:38:11.938257 7f9518370700 10 mon.615yl02@4(peon).paxosservice(mdsmap 1..1) _active - not active | |
| 2016-07-26 10:38:11.938258 7f9518370700 10 mon.615yl02@4(peon).paxosservice(osdmap 1769..2433) election_finished | |
| 2016-07-26 10:38:11.938259 7f9518370700 10 mon.615yl02@4(peon).paxosservice(osdmap 1769..2433) _active - not active | |
| 2016-07-26 10:38:11.938260 7f9518370700 10 mon.615yl02@4(peon).paxosservice(logm 765975..766716) election_finished | |
| 2016-07-26 10:38:11.938261 7f9518370700 10 mon.615yl02@4(peon).paxosservice(logm 765975..766716) _active - not active | |
| 2016-07-26 10:38:11.938262 7f9518370700 10 mon.615yl02@4(peon).paxosservice(monmap 1..5) election_finished | |
| 2016-07-26 10:38:11.938263 7f9518370700 10 mon.615yl02@4(peon).paxosservice(monmap 1..5) _active - not active | |
| 2016-07-26 10:38:11.938264 7f9518370700 10 mon.615yl02@4(peon).paxosservice(auth 2251..2417) election_finished | |
| 2016-07-26 10:38:11.938264 7f9518370700 10 mon.615yl02@4(peon).paxosservice(auth 2251..2417) _active - not active | |
| 2016-07-26 10:38:11.938266 7f9518370700 10 mon.615yl02@4(peon).data_health(11772) start_epoch epoch 11772 | |
| 2016-07-26 10:38:11.938273 7f9518370700 10 mon.615yl02@4(peon) e5 timecheck_finish | |
| 2016-07-26 10:38:11.938279 7f9518370700 10 mon.615yl02@4(peon) e5 do not have session, making new one | |
| 2016-07-26 10:38:11.938281 7f9518370700 10 mon.615yl02@4(peon) e5 ms_dispatch new session MonSession: client.216235 10.98.3.18:0/1012640 is open for client.216235 10.98.3.18:0/1012640 | |
| 2016-07-26 10:38:11.938288 7f9518370700 10 mon.615yl02@4(peon) e5 setting timeout on session | |
| 2016-07-26 10:38:11.938289 7f9518370700 10 mon.615yl02@4(peon).paxosservice(auth 2251..2417) dispatch auth(proto 0 35 bytes epoch 5) v1 from client.216235 10.98.3.18:0/1012640 | |
| 2016-07-26 10:38:11.938294 7f9518370700 5 mon.615yl02@4(peon).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:11.938295 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:11.938299 7f9518370700 10 mon.615yl02@4(peon).paxosservice(auth 2251..2417) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:11.938301 7f9518370700 5 mon.615yl02@4(peon).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:11.938301 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:11.938305 7f9518370700 10 mon.615yl02@4(peon) e5 resend_routed_requests | |
| 2016-07-26 10:38:11.938308 7f9518370700 10 mon.615yl02@4(peon) e5 register_cluster_logger | |
| 2016-07-26 10:38:12.208685 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch | |
| 2016-07-26 10:38:12.208688 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:12.208703 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a3180 con 0x33f11e0 | |
| 2016-07-26 10:38:12.208722 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a3180 con 0x33f11e0 | |
| 2016-07-26 10:38:12.208741 7f951cab2700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished | |
| 2016-07-26 10:38:12.208743 7f951cab2700 0 log_channel(audit) do_log log to syslog | |
| 2016-07-26 10:38:12.208751 7f951cab2700 1 -- 10.98.2.216:6789/0 --> 10.98.2.216:6789/0 -- log(1 entries) v1 -- ?+0 0x37a2ac0 con 0x33f11e0 | |
| 2016-07-26 10:38:12.208760 7f9518370700 10 mon.615yl02@4(peon).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:12.208765 7f9518370700 5 mon.615yl02@4(peon).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:12.208766 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:12.208771 7f9518370700 10 mon.615yl02@4(peon).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:12.208772 7f9518370700 5 mon.615yl02@4(peon).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:12.208773 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:12.208777 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.4 10.98.2.216:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x37a2ac0 con 0x33f11e0 | |
| 2016-07-26 10:38:12.208782 7f9518370700 10 mon.615yl02@4(peon).paxosservice(logm 765975..766716) dispatch log(1 entries) v1 from mon.4 10.98.2.216:6789/0 | |
| 2016-07-26 10:38:12.208786 7f9518370700 5 mon.615yl02@4(peon).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:12.208786 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:12.208789 7f9518370700 10 mon.615yl02@4(peon).paxosservice(logm 765975..766716) waiting for paxos -> readable (v0) | |
| 2016-07-26 10:38:12.208790 7f9518370700 5 mon.615yl02@4(peon).paxos(paxos recovering c 1318755..1319323) is_readable = 0 - now=2016-07-26 10:38:12.208790 lease_expire=0.000000 has v0 lc 1319323 | |
| 2016-07-26 10:38:21.827340 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 5 ==== mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02) v6 ==== 64+0+0 (3448912970 0 0) 0x34fa080 con 0x33f1e40 | |
| 2016-07-26 10:38:21.827354 7f9518370700 10 mon.615yl02@4(peon) e5 handle_probe mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02) v6 | |
| 2016-07-26 10:38:21.827358 7f9518370700 10 mon.615yl02@4(peon) e5 handle_probe_probe mon.2 10.98.2.173:6789/0mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02) v6 features 37154696925806591 | |
| 2016-07-26 10:38:21.827369 7f9518370700 1 -- 10.98.2.216:6789/0 --> 10.98.2.173:6789/0 -- mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02 quorum 1,2,4 paxos( fc 1318755 lc 1319323 )) v6 -- ?+0 0x34f8f00 con 0x33f1e40 | |
| 2016-07-26 10:38:21.938310 7f9518d71700 1 mon.615yl02@4(peon).paxos(paxos recovering c 1318755..1319323) lease_timeout -- calling new election | |
| 2016-07-26 10:38:21.938314 7f9518d71700 10 mon.615yl02@4(peon) e5 bootstrap | |
| 2016-07-26 10:38:21.938317 7f9518d71700 10 mon.615yl02@4(peon) e5 sync_reset_requester | |
| 2016-07-26 10:38:21.938318 7f9518d71700 10 mon.615yl02@4(peon) e5 unregister_cluster_logger | |
| 2016-07-26 10:38:21.938321 7f9518d71700 10 mon.615yl02@4(peon) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:21.938324 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:21.938324 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:21.938325 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:21.938327 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:21.938329 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:21.938333 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:21.938335 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:21.938336 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:21.938337 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:21.938338 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:21.938339 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:21.938341 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:21.938342 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3487440 after 2 seconds | |
| 2016-07-26 10:38:21.938351 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:21.938354 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34f8c80 | |
| 2016-07-26 10:38:21.938363 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fb200 | |
| 2016-07-26 10:38:21.938372 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34faf80 | |
| 2016-07-26 10:38:21.938380 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fb980 | |
| 2016-07-26 10:38:23.938400 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x3487440 | |
| 2016-07-26 10:38:23.938403 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:23.938405 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:23.938406 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:23.938407 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:23.938409 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:23.938410 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:23.938410 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:23.938412 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:23.938413 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:23.938415 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:23.938416 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:23.938418 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:23.938419 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:23.938420 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:23.938421 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:23.938422 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:23.938423 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x33a0240 after 2 seconds | |
| 2016-07-26 10:38:23.938427 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:23.938429 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fb700 | |
| 2016-07-26 10:38:23.938437 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fb480 | |
| 2016-07-26 10:38:23.938445 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fb200 | |
| 2016-07-26 10:38:23.938453 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34faa80 | |
| 2016-07-26 10:38:25.938481 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x33a0240 | |
| 2016-07-26 10:38:25.938484 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:25.938485 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:25.938486 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:25.938487 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:25.938488 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:25.938488 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:25.938489 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:25.938491 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:25.938492 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:25.938494 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:25.938496 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:25.938497 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:25.938498 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:25.938498 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:25.938499 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:25.938500 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:25.938520 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3487440 after 2 seconds | |
| 2016-07-26 10:38:25.938524 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:25.938526 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fa580 | |
| 2016-07-26 10:38:25.938532 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34f9e00 | |
| 2016-07-26 10:38:25.938545 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fbc00 | |
| 2016-07-26 10:38:25.938553 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34faf80 | |
| 2016-07-26 10:38:27.938577 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x3487440 | |
| 2016-07-26 10:38:27.938580 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:27.938581 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:27.938582 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:27.938583 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:27.938584 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:27.938585 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:27.938586 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:27.938587 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:27.938588 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:27.938590 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:27.938592 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:27.938593 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:27.938594 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:27.938595 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:27.938595 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:27.938596 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:27.938598 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x34872c0 after 2 seconds | |
| 2016-07-26 10:38:27.938602 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:27.938603 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34f8f00 | |
| 2016-07-26 10:38:27.938610 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fad00 | |
| 2016-07-26 10:38:27.938618 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fa300 | |
| 2016-07-26 10:38:27.938627 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fcb00 | |
| 2016-07-26 10:38:29.810611 7f951656d700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x37e9000 sd=16 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f2ec0).accept sd=16 10.98.3.15:57196/0 | |
| 2016-07-26 10:38:29.810926 7f951656d700 10 mon.615yl02@4(probing) e5 ms_verify_authorizer 10.98.3.15:0/4187059021 client protocol 0 | |
| 2016-07-26 10:38:29.811087 7f9518370700 1 -- 10.98.2.216:6789/0 <== client.? 10.98.3.15:0/4187059021 1 ==== auth(proto 0 31 bytes epoch 0) v1 ==== 61+0+0 (535495083 0 0) 0x37a33c0 con 0x33f2ec0 | |
| 2016-07-26 10:38:29.811096 7f9518370700 5 mon.615yl02@4(probing) e5 waitlisting message auth(proto 0 31 bytes epoch 0) v1 | |
| 2016-07-26 10:38:29.938655 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x34872c0 | |
| 2016-07-26 10:38:29.938658 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:29.938659 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:29.938659 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:29.938660 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:29.938661 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:29.938662 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:29.938663 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:29.938664 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:29.938665 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:29.938667 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:29.938669 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:29.938670 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:29.938670 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:29.938671 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:29.938672 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:29.938673 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:29.938675 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3487440 after 2 seconds | |
| 2016-07-26 10:38:29.938678 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:29.938680 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fc880 | |
| 2016-07-26 10:38:29.938688 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fc600 | |
| 2016-07-26 10:38:29.938696 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fc380 | |
| 2016-07-26 10:38:29.938703 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fc100 | |
| 2016-07-26 10:38:30.748806 7f9518d71700 10 mon.615yl02@4(probing).data_health(11772) service_tick | |
| 2016-07-26 10:38:30.748818 7f9518d71700 0 mon.615yl02@4(probing).data_health(11772) update_stats avail 94% total 2015 GB, used 4665 MB, avail 1908 GB | |
| 2016-07-26 10:38:31.938732 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x3487440 | |
| 2016-07-26 10:38:31.938735 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:31.938736 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:31.938737 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:31.938738 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:31.938739 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:31.938740 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:31.938741 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:31.938742 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:31.938743 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:31.938745 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:31.938747 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:31.938748 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:31.938749 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:31.938750 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:31.938750 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:31.938752 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:31.938753 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3487350 after 2 seconds | |
| 2016-07-26 10:38:31.938758 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:31.938759 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fbe80 | |
| 2016-07-26 10:38:31.938767 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fdc80 | |
| 2016-07-26 10:38:31.938775 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fda00 | |
| 2016-07-26 10:38:31.938783 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fd780 | |
| 2016-07-26 10:38:31.938791 7f9518d71700 10 mon.615yl02@4(probing) e5 trimming session 0x33f40a0 client.216235 10.98.3.18:0/1012640 because we've been out of quorum too long | |
| 2016-07-26 10:38:31.938795 7f9518d71700 1 -- 10.98.2.216:6789/0 mark_down 0x33f40a0 -- 0x37ed800 | |
| 2016-07-26 10:38:31.938817 7f9518d71700 10 mon.615yl02@4(probing) e5 remove_session MonSession: client.216235 10.98.3.18:0/1012640 is open client.216235 10.98.3.18:0/1012640 | |
| 2016-07-26 10:38:31.938826 7f9518d71700 5 mon.615yl02@4(probing) e5 waitlisting message auth(proto 0 31 bytes epoch 0) v1 | |
| 2016-07-26 10:38:32.810697 7f9518370700 10 mon.615yl02@4(probing) e5 ms_handle_reset 0x33f2ec0 10.98.3.15:0/4187059021 | |
| 2016-07-26 10:38:33.938811 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x3487350 | |
| 2016-07-26 10:38:33.938814 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:33.938815 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:33.938816 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:33.938817 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:33.938818 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:33.938820 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:33.938820 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:33.938821 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:33.938822 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:33.938825 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:33.938826 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:33.938827 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:33.938828 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:33.938829 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:33.938830 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:33.938831 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:33.938832 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x33a0240 after 2 seconds | |
| 2016-07-26 10:38:33.938837 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:33.938838 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fd500 | |
| 2016-07-26 10:38:33.938846 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fd280 | |
| 2016-07-26 10:38:33.938854 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fd000 | |
| 2016-07-26 10:38:33.938861 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fcd80 | |
| 2016-07-26 10:38:35.938890 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x33a0240 | |
| 2016-07-26 10:38:35.938893 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:35.938894 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:35.938895 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:35.938896 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:35.938897 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:35.938898 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:35.938899 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:35.938900 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:35.938900 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:35.938903 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:35.938904 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:35.938905 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:35.938906 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:35.938907 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:35.938908 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:35.938909 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:35.938910 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3487350 after 2 seconds | |
| 2016-07-26 10:38:35.938914 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:35.938916 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fad00 | |
| 2016-07-26 10:38:35.938923 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34f9e00 | |
| 2016-07-26 10:38:35.938930 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fb480 | |
| 2016-07-26 10:38:35.938938 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fda00 | |
| 2016-07-26 10:38:36.938888 7f9518d71700 5 mon.615yl02@4(probing) e5 discarding message auth(proto 0 31 bytes epoch 0) v1 and sending client elsewhere | |
| 2016-07-26 10:38:36.938893 7f9518d71700 1 -- 10.98.2.216:6789/0 mark_down 0x33f2ec0 -- pipe dne | |
| 2016-07-26 10:38:37.740484 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.1 10.98.2.167:6789/0 8 ==== paxos(collect lc 1319324 fc 1318755 pn 470401 opn 0) v3 ==== 80+0+0 (2660417298 0 0) 0x34f9e00 con 0x33f2100 | |
| 2016-07-26 10:38:37.740494 7f9518370700 5 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) election in progress, dropping paxos(collect lc 1319324 fc 1318755 pn 470401 opn 0) v3 | |
| 2016-07-26 10:38:37.740527 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.1 10.98.2.167:6789/0 9 ==== time_check( ping e 11772 r 1 ) v1 ==== 36+0+0 (2044766158 0 0) 0x37a0480 con 0x33f2100 | |
| 2016-07-26 10:38:37.740538 7f9518370700 10 mon.615yl02@4(probing) e5 handle_timecheck time_check( ping e 11772 r 1 ) v1 | |
| 2016-07-26 10:38:37.740540 7f9518370700 1 mon.615yl02@4(probing) e5 handle_timecheck drop unexpected msg | |
| 2016-07-26 10:38:37.938967 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x3487350 | |
| 2016-07-26 10:38:37.938970 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:37.938971 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:37.938971 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:37.938972 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:37.938977 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:37.938978 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:37.938978 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:37.938979 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:37.938980 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:37.938982 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:37.938984 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:37.938985 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:37.938986 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:37.938987 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:37.938988 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:37.938989 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:37.938990 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3487440 after 2 seconds | |
| 2016-07-26 10:38:37.938994 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:37.938996 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fc380 | |
| 2016-07-26 10:38:37.939003 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fa300 | |
| 2016-07-26 10:38:37.939011 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fbc00 | |
| 2016-07-26 10:38:37.939019 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fb200 | |
| 2016-07-26 10:38:39.939048 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x3487440 | |
| 2016-07-26 10:38:39.939051 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:39.939052 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:39.939054 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:39.939055 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:39.939056 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:39.939057 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:39.939057 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:39.939059 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:39.939059 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:39.939062 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:39.939063 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:39.939064 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:39.939065 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:39.939066 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:39.939067 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:39.939068 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:39.939069 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3487350 after 2 seconds | |
| 2016-07-26 10:38:39.939073 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:39.939076 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fe680 | |
| 2016-07-26 10:38:39.939083 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fe400 | |
| 2016-07-26 10:38:39.939091 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fe180 | |
| 2016-07-26 10:38:39.939099 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fdf00 | |
| 2016-07-26 10:38:40.559288 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 6 ==== election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af propose 11773) v5 ==== 841+0+0 (2331296059 0 0) 0x34e2680 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559440 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 7 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34fe180 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559448 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559451 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559456 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:40.559482 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:40.559484 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:40.559485 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:40.559487 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:40.559491 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 8 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34fbc00 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559498 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559501 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559505 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:40.559513 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:40.559514 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:40.559516 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:40.559517 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:40.559553 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 9 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34fb480 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559560 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559563 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559567 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:40.559575 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:40.559576 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:40.559577 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:40.559578 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:40.559581 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 10 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34fd000 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559588 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559590 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559594 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:40.559608 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:40.559609 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:40.559610 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:40.559611 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:40.559640 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 11 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34ff300 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559647 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559650 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559654 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:40.559663 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:40.559665 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:40.559666 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:40.559667 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:40.559690 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 12 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34ff080 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559697 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559700 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559704 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:40.559712 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:40.559714 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:40.559715 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:40.559716 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:40.559725 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 13 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34fee00 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559732 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559735 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559739 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:40.559747 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:40.559748 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:40.559749 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:40.559750 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:40.559753 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 14 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34feb80 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559760 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559763 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559766 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:40.559774 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:40.559776 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:40.559777 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:40.559778 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:40.559781 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 15 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34fe900 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559787 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559790 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559794 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:40.559802 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:40.559804 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:40.559805 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:40.559806 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:40.559809 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 16 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34ff800 con 0x33f1e40 | |
| 2016-07-26 10:38:40.559815 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559817 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:40.559821 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:40.559829 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:40.559830 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:40.559831 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:40.559832 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:41.939121 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x3487350 | |
| 2016-07-26 10:38:41.939124 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:41.939125 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:41.939126 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:41.939127 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:41.939128 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:41.939129 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:41.939130 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:41.939132 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:41.939133 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:41.939135 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:41.939137 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:41.939138 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:41.939139 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:41.939140 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:41.939141 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:41.939142 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:41.939143 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x33a0240 after 2 seconds | |
| 2016-07-26 10:38:41.939148 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:41.939150 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fdc80 | |
| 2016-07-26 10:38:41.939158 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fc600 | |
| 2016-07-26 10:38:41.939165 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34f9e00 | |
| 2016-07-26 10:38:41.939172 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fa080 | |
| 2016-07-26 10:38:41.939401 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 17 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34f9e00 con 0x33f1e40 | |
| 2016-07-26 10:38:41.939409 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:41.939412 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:41.939416 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:41.939431 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:41.939432 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:41.939434 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:41.939435 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:43.939201 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x33a0240 | |
| 2016-07-26 10:38:43.939204 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:43.939222 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:43.939223 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:43.939224 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:43.939225 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:43.939226 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:43.939227 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:43.939228 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:43.939229 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:43.939231 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:43.939233 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:43.939234 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:43.939235 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:43.939236 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:43.939237 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:43.939238 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:43.939239 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3487350 after 2 seconds | |
| 2016-07-26 10:38:43.939244 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:43.939245 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fa800 | |
| 2016-07-26 10:38:43.939252 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34ffd00 | |
| 2016-07-26 10:38:43.939260 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34ffa80 | |
| 2016-07-26 10:38:43.939267 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fe900 | |
| 2016-07-26 10:38:43.939486 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 18 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34ffa80 con 0x33f1e40 | |
| 2016-07-26 10:38:43.939494 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:43.939497 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:43.939502 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:43.939511 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:43.939513 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:43.939514 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:43.939515 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:45.385054 7f951656d700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x371c800 sd=16 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f4fc0).accept sd=16 10.98.3.23:53146/0 | |
| 2016-07-26 10:38:45.385214 7f951656d700 10 mon.615yl02@4(probing) e5 ms_verify_authorizer 10.98.3.23:0/1011397 client protocol 0 | |
| 2016-07-26 10:38:45.385386 7f9518370700 1 -- 10.98.2.216:6789/0 <== client.69334 10.98.3.23:0/1011397 1 ==== auth(proto 0 35 bytes epoch 5) v1 ==== 65+0+0 (4078390826 0 0) 0x37a3a80 con 0x33f4fc0 | |
| 2016-07-26 10:38:45.385395 7f9518370700 5 mon.615yl02@4(probing) e5 waitlisting message auth(proto 0 35 bytes epoch 5) v1 | |
| 2016-07-26 10:38:45.582287 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 19 ==== election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af propose 11773) v5 ==== 841+0+0 (2331296059 0 0) 0x34e18c0 con 0x33f1e40 | |
| 2016-07-26 10:38:45.939297 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x3487350 | |
| 2016-07-26 10:38:45.939299 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:45.939300 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:45.939302 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:45.939302 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:45.939304 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:45.939304 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:45.939305 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:45.939307 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:45.939308 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:45.939311 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:45.939313 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:45.939314 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:45.939315 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:45.939315 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:45.939316 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:45.939317 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:45.939319 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x33a0240 after 2 seconds | |
| 2016-07-26 10:38:45.939323 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:45.939324 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34feb80 | |
| 2016-07-26 10:38:45.939330 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fee00 | |
| 2016-07-26 10:38:45.939337 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34ff080 | |
| 2016-07-26 10:38:45.939345 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34ff300 | |
| 2016-07-26 10:38:45.939565 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 20 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34ff080 con 0x33f1e40 | |
| 2016-07-26 10:38:45.939573 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:45.939576 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:45.939581 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:45.939590 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:45.939592 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:45.939594 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:45.939595 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:46.939021 7f9518d71700 5 mon.615yl02@4(probing) e5 waitlisting message auth(proto 0 35 bytes epoch 5) v1 | |
| 2016-07-26 10:38:47.939376 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x33a0240 | |
| 2016-07-26 10:38:47.939378 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:47.939379 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:47.939380 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:47.939381 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:47.939382 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:47.939383 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:47.939384 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:47.939385 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:47.939386 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:47.939388 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:47.939390 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:47.939390 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:47.939391 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:47.939392 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:47.939393 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:47.939394 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:47.939395 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3487440 after 2 seconds | |
| 2016-07-26 10:38:47.939399 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:47.939402 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fee00 | |
| 2016-07-26 10:38:47.939408 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34ffd00 | |
| 2016-07-26 10:38:47.939416 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fc600 | |
| 2016-07-26 10:38:47.939423 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fe400 | |
| 2016-07-26 10:38:47.939650 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 21 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x34fc600 con 0x33f1e40 | |
| 2016-07-26 10:38:47.939658 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:47.939661 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:47.939665 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:47.939674 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:47.939676 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:47.939677 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:47.939678 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:49.939454 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x3487440 | |
| 2016-07-26 10:38:49.939457 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:49.939458 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:49.939459 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:49.939459 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:49.939461 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:49.939461 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:49.939463 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:49.939465 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:49.939466 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:49.939468 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:49.939470 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:49.939471 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:49.939472 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:49.939473 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:49.939473 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:49.939475 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:49.939476 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x33a0240 after 2 seconds | |
| 2016-07-26 10:38:49.939480 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:49.939482 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fa300 | |
| 2016-07-26 10:38:49.939488 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x34fd280 | |
| 2016-07-26 10:38:49.939515 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x38d1b80 | |
| 2016-07-26 10:38:49.939522 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x38d1900 | |
| 2016-07-26 10:38:49.939737 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 22 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x38d1b80 con 0x33f1e40 | |
| 2016-07-26 10:38:49.939745 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:49.939748 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:49.939757 7f9518370700 10 mon.615yl02@4(probing) e5 monmap is e5: 5 mons at {60z0m02=10.98.2.166:6789/0,60zxl02=10.98.2.167:6789/0,610wl02=10.98.2.173:6789/0,615yl02=10.98.2.216:6789/0,618yl02=10.98.2.214:6789/0} | |
| 2016-07-26 10:38:49.939766 7f9518370700 10 mon.615yl02@4(probing) e5 peer name is 610wl02 | |
| 2016-07-26 10:38:49.939768 7f9518370700 10 mon.615yl02@4(probing) e5 mon.610wl02 is outside the quorum | |
| 2016-07-26 10:38:49.939769 7f9518370700 10 mon.615yl02@4(probing) e5 outside_quorum now 610wl02,615yl02, need 3 | |
| 2016-07-26 10:38:49.939770 7f9518370700 10 mon.615yl02@4(probing) e5 that's not yet enough for a new quorum, waiting | |
| 2016-07-26 10:38:50.605279 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 23 ==== election(2f51a247-3155-4bcf-9aee-c6f6b2c5e2af propose 11773) v5 ==== 841+0+0 (2331296059 0 0) 0x34e2c00 con 0x33f1e40 | |
| 2016-07-26 10:38:50.812451 7f9515061700 1 -- 10.98.2.216:6789/0 >> :/0 pipe(0x3725800 sd=18 :6789 s=0 pgs=0 cs=0 l=0 c=0x33f4e60).accept sd=18 10.98.3.15:57231/0 | |
| 2016-07-26 10:38:50.812527 7f9515061700 10 mon.615yl02@4(probing) e5 ms_verify_authorizer 10.98.3.15:0/4187059021 client protocol 0 | |
| 2016-07-26 10:38:50.812665 7f9518370700 1 -- 10.98.2.216:6789/0 <== client.? 10.98.3.15:0/4187059021 1 ==== auth(proto 0 31 bytes epoch 0) v1 ==== 61+0+0 (535495083 0 0) 0x37a3cc0 con 0x33f4e60 | |
| 2016-07-26 10:38:50.812673 7f9518370700 5 mon.615yl02@4(probing) e5 waitlisting message auth(proto 0 31 bytes epoch 0) v1 | |
| 2016-07-26 10:38:51.939084 7f9518d71700 5 mon.615yl02@4(probing) e5 discarding message auth(proto 0 35 bytes epoch 5) v1 and sending client elsewhere | |
| 2016-07-26 10:38:51.939088 7f9518d71700 1 -- 10.98.2.216:6789/0 mark_down 0x33f4fc0 -- 0x371c800 | |
| 2016-07-26 10:38:51.939108 7f9518d71700 5 mon.615yl02@4(probing) e5 waitlisting message auth(proto 0 31 bytes epoch 0) v1 | |
| 2016-07-26 10:38:51.939529 7f9518d71700 4 mon.615yl02@4(probing) e5 probe_timeout 0x33a0240 | |
| 2016-07-26 10:38:51.939530 7f9518d71700 10 mon.615yl02@4(probing) e5 bootstrap | |
| 2016-07-26 10:38:51.939531 7f9518d71700 10 mon.615yl02@4(probing) e5 sync_reset_requester | |
| 2016-07-26 10:38:51.939532 7f9518d71700 10 mon.615yl02@4(probing) e5 unregister_cluster_logger - not registered | |
| 2016-07-26 10:38:51.939533 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:51.939537 7f9518d71700 10 mon.615yl02@4(probing) e5 _reset | |
| 2016-07-26 10:38:51.939538 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:51.939539 7f9518d71700 10 mon.615yl02@4(probing) e5 timecheck_finish | |
| 2016-07-26 10:38:51.939541 7f9518d71700 10 mon.615yl02@4(probing) e5 scrub_reset | |
| 2016-07-26 10:38:51.939542 7f9518d71700 10 mon.615yl02@4(probing).paxos(paxos recovering c 1318755..1319323) restart -- canceling timeouts | |
| 2016-07-26 10:38:51.939545 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(pgmap 538057..538795) restart | |
| 2016-07-26 10:38:51.939546 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(mdsmap 1..1) restart | |
| 2016-07-26 10:38:51.939547 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(osdmap 1769..2433) restart | |
| 2016-07-26 10:38:51.939548 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(logm 765975..766716) restart | |
| 2016-07-26 10:38:51.939549 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(monmap 1..5) restart | |
| 2016-07-26 10:38:51.939550 7f9518d71700 10 mon.615yl02@4(probing).paxosservice(auth 2251..2417) restart | |
| 2016-07-26 10:38:51.939551 7f9518d71700 10 mon.615yl02@4(probing) e5 cancel_probe_timeout (none scheduled) | |
| 2016-07-26 10:38:51.939552 7f9518d71700 10 mon.615yl02@4(probing) e5 reset_probe_timeout 0x3487350 after 2 seconds | |
| 2016-07-26 10:38:51.939556 7f9518d71700 10 mon.615yl02@4(probing) e5 probing other monitors | |
| 2016-07-26 10:38:51.939558 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.0 10.98.2.166:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x38d1680 | |
| 2016-07-26 10:38:51.939564 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.1 10.98.2.167:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x38d1400 | |
| 2016-07-26 10:38:51.939571 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.2 10.98.2.173:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x38d1180 | |
| 2016-07-26 10:38:51.939579 7f9518d71700 1 -- 10.98.2.216:6789/0 --> mon.3 10.98.2.214:6789/0 -- mon_probe(probe 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 615yl02) v6 -- ?+0 0x38d0f00 | |
| 2016-07-26 10:38:51.939797 7f9518370700 1 -- 10.98.2.216:6789/0 <== mon.2 10.98.2.173:6789/0 24 ==== mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 ==== 845+0+0 (3613516349 0 0) 0x38d1180 con 0x33f1e40 | |
| 2016-07-26 10:38:51.939805 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 | |
| 2016-07-26 10:38:51.939808 7f9518370700 10 mon.615yl02@4(probing) e5 handle_probe_reply mon.2 10.98.2.173:6789/0mon_probe(reply 2f51a247-3155-4bcf-9aee-c6f6b2c5e2af name 610wl02 paxos( fc 1318755 lc 1319324 )) v6 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment