gistfile1.txt

2021.03.24 11:22:38.905603 [ 1084 ] {} <Information> RaftInstance: Raft ASIO listener initiated, UNSECURED
2021.03.24 11:22:38.905657 [ 1084 ] {} <Information> RaftInstance: parameters: timeout 1000 - 2000, heartbeat 500, leadership expiry 10000, max batch 100, backoff 50, snapshot distance 100, log sync stop gap 99999, reserved logs 10, client timeout 10000, auto forwarding ON, API call type BLOCKING, custom commit quorum size 0, custom election quorum size 0, snapshot receiver INCLUDED, leadership transfer wait time 0
2021.03.24 11:22:38.905669 [ 1084 ] {} <Information> RaftInstance: new timeout range: 1000 -- 2000
2021.03.24 11:22:38.905687 [ 1084 ] {} <Information> RaftInstance:    === INIT RAFT SERVER ===
2021.03.24 11:22:38.905746 [ 1084 ] {} <Trace> RaftInstance: asio client created: 0x7faaa9b7a618
2021.03.24 11:22:38.905792 [ 1084 ] {} <Trace> RaftInstance: asio client created: 0x7faaa6751018
2021.03.24 11:22:38.905808 [ 1084 ] {} <Information> RaftInstance: peer 1: DC ID 0, ch-iva.iva.yp-c.yandex.net:9444, voting member, 1
2021.03.24 11:22:38.905817 [ 1084 ] {} <Information> RaftInstance: global manager does not exist. will use local thread for commit and append
2021.03.24 11:22:38.905945 [ 1084 ] {} <Information> RaftInstance: wait for HB, for 50 + [1000, 2000] ms
2021.03.24 11:22:38.906021 [ 1097 ] {} <Information> RaftInstance: bg append_entries thread initiated
2021.03.24 11:22:38.906074 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:38.956149 [ 1084 ] {} <Trace> RaftInstance: re-schedule election timer
2021.03.24 11:22:38.956219 [ 1084 ] {} <Debug> RaftInstance: server 2 started
2021.03.24 11:22:38.956294 [ 1084 ] {} <Trace> RaftInstance: asio rpc session created: 0x7faaa6751a18
2021.03.24 11:22:39.974479 [ 1093 ] {} <Trace> RaftInstance: election timeout
2021.03.24 11:22:39.974551 [ 1093 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2021.03.24 11:22:39.974594 [ 1093 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
2021.03.24 11:22:39.974627 [ 1093 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
2021.03.24 11:22:39.974648 [ 1093 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 2, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
2021.03.24 11:22:39.974659 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type pre_vote_request
2021.03.24 11:22:39.974674 [ 1093 ] {} <Debug> RaftInstance: socket 0x7faaa6751018 to ch-man.man.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:39.974809 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type pre_vote_request
2021.03.24 11:22:39.974824 [ 1093 ] {} <Debug> RaftInstance: socket 0x7faaa9b7a618 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:39.974834 [ 1093 ] {} <Trace> RaftInstance: cancel existing timer
2021.03.24 11:22:39.974842 [ 1093 ] {} <Trace> RaftInstance: re-schedule election timer
2021.03.24 11:22:39.985887 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type pre_vote_request, failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:39.985951 [ 1093 ] {} <Warning> RaftInstance: peer (1) response error: failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:39.985990 [ 1093 ] {} <Trace> RaftInstance: asio client destroyed: 0x7faaa9b7a618
2021.03.24 11:22:39.996813 [ 1093 ] {} <Information> RaftInstance: 0x7faaa6751018 connected to ch-man.man.yp-c.yandex.net:9444 (as a client)
2021.03.24 11:22:40.017725 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type pre_vote_request, OK
2021.03.24 11:22:40.017813 [ 1095 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 3 with Result=1, Term=0, NextIndex=1
2021.03.24 11:22:40.017869 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 0
2021.03.24 11:22:40.017896 [ 1095 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 3 (O), term 0, resp term 0, my role follower, dead 2, live 0, num voting members 3, quorum 2
2021.03.24 11:22:40.017905 [ 1095 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0
2021.03.24 11:22:40.017913 [ 1095 ] {} <Information> RaftInstance: [PRE-VOTE DONE] initiate actual vote
2021.03.24 11:22:40.017926 [ 1095 ] {} <Information> RaftInstance: [VOTE INIT] my id 2, my role candidate, term 1, log idx 0, log term 0, priority (target 1 / mine 1)
2021.03.24 11:22:40.017944 [ 1095 ] {} <Debug> RaftInstance: send request_vote_request to server 3 with term 1
2021.03.24 11:22:40.017953 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type request_vote_request
2021.03.24 11:22:40.018035 [ 1095 ] {} <Debug> RaftInstance: send request_vote_request to server 1 with term 1
2021.03.24 11:22:40.018048 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type request_vote_request
2021.03.24 11:22:40.018055 [ 1095 ] {} <Trace> RaftInstance: rpc local is null
2021.03.24 11:22:40.018064 [ 1095 ] {} <Trace> RaftInstance: cancel existing timer
2021.03.24 11:22:40.018176 [ 1095 ] {} <Trace> RaftInstance: re-schedule election timer
2021.03.24 11:22:40.038591 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type request_vote_request, OK
2021.03.24 11:22:40.038662 [ 1092 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 3 with Result=1, Term=1, NextIndex=1
2021.03.24 11:22:40.038687 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:40.038701 [ 1092 ] {} <Information> RaftInstance: [VOTE RESP] peer 3 (O), resp term 1, my role candidate, granted 2, responded 2, num voting members 3, quorum 2
2021.03.24 11:22:40.038710 [ 1092 ] {} <Information> RaftInstance: Server is elected as leader for term 1
2021.03.24 11:22:40.038726 [ 1092 ] {} <Information> RaftInstance: number of pending commit elements: 0
2021.03.24 11:22:40.038735 [ 1092 ] {} <Information> RaftInstance: state machine commit index 0, precommit index 0, last log index 0
2021.03.24 11:22:40.038744 [ 1092 ] {} <Trace> RaftInstance: peer 3, interval: 500
2021.03.24 11:22:40.038760 [ 1092 ] {} <Trace> RaftInstance: peer 1, interval: 500
2021.03.24 11:22:40.038806 [ 1092 ] {} <Information> RaftInstance: [BECOME LEADER] appended new config at 1
2021.03.24 11:22:40.068874 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:40.068939 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:40.068997 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0, Term=1, peer_last_sent_idx 0
2021.03.24 11:22:40.069007 [ 1092 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:40.069019 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:40.069163 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:40.069180 [ 1092 ] {} <Debug> RaftInstance: reset RPC client for peer 1
2021.03.24 11:22:40.069266 [ 1092 ] {} <Trace> RaftInstance: asio client created: 0x7faa1f071018
2021.03.24 11:22:40.069277 [ 1092 ] {} <Trace> RaftInstance: 0x7faa1f071018 reconnect peer 1
2021.03.24 11:22:40.069284 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:40.069292 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:40.069302 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0, Term=1, peer_last_sent_idx 0
2021.03.24 11:22:40.069309 [ 1092 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:40.069318 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:40.069330 [ 1092 ] {} <Debug> RaftInstance: socket 0x7faa1f071018 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:40.069363 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:40.069373 [ 1092 ] {} <Information> RaftInstance:   === LEADER (term 1) ===
2021.03.24 11:22:40.076665 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:40.076732 [ 1094 ] {} <Warning> RaftInstance: peer (1) response error: failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:40.076791 [ 1094 ] {} <Trace> RaftInstance: asio client destroyed: 0x7faa1f071018
2021.03.24 11:22:40.154090 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:40.154152 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:40.154163 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:40.154179 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:40.154187 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:40.154195 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 0, new matched idx: 1
2021.03.24 11:22:40.154208 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:40.154216 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 1
2021.03.24 11:22:40.154225 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:40.154233 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:40.154244 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:40.154251 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:40.154260 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:40.154338 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:40.154352 [ 1094 ] {} <Debug> RaftInstance: reset RPC client for peer 1
2021.03.24 11:22:40.154387 [ 1094 ] {} <Trace> RaftInstance: asio client created: 0x7faa1f071018
2021.03.24 11:22:40.154396 [ 1094 ] {} <Trace> RaftInstance: 0x7faa1f071018 reconnect peer 1
2021.03.24 11:22:40.154404 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:40.154412 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:40.154421 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:40.154429 [ 1094 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:40.154437 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:40.154456 [ 1094 ] {} <Debug> RaftInstance: socket 0x7faa1f071018 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:40.154471 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:40.154481 [ 1094 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 0
2021.03.24 11:22:40.154497 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:40.154558 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:40.154598 [ 1096 ] {} <Debug> RaftInstance: commit upto 1, current idx 0
2021.03.24 11:22:40.154610 [ 1096 ] {} <Trace> RaftInstance: commit upto 1, current idx 1
2021.03.24 11:22:40.154655 [ 1096 ] {} <Information> RaftInstance: config at index 1 is committed, prev config log idx 0
2021.03.24 11:22:40.154690 [ 1096 ] {} <Information> RaftInstance: new config log idx 1, prev log idx 0, cur config log idx 0, prev log idx 0
2021.03.24 11:22:40.154715 [ 1096 ] {} <Debug> RaftInstance: system is reconfigured to have 3 servers, last config index: 0, this config index: 1
2021.03.24 11:22:40.154738 [ 1096 ] {} <Information> RaftInstance: new configuration: log idx 1, prev log idx 0
2021.03.24 11:22:40.154792 [ 1096 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:40.154806 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 1, current idx 1
2021.03.24 11:22:40.155368 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:40.161614 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:40.161677 [ 1094 ] {} <Warning> RaftInstance: peer (1) response error: failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:40.161700 [ 1094 ] {} <Trace> RaftInstance: asio client destroyed: 0x7faa1f071018
2021.03.24 11:22:40.189634 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:40.210400 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:40.210428 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:40.210441 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:40.210450 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:40.210458 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:40.210471 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:40.210479 [ 1095 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:40.538930 [ 1092 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:40.538991 [ 1092 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:40.539001 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:40.539010 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:40.539023 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:40.539031 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:40.539040 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:40.539149 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:40.539218 [ 1092 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:40.539230 [ 1092 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:40.539240 [ 1092 ] {} <Debug> RaftInstance: reset RPC client for peer 1
2021.03.24 11:22:40.539342 [ 1092 ] {} <Trace> RaftInstance: asio client created: 0x7faa1f071a18
2021.03.24 11:22:40.539353 [ 1092 ] {} <Trace> RaftInstance: 0x7faa1f071a18 reconnect peer 1
2021.03.24 11:22:40.539360 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:40.539368 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:40.539378 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:40.539386 [ 1092 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:40.539395 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:40.539404 [ 1092 ] {} <Debug> RaftInstance: socket 0x7faa1f071a18 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:40.539427 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:40.547609 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:40.547673 [ 1093 ] {} <Warning> RaftInstance: peer (1) response error: failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:40.547696 [ 1093 ] {} <Trace> RaftInstance: asio client destroyed: 0x7faa1f071a18
2021.03.24 11:22:40.559814 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:40.559877 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:40.559888 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:40.559900 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:40.559908 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:40.559916 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:40.559929 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:40.559966 [ 1093 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:41.039259 [ 1093 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:41.039307 [ 1093 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:41.039317 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:41.039327 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:41.039340 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:41.039348 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:41.039358 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:41.039443 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:41.060044 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:41.060083 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:41.060094 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:41.060105 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:41.060113 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:41.060135 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:41.060147 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:41.060157 [ 1095 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:41.139534 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:41.139588 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:41.139600 [ 1094 ] {} <Debug> RaftInstance: reset RPC client for peer 1
2021.03.24 11:22:41.139636 [ 1094 ] {} <Trace> RaftInstance: asio client created: 0x7faa1f071018
2021.03.24 11:22:41.139647 [ 1094 ] {} <Trace> RaftInstance: 0x7faa1f071018 reconnect peer 1
2021.03.24 11:22:41.139678 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:41.139688 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:41.139702 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:41.139709 [ 1094 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:41.139719 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:41.139729 [ 1094 ] {} <Debug> RaftInstance: socket 0x7faa1f071018 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:41.139746 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:41.147196 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:41.147238 [ 1094 ] {} <Warning> RaftInstance: peer (1) response error: failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:41.147259 [ 1094 ] {} <Trace> RaftInstance: asio client destroyed: 0x7faa1f071018
2021.03.24 11:22:41.539683 [ 1095 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:41.539732 [ 1095 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:41.539743 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:41.539752 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:41.539764 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:41.539772 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:41.539782 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:41.539868 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:41.560540 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:41.560601 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:41.560612 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:41.560624 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:41.560632 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:41.560640 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:41.560652 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:41.560661 [ 1093 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:41.789938 [ 1093 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:41.789997 [ 1093 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:41.790008 [ 1093 ] {} <Debug> RaftInstance: reset RPC client for peer 1
2021.03.24 11:22:41.790045 [ 1093 ] {} <Trace> RaftInstance: asio client created: 0x7faa1f071a18
2021.03.24 11:22:41.790056 [ 1093 ] {} <Trace> RaftInstance: 0x7faa1f071a18 reconnect peer 1
2021.03.24 11:22:41.790064 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:41.790073 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:41.790086 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:41.790094 [ 1093 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:41.790104 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:41.790114 [ 1093 ] {} <Debug> RaftInstance: socket 0x7faa1f071a18 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:41.790131 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:41.798274 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:41.798322 [ 1095 ] {} <Warning> RaftInstance: peer (1) response error: failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:41.798354 [ 1095 ] {} <Trace> RaftInstance: asio client destroyed: 0x7faa1f071a18
2021.03.24 11:22:42.040063 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:42.040131 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:42.040142 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:42.040164 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:42.040176 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:42.040184 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:42.040194 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:42.040298 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:42.060934 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:42.061016 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:42.061040 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:42.061052 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:42.061059 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:42.061068 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:42.061080 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:42.061088 [ 1094 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:42.490322 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:42.490379 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:42.490391 [ 1094 ] {} <Debug> RaftInstance: reset RPC client for peer 1
2021.03.24 11:22:42.490427 [ 1094 ] {} <Trace> RaftInstance: asio client created: 0x7faa1f071018
2021.03.24 11:22:42.490438 [ 1094 ] {} <Trace> RaftInstance: 0x7faa1f071018 reconnect peer 1
2021.03.24 11:22:42.490446 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:42.490455 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:42.490468 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:42.490476 [ 1094 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:42.490487 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:42.490498 [ 1094 ] {} <Debug> RaftInstance: socket 0x7faa1f071018 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:42.490514 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:42.499958 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:42.500011 [ 1094 ] {} <Warning> RaftInstance: peer (1) response error: failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:42.500033 [ 1094 ] {} <Trace> RaftInstance: asio client destroyed: 0x7faa1f071018
2021.03.24 11:22:42.540432 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:42.540486 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:42.540497 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:42.540506 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:42.540542 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:42.540550 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:42.540560 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:42.540653 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:42.561459 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:42.561522 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:42.561533 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:42.561544 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:42.561551 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:42.561560 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:42.561572 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:42.561581 [ 1094 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:43.040807 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:43.040853 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:43.040864 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:43.040874 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:43.040887 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:43.040895 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:43.040904 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:43.040984 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:43.061605 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:43.061658 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:43.061669 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:43.061681 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:43.061689 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:43.061697 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:43.061710 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:43.061719 [ 1094 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:43.240647 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:43.240705 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:43.240717 [ 1094 ] {} <Debug> RaftInstance: reset RPC client for peer 1
2021.03.24 11:22:43.240754 [ 1094 ] {} <Trace> RaftInstance: asio client created: 0x7faa1f071018
2021.03.24 11:22:43.240765 [ 1094 ] {} <Trace> RaftInstance: 0x7faa1f071018 reconnect peer 1
2021.03.24 11:22:43.240773 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:43.240781 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:43.240793 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:43.240800 [ 1094 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:43.240810 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:43.240845 [ 1094 ] {} <Debug> RaftInstance: socket 0x7faa1f071018 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:43.240863 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:43.248461 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:43.248506 [ 1095 ] {} <Warning> RaftInstance: peer (1) response error: failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:43.248529 [ 1095 ] {} <Trace> RaftInstance: asio client destroyed: 0x7faa1f071018
2021.03.24 11:22:43.541195 [ 1093 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:43.541251 [ 1093 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:43.541261 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:43.541271 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:43.541285 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:43.541293 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:43.541303 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:43.541411 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:43.562268 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:43.562331 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:43.562343 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:43.562355 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:43.562363 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:43.562371 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:43.562397 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:43.562407 [ 1094 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:43.991002 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:43.991058 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:43.991071 [ 1094 ] {} <Debug> RaftInstance: reset RPC client for peer 1
2021.03.24 11:22:43.991175 [ 1094 ] {} <Trace> RaftInstance: asio client created: 0x7faa1f16e018
2021.03.24 11:22:43.991187 [ 1094 ] {} <Trace> RaftInstance: 0x7faa1f16e018 reconnect peer 1
2021.03.24 11:22:43.991195 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:43.991204 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:43.991217 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:43.991224 [ 1094 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:43.991234 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:43.991244 [ 1094 ] {} <Debug> RaftInstance: socket 0x7faa1f16e018 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:43.991260 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:43.999656 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:43.999704 [ 1092 ] {} <Warning> RaftInstance: peer (1) response error: failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:43.999726 [ 1092 ] {} <Trace> RaftInstance: asio client destroyed: 0x7faa1f16e018
2021.03.24 11:22:44.041586 [ 1092 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:44.041666 [ 1092 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:44.041677 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:44.041700 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:44.041714 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:44.041722 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:44.041732 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:44.041839 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:44.062514 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:44.062573 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:44.062584 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:44.062595 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:44.062603 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:44.062611 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:44.062624 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:44.062633 [ 1092 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:44.541964 [ 1092 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:44.542025 [ 1092 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:44.542035 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:44.542045 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:44.542059 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:44.542067 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:44.542077 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:44.542173 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:44.562868 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:44.562929 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:44.562940 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:44.562951 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:44.562959 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:44.562967 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:44.562979 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:44.562988 [ 1092 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:44.741366 [ 1092 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:44.741421 [ 1092 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:44.741432 [ 1092 ] {} <Debug> RaftInstance: reset RPC client for peer 1
2021.03.24 11:22:44.741468 [ 1092 ] {} <Trace> RaftInstance: asio client created: 0x7faa1f16e018
2021.03.24 11:22:44.741478 [ 1092 ] {} <Trace> RaftInstance: 0x7faa1f16e018 reconnect peer 1
2021.03.24 11:22:44.741486 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:44.741495 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:44.741506 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:44.741538 [ 1092 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:44.741548 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:44.741557 [ 1092 ] {} <Debug> RaftInstance: socket 0x7faa1f16e018 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:44.741573 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:44.749708 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:44.749757 [ 1095 ] {} <Warning> RaftInstance: peer (1) response error: failed to connect to peer 1, ch-iva.iva.yp-c.yandex.net:9444, error 111
2021.03.24 11:22:44.749779 [ 1095 ] {} <Trace> RaftInstance: asio client destroyed: 0x7faa1f16e018
2021.03.24 11:22:45.042280 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:45.042326 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:45.042336 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:45.042345 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:45.042358 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:45.042366 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:45.042375 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:45.042490 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:45.062960 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:45.063009 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:45.063020 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:45.063031 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:45.063039 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:45.063047 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:45.063059 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2021.03.24 11:22:45.063074 [ 1092 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:45.491695 [ 1092 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:45.491747 [ 1092 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:45.491759 [ 1092 ] {} <Debug> RaftInstance: reset RPC client for peer 1
2021.03.24 11:22:45.491869 [ 1092 ] {} <Trace> RaftInstance: asio client created: 0x7faa1f072418
2021.03.24 11:22:45.491883 [ 1092 ] {} <Trace> RaftInstance: 0x7faa1f072418 reconnect peer 1
2021.03.24 11:22:45.491892 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:45.491902 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:45.491915 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:45.491923 [ 1092 ] {} <Debug> RaftInstance: idx: 1
2021.03.24 11:22:45.491934 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:45.491944 [ 1092 ] {} <Debug> RaftInstance: socket 0x7faa1f072418 to ch-iva.iva.yp-c.yandex.net:9444 is not opened yet
2021.03.24 11:22:45.491961 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:45.499551 [ 1092 ] {} <Information> RaftInstance: 0x7faa1f072418 connected to ch-iva.iva.yp-c.yandex.net:9444 (as a client)
2021.03.24 11:22:45.542599 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:45.542648 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:45.542677 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:45.542687 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:45.542701 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:45.542709 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:45.542719 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:45.542870 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:45.548984 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:45.549022 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:45.549032 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:45.549042 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 2
2021.03.24 11:22:45.549050 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:45.549058 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 0, new matched idx: 1
2021.03.24 11:22:45.549070 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 
2021.03.24 11:22:45.549131 [ 1093 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:45.580702 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:45.580755 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:45.580766 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:45.580777 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:45.580785 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:45.580794 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:45.580806 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 
2021.03.24 11:22:45.580815 [ 1093 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:46.043010 [ 1095 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:46.043059 [ 1095 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:46.043070 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.043080 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:46.043093 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:46.043101 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.043111 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.043196 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.063991 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.064036 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:46.064046 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.064057 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2021.03.24 11:22:46.064065 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.064073 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:46.064085 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 
2021.03.24 11:22:46.064095 [ 1094 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:46.242105 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:46.242151 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:46.242161 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.242171 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2021.03.24 11:22:46.242185 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 1
2021.03.24 11:22:46.242193 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.242203 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.242285 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.249214 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.249248 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=2
2021.03.24 11:22:46.249258 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.249269 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 2
2021.03.24 11:22:46.249277 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.249285 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 1, new matched idx: 1
2021.03.24 11:22:46.249297 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 
2021.03.24 11:22:46.249306 [ 1094 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2021.03.24 11:22:46.373471 [ 1090 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.373498 [ 1090 ] {} <Debug> RaftInstance: append at log_idx 2
2021.03.24 11:22:46.373529 [ 1090 ] {} <Debug> RaftInstance: commit_ret_cv 2 0x7faa1f1c3020 sleep
2021.03.24 11:22:46.373606 [ 1097 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.373665 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 3
2021.03.24 11:22:46.373690 [ 1097 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:46.373707 [ 1097 ] {} <Debug> RaftInstance: idx: 2
2021.03.24 11:22:46.373720 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.373798 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.373812 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.373821 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 3
2021.03.24 11:22:46.373831 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=1, LastLogTerm=1, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:46.373838 [ 1097 ] {} <Debug> RaftInstance: idx: 2
2021.03.24 11:22:46.373923 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.373970 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.381603 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.381658 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=3
2021.03.24 11:22:46.381669 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.381681 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 3
2021.03.24 11:22:46.381690 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.381726 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 1, new matched idx: 2
2021.03.24 11:22:46.381740 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 2 2 1 
2021.03.24 11:22:46.381748 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 2
2021.03.24 11:22:46.381757 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.381765 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.381773 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 2, starting_idx: 1, cur_nxt_idx: 3
2021.03.24 11:22:46.381784 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=2, LastLogTerm=1, EntriesLength=0, CommitIndex=2, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:46.381791 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.381799 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.381852 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.381865 [ 1095 ] {} <Trace> RaftInstance: local log idx 2, target_commit_idx 2, quick_commit_index_ 2, state_->get_commit_idx() 1
2021.03.24 11:22:46.381873 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.381908 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.381920 [ 1096 ] {} <Debug> RaftInstance: commit upto 2, current idx 1
2021.03.24 11:22:46.381929 [ 1096 ] {} <Trace> RaftInstance: commit upto 2, current idx 2
2021.03.24 11:22:46.381993 [ 1096 ] {} <Debug> RaftInstance: notify cb 2 0x7faa1f1c3020
2021.03.24 11:22:46.382012 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 2, current idx 2
2021.03.24 11:22:46.382020 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.382081 [ 1090 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.382122 [ 1090 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 2 wake up (8540 us), return value 0x7faaa9a69a30
2021.03.24 11:22:46.383114 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.383156 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 3
2021.03.24 11:22:46.383209 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 3 0x7faa0daae020 sleep
2021.03.24 11:22:46.383253 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.383277 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.388304 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.388351 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=3
2021.03.24 11:22:46.388361 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.388374 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 3
2021.03.24 11:22:46.388382 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.388390 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 2, new matched idx: 2
2021.03.24 11:22:46.388403 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 3 2 1 
2021.03.24 11:22:46.388412 [ 1094 ] {} <Trace> RaftInstance: local log idx 3, target_commit_idx 2, quick_commit_index_ 2, state_->get_commit_idx() 2
2021.03.24 11:22:46.388420 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.388429 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.388438 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 2, starting_idx: 1, cur_nxt_idx: 4
2021.03.24 11:22:46.388451 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=2, LastLogTerm=1, EntriesLength=1, CommitIndex=2, Term=1, peer_last_sent_idx 3
2021.03.24 11:22:46.388476 [ 1094 ] {} <Debug> RaftInstance: idx: 3
2021.03.24 11:22:46.388486 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.388611 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.394359 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.394393 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=3
2021.03.24 11:22:46.394403 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.394413 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 3
2021.03.24 11:22:46.394421 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.394429 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 2
2021.03.24 11:22:46.394439 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 3 2 2 
2021.03.24 11:22:46.394448 [ 1092 ] {} <Trace> RaftInstance: local log idx 3, target_commit_idx 2, quick_commit_index_ 2, state_->get_commit_idx() 2
2021.03.24 11:22:46.394456 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.394464 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.394473 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 2, starting_idx: 1, cur_nxt_idx: 4
2021.03.24 11:22:46.394484 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=2, LastLogTerm=1, EntriesLength=1, CommitIndex=2, Term=1, peer_last_sent_idx 2
2021.03.24 11:22:46.394491 [ 1092 ] {} <Debug> RaftInstance: idx: 3
2021.03.24 11:22:46.394500 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.394555 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.394832 [ 1092 ] {} <Information> RaftInstance: receive a incoming rpc connection
2021.03.24 11:22:46.394865 [ 1092 ] {} <Information> RaftInstance: session 1 got connection from 2a02:6b8:c0c:511e:0:696:b7ab:0:36638 (as a server)
2021.03.24 11:22:46.394954 [ 1092 ] {} <Trace> RaftInstance: asio rpc session created: 0x7faa1f072e18
2021.03.24 11:22:46.395065 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.395082 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=4
2021.03.24 11:22:46.395090 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.395099 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 4
2021.03.24 11:22:46.395107 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.395115 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 2, new matched idx: 3
2021.03.24 11:22:46.395130 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 3 3 2 
2021.03.24 11:22:46.395138 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 3
2021.03.24 11:22:46.395156 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.395165 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.395173 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 3, starting_idx: 1, cur_nxt_idx: 4
2021.03.24 11:22:46.395182 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=3, LastLogTerm=1, EntriesLength=0, CommitIndex=3, Term=1, peer_last_sent_idx 3
2021.03.24 11:22:46.395190 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.395198 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.395238 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.395251 [ 1092 ] {} <Trace> RaftInstance: local log idx 3, target_commit_idx 3, quick_commit_index_ 3, state_->get_commit_idx() 2
2021.03.24 11:22:46.395259 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.395352 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.395400 [ 1096 ] {} <Debug> RaftInstance: commit upto 3, current idx 2
2021.03.24 11:22:46.395413 [ 1096 ] {} <Trace> RaftInstance: commit upto 3, current idx 3
2021.03.24 11:22:46.395515 [ 1096 ] {} <Debug> RaftInstance: notify cb 3 0x7faa0daae020
2021.03.24 11:22:46.395542 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 3, current idx 3
2021.03.24 11:22:46.395551 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.395579 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.395614 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 3 wake up (12397 us), return value (nil)
2021.03.24 11:22:46.395661 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.395708 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 4
2021.03.24 11:22:46.395734 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 4 0x7faaa6f93120 sleep
2021.03.24 11:22:46.395765 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.395785 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.401847 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.401880 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=4
2021.03.24 11:22:46.401891 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.401901 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 4
2021.03.24 11:22:46.401909 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.401917 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 3, new matched idx: 3
2021.03.24 11:22:46.401928 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 4 3 2 
2021.03.24 11:22:46.401937 [ 1093 ] {} <Trace> RaftInstance: local log idx 4, target_commit_idx 3, quick_commit_index_ 3, state_->get_commit_idx() 3
2021.03.24 11:22:46.401945 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.401953 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.401961 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 3, starting_idx: 1, cur_nxt_idx: 5
2021.03.24 11:22:46.401973 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=3, LastLogTerm=1, EntriesLength=1, CommitIndex=3, Term=1, peer_last_sent_idx 4
2021.03.24 11:22:46.401980 [ 1093 ] {} <Debug> RaftInstance: idx: 4
2021.03.24 11:22:46.401989 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.402044 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.408748 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.408787 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=5
2021.03.24 11:22:46.408796 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.408807 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 5
2021.03.24 11:22:46.408815 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.408831 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 3, new matched idx: 4
2021.03.24 11:22:46.408842 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 4 4 2 
2021.03.24 11:22:46.408850 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 4
2021.03.24 11:22:46.408860 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.408868 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.408876 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 4, starting_idx: 1, cur_nxt_idx: 5
2021.03.24 11:22:46.408886 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=4, LastLogTerm=1, EntriesLength=0, CommitIndex=4, Term=1, peer_last_sent_idx 4
2021.03.24 11:22:46.408912 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.408921 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.408979 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.408992 [ 1093 ] {} <Trace> RaftInstance: local log idx 4, target_commit_idx 4, quick_commit_index_ 4, state_->get_commit_idx() 3
2021.03.24 11:22:46.409000 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.409063 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.409149 [ 1096 ] {} <Debug> RaftInstance: commit upto 4, current idx 3
2021.03.24 11:22:46.409162 [ 1096 ] {} <Trace> RaftInstance: commit upto 4, current idx 4
2021.03.24 11:22:46.409190 [ 1096 ] {} <Debug> RaftInstance: notify cb 4 0x7faaa6f93120
2021.03.24 11:22:46.409206 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 4, current idx 4
2021.03.24 11:22:46.409213 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.409235 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.409247 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 4 wake up (13512 us), return value 0x7faaa9a69a90
2021.03.24 11:22:46.415161 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.415219 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=4
2021.03.24 11:22:46.415241 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.415253 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 4
2021.03.24 11:22:46.415262 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.415270 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 2, new matched idx: 3
2021.03.24 11:22:46.415281 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 4 4 3 
2021.03.24 11:22:46.415291 [ 1094 ] {} <Trace> RaftInstance: local log idx 4, target_commit_idx 4, quick_commit_index_ 4, state_->get_commit_idx() 4
2021.03.24 11:22:46.415299 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.415308 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.415322 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 3, starting_idx: 1, cur_nxt_idx: 5
2021.03.24 11:22:46.415335 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=3, LastLogTerm=1, EntriesLength=1, CommitIndex=4, Term=1, peer_last_sent_idx 3
2021.03.24 11:22:46.415342 [ 1094 ] {} <Debug> RaftInstance: idx: 4
2021.03.24 11:22:46.415351 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.415431 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.415656 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.415697 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=5
2021.03.24 11:22:46.415707 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.415717 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 5
2021.03.24 11:22:46.415726 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.415734 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 4, new matched idx: 4
2021.03.24 11:22:46.415745 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 4 4 3 
2021.03.24 11:22:46.415754 [ 1092 ] {} <Trace> RaftInstance: local log idx 4, target_commit_idx 4, quick_commit_index_ 4, state_->get_commit_idx() 4
2021.03.24 11:22:46.426144 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.426242 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 5
2021.03.24 11:22:46.426268 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 5 0x7faaa6f93120 sleep
2021.03.24 11:22:46.426340 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.426363 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.426373 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 4, starting_idx: 1, cur_nxt_idx: 6
2021.03.24 11:22:46.426386 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=4, LastLogTerm=1, EntriesLength=1, CommitIndex=4, Term=1, peer_last_sent_idx 5
2021.03.24 11:22:46.426395 [ 1097 ] {} <Debug> RaftInstance: idx: 5
2021.03.24 11:22:46.426406 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.426524 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.433399 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.433451 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=6
2021.03.24 11:22:46.433461 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.433473 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 6
2021.03.24 11:22:46.433481 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.433489 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 4, new matched idx: 5
2021.03.24 11:22:46.433524 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 5 5 3 
2021.03.24 11:22:46.433533 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 5
2021.03.24 11:22:46.433543 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.433551 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.433559 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 5, starting_idx: 1, cur_nxt_idx: 6
2021.03.24 11:22:46.433571 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=5, LastLogTerm=1, EntriesLength=0, CommitIndex=5, Term=1, peer_last_sent_idx 5
2021.03.24 11:22:46.433578 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.433587 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.433682 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.433696 [ 1093 ] {} <Trace> RaftInstance: local log idx 5, target_commit_idx 5, quick_commit_index_ 5, state_->get_commit_idx() 4
2021.03.24 11:22:46.433704 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.433804 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.433854 [ 1096 ] {} <Debug> RaftInstance: commit upto 5, current idx 4
2021.03.24 11:22:46.433866 [ 1096 ] {} <Trace> RaftInstance: commit upto 5, current idx 5
2021.03.24 11:22:46.433902 [ 1096 ] {} <Debug> RaftInstance: notify cb 5 0x7faaa6f93120
2021.03.24 11:22:46.433917 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 5, current idx 5
2021.03.24 11:22:46.433924 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.433976 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.433989 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 5 wake up (7717 us), return value (nil)
2021.03.24 11:22:46.435953 [ 1093 ] {} <Information> RaftInstance: receive a incoming rpc connection
2021.03.24 11:22:46.436030 [ 1093 ] {} <Information> RaftInstance: session 2 got connection from 2a02:6b8:c0b:6705:0:696:e66a:0:48298 (as a server)
2021.03.24 11:22:46.436083 [ 1093 ] {} <Trace> RaftInstance: asio rpc session created: 0x7faaa9b7a618
2021.03.24 11:22:46.436163 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.436168 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.436195 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 6
2021.03.24 11:22:46.436232 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=5
2021.03.24 11:22:46.436258 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.436262 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 6 0x7faa1f09b120 sleep
2021.03.24 11:22:46.436272 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 5
2021.03.24 11:22:46.436282 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.436290 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 3, new matched idx: 4
2021.03.24 11:22:46.436301 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 6 5 4 
2021.03.24 11:22:46.436310 [ 1095 ] {} <Trace> RaftInstance: local log idx 6, target_commit_idx 5, quick_commit_index_ 5, state_->get_commit_idx() 5
2021.03.24 11:22:46.436318 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.436326 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.436334 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 4, starting_idx: 1, cur_nxt_idx: 7
2021.03.24 11:22:46.436346 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=4, LastLogTerm=1, EntriesLength=2, CommitIndex=5, Term=1, peer_last_sent_idx 4
2021.03.24 11:22:46.436353 [ 1095 ] {} <Debug> RaftInstance: idx range: 5-6
2021.03.24 11:22:46.436362 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.436412 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.436496 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.436518 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.440655 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.440694 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=6
2021.03.24 11:22:46.440703 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.440714 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 6
2021.03.24 11:22:46.440722 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.440730 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 5, new matched idx: 5
2021.03.24 11:22:46.440742 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 6 5 4 
2021.03.24 11:22:46.440750 [ 1092 ] {} <Trace> RaftInstance: local log idx 6, target_commit_idx 5, quick_commit_index_ 5, state_->get_commit_idx() 5
2021.03.24 11:22:46.440758 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.440767 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.440775 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 5, starting_idx: 1, cur_nxt_idx: 7
2021.03.24 11:22:46.440786 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=5, LastLogTerm=1, EntriesLength=1, CommitIndex=5, Term=1, peer_last_sent_idx 6
2021.03.24 11:22:46.440793 [ 1092 ] {} <Debug> RaftInstance: idx: 6
2021.03.24 11:22:46.440802 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.440906 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.447801 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.447861 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=7
2021.03.24 11:22:46.447872 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.447884 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 7
2021.03.24 11:22:46.447926 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.447935 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 5, new matched idx: 6
2021.03.24 11:22:46.447947 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 6 6 4 
2021.03.24 11:22:46.447955 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 6
2021.03.24 11:22:46.447964 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.447972 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.447981 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 6, starting_idx: 1, cur_nxt_idx: 7
2021.03.24 11:22:46.447992 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=6, LastLogTerm=1, EntriesLength=0, CommitIndex=6, Term=1, peer_last_sent_idx 6
2021.03.24 11:22:46.447999 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.448007 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.448091 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.448105 [ 1095 ] {} <Trace> RaftInstance: local log idx 6, target_commit_idx 6, quick_commit_index_ 6, state_->get_commit_idx() 5
2021.03.24 11:22:46.448112 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.448140 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.448153 [ 1096 ] {} <Debug> RaftInstance: commit upto 6, current idx 5
2021.03.24 11:22:46.448188 [ 1096 ] {} <Trace> RaftInstance: commit upto 6, current idx 6
2021.03.24 11:22:46.448223 [ 1096 ] {} <Debug> RaftInstance: notify cb 6 0x7faa1f09b120
2021.03.24 11:22:46.448241 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 6, current idx 6
2021.03.24 11:22:46.448249 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.448285 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.448320 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 6 wake up (12007 us), return value 0x7faaa9a69aa0
2021.03.24 11:22:46.454849 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.454877 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=7
2021.03.24 11:22:46.454887 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.454899 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 7
2021.03.24 11:22:46.454907 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.454920 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 6, new matched idx: 6
2021.03.24 11:22:46.454933 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 6 6 4 
2021.03.24 11:22:46.454941 [ 1094 ] {} <Trace> RaftInstance: local log idx 6, target_commit_idx 6, quick_commit_index_ 6, state_->get_commit_idx() 6
2021.03.24 11:22:46.457451 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.457501 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=7
2021.03.24 11:22:46.457511 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.457522 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 7
2021.03.24 11:22:46.457530 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.457538 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 4, new matched idx: 6
2021.03.24 11:22:46.457549 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 6 6 6 
2021.03.24 11:22:46.457559 [ 1095 ] {} <Trace> RaftInstance: local log idx 6, target_commit_idx 6, quick_commit_index_ 6, state_->get_commit_idx() 6
2021.03.24 11:22:46.457567 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.457595 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.457603 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 6, starting_idx: 1, cur_nxt_idx: 7
2021.03.24 11:22:46.457614 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=6, LastLogTerm=1, EntriesLength=0, CommitIndex=6, Term=1, peer_last_sent_idx 5
2021.03.24 11:22:46.457621 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.457629 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.457695 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.478378 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.478420 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=7
2021.03.24 11:22:46.478430 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.478440 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 7
2021.03.24 11:22:46.478449 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.478458 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 6, new matched idx: 6
2021.03.24 11:22:46.478470 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 6 6 6 
2021.03.24 11:22:46.478478 [ 1093 ] {} <Trace> RaftInstance: local log idx 6, target_commit_idx 6, quick_commit_index_ 6, state_->get_commit_idx() 6
2021.03.24 11:22:46.492886 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.492940 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 7
2021.03.24 11:22:46.492964 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 7 0x7faaa6f93420 sleep
2021.03.24 11:22:46.493130 [ 1097 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.493182 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 6, starting_idx: 1, cur_nxt_idx: 8
2021.03.24 11:22:46.493198 [ 1097 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=6, LastLogTerm=1, EntriesLength=1, CommitIndex=6, Term=1, peer_last_sent_idx 7
2021.03.24 11:22:46.493207 [ 1097 ] {} <Debug> RaftInstance: idx: 7
2021.03.24 11:22:46.493218 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.493324 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.493338 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.493346 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 6, starting_idx: 1, cur_nxt_idx: 8
2021.03.24 11:22:46.493356 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=6, LastLogTerm=1, EntriesLength=1, CommitIndex=6, Term=1, peer_last_sent_idx 7
2021.03.24 11:22:46.493364 [ 1097 ] {} <Debug> RaftInstance: idx: 7
2021.03.24 11:22:46.493372 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.493409 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.500503 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.500580 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=8
2021.03.24 11:22:46.500591 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.500603 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 8
2021.03.24 11:22:46.500611 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.500619 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 6, new matched idx: 7
2021.03.24 11:22:46.500631 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 7 7 6 
2021.03.24 11:22:46.500652 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 7
2021.03.24 11:22:46.500661 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.500669 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.500703 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 7, starting_idx: 1, cur_nxt_idx: 8
2021.03.24 11:22:46.500714 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=7, LastLogTerm=1, EntriesLength=0, CommitIndex=7, Term=1, peer_last_sent_idx 7
2021.03.24 11:22:46.500721 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.500729 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.500803 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.500827 [ 1095 ] {} <Trace> RaftInstance: local log idx 7, target_commit_idx 7, quick_commit_index_ 7, state_->get_commit_idx() 6
2021.03.24 11:22:46.500836 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.500882 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.500895 [ 1096 ] {} <Debug> RaftInstance: commit upto 7, current idx 6
2021.03.24 11:22:46.500904 [ 1096 ] {} <Trace> RaftInstance: commit upto 7, current idx 7
2021.03.24 11:22:46.500941 [ 1096 ] {} <Debug> RaftInstance: notify cb 7 0x7faaa6f93420
2021.03.24 11:22:46.500963 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 7, current idx 7
2021.03.24 11:22:46.500971 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.501001 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.501020 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 7 wake up (8033 us), return value (nil)
2021.03.24 11:22:46.507785 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.507849 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=8
2021.03.24 11:22:46.507872 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.507884 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 8
2021.03.24 11:22:46.507892 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.507899 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 7, new matched idx: 7
2021.03.24 11:22:46.507911 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 7 7 6 
2021.03.24 11:22:46.507920 [ 1092 ] {} <Trace> RaftInstance: local log idx 7, target_commit_idx 7, quick_commit_index_ 7, state_->get_commit_idx() 7
2021.03.24 11:22:46.514084 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.514151 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=8
2021.03.24 11:22:46.514162 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.514173 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 8
2021.03.24 11:22:46.514181 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.514189 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 6, new matched idx: 7
2021.03.24 11:22:46.514201 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 7 7 7 
2021.03.24 11:22:46.514210 [ 1092 ] {} <Trace> RaftInstance: local log idx 7, target_commit_idx 7, quick_commit_index_ 7, state_->get_commit_idx() 7
2021.03.24 11:22:46.514218 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.514227 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.514235 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 7, starting_idx: 1, cur_nxt_idx: 8
2021.03.24 11:22:46.514246 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=7, LastLogTerm=1, EntriesLength=0, CommitIndex=7, Term=1, peer_last_sent_idx 7
2021.03.24 11:22:46.514253 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.514262 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.514336 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.535019 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.535070 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=8
2021.03.24 11:22:46.535080 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.535092 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 8
2021.03.24 11:22:46.535100 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.535109 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 7, new matched idx: 7
2021.03.24 11:22:46.535121 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 7 7 7 
2021.03.24 11:22:46.535130 [ 1093 ] {} <Trace> RaftInstance: local log idx 7, target_commit_idx 7, quick_commit_index_ 7, state_->get_commit_idx() 7
2021.03.24 11:22:46.543371 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:46.543429 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:46.543438 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.543448 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 7, starting_idx: 1, cur_nxt_idx: 8
2021.03.24 11:22:46.543461 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=7, LastLogTerm=1, EntriesLength=0, CommitIndex=7, Term=1, peer_last_sent_idx 8
2021.03.24 11:22:46.543469 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.543480 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.543576 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.564231 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.564294 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=8
2021.03.24 11:22:46.564304 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.564315 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 8
2021.03.24 11:22:46.564323 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.564344 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 7, new matched idx: 7
2021.03.24 11:22:46.564358 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 7 7 7 
2021.03.24 11:22:46.564367 [ 1094 ] {} <Trace> RaftInstance: local log idx 7, target_commit_idx 7, quick_commit_index_ 7, state_->get_commit_idx() 7
2021.03.24 11:22:46.583713 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.583789 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 8
2021.03.24 11:22:46.583816 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 8 0x7faaa6f93520 sleep
2021.03.24 11:22:46.583852 [ 1097 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.583876 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 7, starting_idx: 1, cur_nxt_idx: 9
2021.03.24 11:22:46.583891 [ 1097 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=7, LastLogTerm=1, EntriesLength=1, CommitIndex=7, Term=1, peer_last_sent_idx 8
2021.03.24 11:22:46.583900 [ 1097 ] {} <Debug> RaftInstance: idx: 8
2021.03.24 11:22:46.583911 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.583997 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.584010 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.584019 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 7, starting_idx: 1, cur_nxt_idx: 9
2021.03.24 11:22:46.584029 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=7, LastLogTerm=1, EntriesLength=1, CommitIndex=7, Term=1, peer_last_sent_idx 8
2021.03.24 11:22:46.584062 [ 1097 ] {} <Debug> RaftInstance: idx: 8
2021.03.24 11:22:46.584071 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.584110 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.589224 [ 1091 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.589254 [ 1091 ] {} <Debug> RaftInstance: append at log_idx 9
2021.03.24 11:22:46.589285 [ 1091 ] {} <Debug> RaftInstance: commit_ret_cv 9 0x7faa1ad2d020 sleep
2021.03.24 11:22:46.589315 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.589363 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.591176 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.591201 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=9
2021.03.24 11:22:46.591212 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.591222 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 9
2021.03.24 11:22:46.591231 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.591239 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 7, new matched idx: 8
2021.03.24 11:22:46.591250 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 9 8 7 
2021.03.24 11:22:46.591257 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 8
2021.03.24 11:22:46.591265 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.591273 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.591282 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 8, starting_idx: 1, cur_nxt_idx: 10
2021.03.24 11:22:46.591294 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=8, LastLogTerm=1, EntriesLength=1, CommitIndex=8, Term=1, peer_last_sent_idx 8
2021.03.24 11:22:46.591301 [ 1094 ] {} <Debug> RaftInstance: idx: 9
2021.03.24 11:22:46.591310 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.591357 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.591370 [ 1094 ] {} <Trace> RaftInstance: local log idx 9, target_commit_idx 8, quick_commit_index_ 8, state_->get_commit_idx() 7
2021.03.24 11:22:46.591378 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.591415 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.591427 [ 1096 ] {} <Debug> RaftInstance: commit upto 8, current idx 7
2021.03.24 11:22:46.591437 [ 1096 ] {} <Trace> RaftInstance: commit upto 8, current idx 8
2021.03.24 11:22:46.591462 [ 1096 ] {} <Debug> RaftInstance: notify cb 8 0x7faaa6f93520
2021.03.24 11:22:46.591475 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 8, current idx 8
2021.03.24 11:22:46.591483 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.591501 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.591512 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 8 wake up (7697 us), return value 0x7faaa9a69a50
2021.03.24 11:22:46.591563 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.591575 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.598038 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.598087 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=10
2021.03.24 11:22:46.598099 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.598110 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 10
2021.03.24 11:22:46.598118 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.598126 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 8, new matched idx: 9
2021.03.24 11:22:46.598138 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 9 9 7 
2021.03.24 11:22:46.598146 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 9
2021.03.24 11:22:46.598154 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.598162 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.598170 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 9, starting_idx: 1, cur_nxt_idx: 10
2021.03.24 11:22:46.598181 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=9, LastLogTerm=1, EntriesLength=0, CommitIndex=9, Term=1, peer_last_sent_idx 9
2021.03.24 11:22:46.598188 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.598197 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.598263 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.598277 [ 1093 ] {} <Trace> RaftInstance: local log idx 9, target_commit_idx 9, quick_commit_index_ 9, state_->get_commit_idx() 8
2021.03.24 11:22:46.598285 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.598328 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.598378 [ 1096 ] {} <Debug> RaftInstance: commit upto 9, current idx 8
2021.03.24 11:22:46.598389 [ 1096 ] {} <Trace> RaftInstance: commit upto 9, current idx 9
2021.03.24 11:22:46.598418 [ 1096 ] {} <Debug> RaftInstance: notify cb 9 0x7faa1ad2d020
2021.03.24 11:22:46.598440 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 9, current idx 9
2021.03.24 11:22:46.598448 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.598468 [ 1091 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.598485 [ 1091 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 9 wake up (9189 us), return value 0x7faaa9a69a40
2021.03.24 11:22:46.598568 [ 1161 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.598607 [ 1161 ] {} <Debug> RaftInstance: append at log_idx 10
2021.03.24 11:22:46.598643 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.598656 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.598668 [ 1161 ] {} <Debug> RaftInstance: commit_ret_cv 10 0x7faaa6f93f20 sleep
2021.03.24 11:22:46.604483 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.604535 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=9
2021.03.24 11:22:46.604546 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.604557 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 9
2021.03.24 11:22:46.604566 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.604574 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 7, new matched idx: 8
2021.03.24 11:22:46.604586 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 10 9 8 
2021.03.24 11:22:46.604595 [ 1092 ] {} <Trace> RaftInstance: local log idx 10, target_commit_idx 9, quick_commit_index_ 9, state_->get_commit_idx() 9
2021.03.24 11:22:46.604625 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.604634 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.604642 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 8, starting_idx: 1, cur_nxt_idx: 11
2021.03.24 11:22:46.604654 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=8, LastLogTerm=1, EntriesLength=2, CommitIndex=9, Term=1, peer_last_sent_idx 8
2021.03.24 11:22:46.604662 [ 1092 ] {} <Debug> RaftInstance: idx range: 9-10
2021.03.24 11:22:46.604671 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.604780 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.604876 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.604893 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=10
2021.03.24 11:22:46.604903 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.604911 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 10
2021.03.24 11:22:46.604919 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.604926 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 9, new matched idx: 9
2021.03.24 11:22:46.604935 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 10 9 8 
2021.03.24 11:22:46.604943 [ 1094 ] {} <Trace> RaftInstance: local log idx 10, target_commit_idx 9, quick_commit_index_ 9, state_->get_commit_idx() 9
2021.03.24 11:22:46.604951 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.604958 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.604966 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 9, starting_idx: 1, cur_nxt_idx: 11
2021.03.24 11:22:46.604976 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=9, LastLogTerm=1, EntriesLength=1, CommitIndex=9, Term=1, peer_last_sent_idx 10
2021.03.24 11:22:46.604983 [ 1094 ] {} <Debug> RaftInstance: idx: 10
2021.03.24 11:22:46.604991 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.605028 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.611264 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.611318 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 11
2021.03.24 11:22:46.611342 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 11 0x7faaa6f93720 sleep
2021.03.24 11:22:46.611403 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.611448 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.611510 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.611533 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=11
2021.03.24 11:22:46.611544 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.611554 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 11
2021.03.24 11:22:46.611563 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.611571 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 9, new matched idx: 10
2021.03.24 11:22:46.611581 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 11 10 8 
2021.03.24 11:22:46.611589 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 10
2021.03.24 11:22:46.611597 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.611605 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.611613 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 10, starting_idx: 1, cur_nxt_idx: 12
2021.03.24 11:22:46.611650 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=10, LastLogTerm=1, EntriesLength=1, CommitIndex=10, Term=1, peer_last_sent_idx 10
2021.03.24 11:22:46.611657 [ 1092 ] {} <Debug> RaftInstance: idx: 11
2021.03.24 11:22:46.611666 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.611735 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.611748 [ 1092 ] {} <Trace> RaftInstance: local log idx 11, target_commit_idx 10, quick_commit_index_ 10, state_->get_commit_idx() 9
2021.03.24 11:22:46.611756 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.611791 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.611803 [ 1096 ] {} <Debug> RaftInstance: commit upto 10, current idx 9
2021.03.24 11:22:46.611812 [ 1096 ] {} <Trace> RaftInstance: commit upto 10, current idx 10
2021.03.24 11:22:46.611840 [ 1096 ] {} <Debug> RaftInstance: notify cb 10 0x7faaa6f93f20
2021.03.24 11:22:46.611859 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 10, current idx 10
2021.03.24 11:22:46.611868 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.611879 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.611882 [ 1161 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.611907 [ 1161 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 10 wake up (13253 us), return value 0x7faaa9a69a70
2021.03.24 11:22:46.611888 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.611976 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.612006 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 12
2021.03.24 11:22:46.612038 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 12 0x7faa0daae120 sleep
2021.03.24 11:22:46.612041 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.612071 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.618449 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.618502 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=12
2021.03.24 11:22:46.618514 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.618524 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 12
2021.03.24 11:22:46.618533 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.618541 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 10, new matched idx: 11
2021.03.24 11:22:46.618553 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 12 11 8 
2021.03.24 11:22:46.618562 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 11
2021.03.24 11:22:46.618571 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.618579 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.618588 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 11, starting_idx: 1, cur_nxt_idx: 13
2021.03.24 11:22:46.618600 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=11, LastLogTerm=1, EntriesLength=1, CommitIndex=11, Term=1, peer_last_sent_idx 11
2021.03.24 11:22:46.618607 [ 1093 ] {} <Debug> RaftInstance: idx: 12
2021.03.24 11:22:46.618616 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.618685 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.618698 [ 1093 ] {} <Trace> RaftInstance: local log idx 12, target_commit_idx 11, quick_commit_index_ 11, state_->get_commit_idx() 10
2021.03.24 11:22:46.618728 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.618753 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.618765 [ 1096 ] {} <Debug> RaftInstance: commit upto 11, current idx 10
2021.03.24 11:22:46.618797 [ 1096 ] {} <Trace> RaftInstance: commit upto 11, current idx 11
2021.03.24 11:22:46.618827 [ 1096 ] {} <Debug> RaftInstance: notify cb 11 0x7faaa6f93720
2021.03.24 11:22:46.618846 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 11, current idx 11
2021.03.24 11:22:46.618854 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.618868 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.618877 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.618882 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.618905 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 11 wake up (7543 us), return value 0x7faaa9a69ab0
2021.03.24 11:22:46.625226 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.625275 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=11
2021.03.24 11:22:46.625286 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.625297 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 11
2021.03.24 11:22:46.625305 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.625312 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 8, new matched idx: 10
2021.03.24 11:22:46.625324 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 12 11 10 
2021.03.24 11:22:46.625333 [ 1093 ] {} <Trace> RaftInstance: local log idx 12, target_commit_idx 11, quick_commit_index_ 11, state_->get_commit_idx() 11
2021.03.24 11:22:46.625341 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.625350 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.625358 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 10, starting_idx: 1, cur_nxt_idx: 13
2021.03.24 11:22:46.625370 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=10, LastLogTerm=1, EntriesLength=2, CommitIndex=11, Term=1, peer_last_sent_idx 9
2021.03.24 11:22:46.625378 [ 1093 ] {} <Debug> RaftInstance: idx range: 11-12
2021.03.24 11:22:46.625387 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.625470 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.625553 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.625569 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=13
2021.03.24 11:22:46.625578 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.625587 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 13
2021.03.24 11:22:46.625594 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.625602 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 11, new matched idx: 12
2021.03.24 11:22:46.625610 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 12 12 10 
2021.03.24 11:22:46.625618 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 12
2021.03.24 11:22:46.625626 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.625633 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.625641 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 12, starting_idx: 1, cur_nxt_idx: 13
2021.03.24 11:22:46.625650 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=12, LastLogTerm=1, EntriesLength=0, CommitIndex=12, Term=1, peer_last_sent_idx 12
2021.03.24 11:22:46.625679 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.625687 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.625723 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.625736 [ 1094 ] {} <Trace> RaftInstance: local log idx 12, target_commit_idx 12, quick_commit_index_ 12, state_->get_commit_idx() 11
2021.03.24 11:22:46.625743 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.625772 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.625784 [ 1096 ] {} <Debug> RaftInstance: commit upto 12, current idx 11
2021.03.24 11:22:46.625794 [ 1096 ] {} <Trace> RaftInstance: commit upto 12, current idx 12
2021.03.24 11:22:46.625823 [ 1096 ] {} <Debug> RaftInstance: notify cb 12 0x7faa0daae120
2021.03.24 11:22:46.625842 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 12, current idx 12
2021.03.24 11:22:46.625850 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.625873 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.625900 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 12 wake up (13845 us), return value (nil)
2021.03.24 11:22:46.625928 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.625948 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 13
2021.03.24 11:22:46.625972 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 13 0x7faa0daae020 sleep
2021.03.24 11:22:46.625999 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.626017 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.632201 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.632241 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=13
2021.03.24 11:22:46.632251 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.632262 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 13
2021.03.24 11:22:46.632270 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.632278 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 12, new matched idx: 12
2021.03.24 11:22:46.632290 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 13 12 10 
2021.03.24 11:22:46.632298 [ 1092 ] {} <Trace> RaftInstance: local log idx 13, target_commit_idx 12, quick_commit_index_ 12, state_->get_commit_idx() 12
2021.03.24 11:22:46.632307 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.632315 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.632324 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 12, starting_idx: 1, cur_nxt_idx: 14
2021.03.24 11:22:46.632336 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=12, LastLogTerm=1, EntriesLength=1, CommitIndex=12, Term=1, peer_last_sent_idx 13
2021.03.24 11:22:46.632343 [ 1092 ] {} <Debug> RaftInstance: idx: 13
2021.03.24 11:22:46.632352 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.632399 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.638929 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.638979 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 14
2021.03.24 11:22:46.638987 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.639031 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.639004 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 14 0x7faaa6f93720 sleep
2021.03.24 11:22:46.639048 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.639086 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 12, starting_idx: 1, cur_nxt_idx: 15
2021.03.24 11:22:46.639096 [ 1097 ] {} <Debug> RaftInstance: last sent log (13) to peer 1 is not applied, cnt 1
2021.03.24 11:22:46.639109 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=12, LastLogTerm=1, EntriesLength=2, CommitIndex=12, Term=1, peer_last_sent_idx 13
2021.03.24 11:22:46.639118 [ 1097 ] {} <Debug> RaftInstance: idx range: 13-14
2021.03.24 11:22:46.639128 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.639197 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.639234 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=14
2021.03.24 11:22:46.639251 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.639262 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 14
2021.03.24 11:22:46.639271 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.639279 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 12, new matched idx: 13
2021.03.24 11:22:46.639290 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 14 13 10 
2021.03.24 11:22:46.639298 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 13
2021.03.24 11:22:46.639307 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.639314 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.639323 [ 1094 ] {} <Trace> RaftInstance: local log idx 14, target_commit_idx 13, quick_commit_index_ 13, state_->get_commit_idx() 12
2021.03.24 11:22:46.639331 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.639352 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.639363 [ 1096 ] {} <Debug> RaftInstance: commit upto 13, current idx 12
2021.03.24 11:22:46.639373 [ 1096 ] {} <Trace> RaftInstance: commit upto 13, current idx 13
2021.03.24 11:22:46.639401 [ 1096 ] {} <Debug> RaftInstance: notify cb 13 0x7faa0daae020
2021.03.24 11:22:46.639422 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 13, current idx 13
2021.03.24 11:22:46.639431 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.639442 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.639459 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 13 wake up (13476 us), return value (nil)
2021.03.24 11:22:46.639481 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.639494 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.639495 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.639537 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 15
2021.03.24 11:22:46.639558 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 15 0x7faa0daae120 sleep
2021.03.24 11:22:46.639576 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.639591 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.645867 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.645924 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=15
2021.03.24 11:22:46.645957 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.645970 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 15
2021.03.24 11:22:46.645978 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.645986 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 13, new matched idx: 14
2021.03.24 11:22:46.645999 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 15 14 10 
2021.03.24 11:22:46.646007 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 14
2021.03.24 11:22:46.646015 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.646023 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.646032 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 14, starting_idx: 1, cur_nxt_idx: 16
2021.03.24 11:22:46.646044 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=14, LastLogTerm=1, EntriesLength=1, CommitIndex=14, Term=1, peer_last_sent_idx 13
2021.03.24 11:22:46.646051 [ 1093 ] {} <Debug> RaftInstance: idx: 15
2021.03.24 11:22:46.646060 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.646132 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.646145 [ 1093 ] {} <Trace> RaftInstance: local log idx 15, target_commit_idx 14, quick_commit_index_ 14, state_->get_commit_idx() 13
2021.03.24 11:22:46.646153 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.646178 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.646190 [ 1096 ] {} <Debug> RaftInstance: commit upto 14, current idx 13
2021.03.24 11:22:46.646198 [ 1096 ] {} <Trace> RaftInstance: commit upto 14, current idx 14
2021.03.24 11:22:46.646226 [ 1096 ] {} <Debug> RaftInstance: notify cb 14 0x7faaa6f93720
2021.03.24 11:22:46.646248 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 14, current idx 14
2021.03.24 11:22:46.646257 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.646281 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.646318 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 14 wake up (7277 us), return value (nil)
2021.03.24 11:22:46.646323 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.646291 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.646348 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.646375 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=13
2021.03.24 11:22:46.646386 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.646395 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 13
2021.03.24 11:22:46.646402 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.646410 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 10, new matched idx: 12
2021.03.24 11:22:46.646419 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 15 14 12 
2021.03.24 11:22:46.646426 [ 1092 ] {} <Trace> RaftInstance: local log idx 15, target_commit_idx 14, quick_commit_index_ 14, state_->get_commit_idx() 14
2021.03.24 11:22:46.646434 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.646441 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.646449 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 12, starting_idx: 1, cur_nxt_idx: 16
2021.03.24 11:22:46.646459 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=12, LastLogTerm=1, EntriesLength=3, CommitIndex=14, Term=1, peer_last_sent_idx 11
2021.03.24 11:22:46.646466 [ 1092 ] {} <Debug> RaftInstance: idx range: 13-15
2021.03.24 11:22:46.646474 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.646554 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.652779 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.652826 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=16
2021.03.24 11:22:46.652836 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.652847 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 16
2021.03.24 11:22:46.652856 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.652864 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 14, new matched idx: 15
2021.03.24 11:22:46.652876 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 15 15 12 
2021.03.24 11:22:46.652884 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 15
2021.03.24 11:22:46.652893 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.652901 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.652909 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 15, starting_idx: 1, cur_nxt_idx: 16
2021.03.24 11:22:46.652919 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=15, LastLogTerm=1, EntriesLength=0, CommitIndex=15, Term=1, peer_last_sent_idx 15
2021.03.24 11:22:46.652927 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.652935 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.652993 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.653006 [ 1093 ] {} <Trace> RaftInstance: local log idx 15, target_commit_idx 15, quick_commit_index_ 15, state_->get_commit_idx() 14
2021.03.24 11:22:46.653014 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.653036 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.653047 [ 1096 ] {} <Debug> RaftInstance: commit upto 15, current idx 14
2021.03.24 11:22:46.653056 [ 1096 ] {} <Trace> RaftInstance: commit upto 15, current idx 15
2021.03.24 11:22:46.653185 [ 1096 ] {} <Debug> RaftInstance: notify cb 15 0x7faa0daae120
2021.03.24 11:22:46.653204 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 15, current idx 15
2021.03.24 11:22:46.653212 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.653234 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.653259 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 15 wake up (13675 us), return value (nil)
2021.03.24 11:22:46.653290 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.653312 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 16
2021.03.24 11:22:46.653335 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 16 0x7faa0daae020 sleep
2021.03.24 11:22:46.653373 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.653394 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.659645 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.659679 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=16
2021.03.24 11:22:46.659690 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.659701 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 16
2021.03.24 11:22:46.659709 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.659718 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 15, new matched idx: 15
2021.03.24 11:22:46.659730 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 16 15 12 
2021.03.24 11:22:46.659759 [ 1094 ] {} <Trace> RaftInstance: local log idx 16, target_commit_idx 15, quick_commit_index_ 15, state_->get_commit_idx() 15
2021.03.24 11:22:46.659767 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.659776 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.659785 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 15, starting_idx: 1, cur_nxt_idx: 17
2021.03.24 11:22:46.659797 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=15, LastLogTerm=1, EntriesLength=1, CommitIndex=15, Term=1, peer_last_sent_idx 16
2021.03.24 11:22:46.659804 [ 1094 ] {} <Debug> RaftInstance: idx: 16
2021.03.24 11:22:46.659814 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.659868 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.666118 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.666160 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 17
2021.03.24 11:22:46.666184 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 17 0x7faaa6f93720 sleep
2021.03.24 11:22:46.666224 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.666266 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.666381 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.666401 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=17
2021.03.24 11:22:46.666412 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.666422 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 17
2021.03.24 11:22:46.666430 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.666439 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 15, new matched idx: 16
2021.03.24 11:22:46.666449 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 17 16 12 
2021.03.24 11:22:46.666457 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 16
2021.03.24 11:22:46.666465 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.666473 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.666482 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 16, starting_idx: 1, cur_nxt_idx: 18
2021.03.24 11:22:46.666494 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=16, LastLogTerm=1, EntriesLength=1, CommitIndex=16, Term=1, peer_last_sent_idx 16
2021.03.24 11:22:46.666502 [ 1093 ] {} <Debug> RaftInstance: idx: 17
2021.03.24 11:22:46.666511 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.666562 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.666576 [ 1093 ] {} <Trace> RaftInstance: local log idx 17, target_commit_idx 16, quick_commit_index_ 16, state_->get_commit_idx() 15
2021.03.24 11:22:46.666585 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.666598 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.666606 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.666627 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.666638 [ 1096 ] {} <Debug> RaftInstance: commit upto 16, current idx 15
2021.03.24 11:22:46.666648 [ 1096 ] {} <Trace> RaftInstance: commit upto 16, current idx 16
2021.03.24 11:22:46.666688 [ 1096 ] {} <Debug> RaftInstance: notify cb 16 0x7faa0daae020
2021.03.24 11:22:46.666706 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 16, current idx 16
2021.03.24 11:22:46.666727 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.666753 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 16 wake up (13397 us), return value (nil)
2021.03.24 11:22:46.666730 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.666847 [ 1162 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.666875 [ 1162 ] {} <Debug> RaftInstance: append at log_idx 18
2021.03.24 11:22:46.666903 [ 1162 ] {} <Debug> RaftInstance: commit_ret_cv 18 0x7faa1ad62020 sleep
2021.03.24 11:22:46.666923 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.666937 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.666942 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.666988 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=16
2021.03.24 11:22:46.667002 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.667013 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 16
2021.03.24 11:22:46.667021 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.667051 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 12, new matched idx: 15
2021.03.24 11:22:46.667062 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 18 16 15 
2021.03.24 11:22:46.667070 [ 1092 ] {} <Trace> RaftInstance: local log idx 18, target_commit_idx 16, quick_commit_index_ 16, state_->get_commit_idx() 16
2021.03.24 11:22:46.667079 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.667087 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.667095 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 15, starting_idx: 1, cur_nxt_idx: 19
2021.03.24 11:22:46.667106 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=15, LastLogTerm=1, EntriesLength=3, CommitIndex=16, Term=1, peer_last_sent_idx 13
2021.03.24 11:22:46.667113 [ 1092 ] {} <Debug> RaftInstance: idx range: 16-18
2021.03.24 11:22:46.667122 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.667178 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.673266 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.673302 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=18
2021.03.24 11:22:46.673313 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.673323 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 18
2021.03.24 11:22:46.673330 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.673338 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 16, new matched idx: 17
2021.03.24 11:22:46.673349 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 18 17 15 
2021.03.24 11:22:46.673356 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 17
2021.03.24 11:22:46.673407 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.673425 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.673433 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 17, starting_idx: 1, cur_nxt_idx: 19
2021.03.24 11:22:46.673444 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=17, LastLogTerm=1, EntriesLength=1, CommitIndex=17, Term=1, peer_last_sent_idx 17
2021.03.24 11:22:46.673452 [ 1094 ] {} <Debug> RaftInstance: idx: 18
2021.03.24 11:22:46.673539 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.673608 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.673622 [ 1094 ] {} <Trace> RaftInstance: local log idx 18, target_commit_idx 17, quick_commit_index_ 17, state_->get_commit_idx() 16
2021.03.24 11:22:46.673630 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.673644 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.673654 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.673717 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.673771 [ 1096 ] {} <Debug> RaftInstance: commit upto 17, current idx 16
2021.03.24 11:22:46.673783 [ 1096 ] {} <Trace> RaftInstance: commit upto 17, current idx 17
2021.03.24 11:22:46.673814 [ 1096 ] {} <Debug> RaftInstance: notify cb 17 0x7faaa6f93720
2021.03.24 11:22:46.673836 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 17, current idx 17
2021.03.24 11:22:46.673884 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.673859 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.673908 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 17 wake up (7681 us), return value (nil)
2021.03.24 11:22:46.680448 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.680487 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=19
2021.03.24 11:22:46.680498 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.680509 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 19
2021.03.24 11:22:46.680518 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.680526 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 17, new matched idx: 18
2021.03.24 11:22:46.680538 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 18 18 15 
2021.03.24 11:22:46.680545 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 18
2021.03.24 11:22:46.680555 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.680563 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.680571 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 18, starting_idx: 1, cur_nxt_idx: 19
2021.03.24 11:22:46.680582 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=18, LastLogTerm=1, EntriesLength=0, CommitIndex=18, Term=1, peer_last_sent_idx 18
2021.03.24 11:22:46.680590 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.680598 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.680668 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.680682 [ 1092 ] {} <Trace> RaftInstance: local log idx 18, target_commit_idx 18, quick_commit_index_ 18, state_->get_commit_idx() 17
2021.03.24 11:22:46.680689 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.680748 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.680804 [ 1096 ] {} <Debug> RaftInstance: commit upto 18, current idx 17
2021.03.24 11:22:46.680817 [ 1096 ] {} <Trace> RaftInstance: commit upto 18, current idx 18
2021.03.24 11:22:46.680847 [ 1096 ] {} <Debug> RaftInstance: notify cb 18 0x7faa1ad62020
2021.03.24 11:22:46.680870 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 18, current idx 18
2021.03.24 11:22:46.680879 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.680898 [ 1162 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.680927 [ 1162 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 18 wake up (13998 us), return value 0x7faaa9a69ac0
2021.03.24 11:22:46.681092 [ 1163 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.681121 [ 1163 ] {} <Debug> RaftInstance: append at log_idx 19
2021.03.24 11:22:46.681150 [ 1163 ] {} <Debug> RaftInstance: commit_ret_cv 19 0x7faa0db00020 sleep
2021.03.24 11:22:46.681180 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.681203 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.687160 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.687218 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=19
2021.03.24 11:22:46.687229 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.687241 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 19
2021.03.24 11:22:46.687249 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.687257 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 18, new matched idx: 18
2021.03.24 11:22:46.687269 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 19 18 15 
2021.03.24 11:22:46.687278 [ 1093 ] {} <Trace> RaftInstance: local log idx 19, target_commit_idx 18, quick_commit_index_ 18, state_->get_commit_idx() 18
2021.03.24 11:22:46.687286 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.687295 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.687304 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 18, starting_idx: 1, cur_nxt_idx: 20
2021.03.24 11:22:46.687317 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=18, LastLogTerm=1, EntriesLength=1, CommitIndex=18, Term=1, peer_last_sent_idx 19
2021.03.24 11:22:46.687324 [ 1093 ] {} <Debug> RaftInstance: idx: 19
2021.03.24 11:22:46.687333 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.687390 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.687716 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.687743 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=19
2021.03.24 11:22:46.687753 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.687764 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 19
2021.03.24 11:22:46.687772 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.687780 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 15, new matched idx: 18
2021.03.24 11:22:46.687791 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 19 18 18 
2021.03.24 11:22:46.687800 [ 1094 ] {} <Trace> RaftInstance: local log idx 19, target_commit_idx 18, quick_commit_index_ 18, state_->get_commit_idx() 18
2021.03.24 11:22:46.687808 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.687816 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.687824 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 18, starting_idx: 1, cur_nxt_idx: 20
2021.03.24 11:22:46.687836 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=18, LastLogTerm=1, EntriesLength=1, CommitIndex=18, Term=1, peer_last_sent_idx 16
2021.03.24 11:22:46.687843 [ 1094 ] {} <Debug> RaftInstance: idx: 19
2021.03.24 11:22:46.687852 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.687924 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.693726 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.693764 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 20
2021.03.24 11:22:46.693786 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 20 0x7faaa6f93920 sleep
2021.03.24 11:22:46.693837 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.693868 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.693893 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.693947 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=20
2021.03.24 11:22:46.693961 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.693972 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 20
2021.03.24 11:22:46.693981 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.693989 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 18, new matched idx: 19
2021.03.24 11:22:46.693999 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 20 19 18 
2021.03.24 11:22:46.694007 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 19
2021.03.24 11:22:46.694016 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.694023 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.694032 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 19, starting_idx: 1, cur_nxt_idx: 21
2021.03.24 11:22:46.694057 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=19, LastLogTerm=1, EntriesLength=1, CommitIndex=19, Term=1, peer_last_sent_idx 19
2021.03.24 11:22:46.694064 [ 1093 ] {} <Debug> RaftInstance: idx: 20
2021.03.24 11:22:46.694073 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.694140 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.694152 [ 1093 ] {} <Trace> RaftInstance: local log idx 20, target_commit_idx 19, quick_commit_index_ 19, state_->get_commit_idx() 18
2021.03.24 11:22:46.694160 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.694172 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.694181 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.694196 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.694215 [ 1096 ] {} <Debug> RaftInstance: commit upto 19, current idx 18
2021.03.24 11:22:46.694225 [ 1096 ] {} <Trace> RaftInstance: commit upto 19, current idx 19
2021.03.24 11:22:46.694250 [ 1096 ] {} <Debug> RaftInstance: notify cb 19 0x7faa0db00020
2021.03.24 11:22:46.694272 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 19, current idx 19
2021.03.24 11:22:46.694282 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.694293 [ 1163 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.694311 [ 1163 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 19 wake up (13146 us), return value 0x7faaa9a69ad0
2021.03.24 11:22:46.694394 [ 1164 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.694423 [ 1164 ] {} <Debug> RaftInstance: append at log_idx 21
2021.03.24 11:22:46.694453 [ 1164 ] {} <Debug> RaftInstance: commit_ret_cv 21 0x7faa1bb5d020 sleep
2021.03.24 11:22:46.694464 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.694478 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.701056 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.701164 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=21
2021.03.24 11:22:46.701177 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.701188 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 21
2021.03.24 11:22:46.701196 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.701204 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 19, new matched idx: 20
2021.03.24 11:22:46.701216 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 21 20 18 
2021.03.24 11:22:46.701224 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 20
2021.03.24 11:22:46.701233 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.701241 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.701249 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 20, starting_idx: 1, cur_nxt_idx: 22
2021.03.24 11:22:46.701274 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=20, LastLogTerm=1, EntriesLength=1, CommitIndex=20, Term=1, peer_last_sent_idx 20
2021.03.24 11:22:46.701344 [ 1095 ] {} <Debug> RaftInstance: idx: 21
2021.03.24 11:22:46.701361 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.701406 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.701419 [ 1095 ] {} <Trace> RaftInstance: local log idx 21, target_commit_idx 20, quick_commit_index_ 20, state_->get_commit_idx() 19
2021.03.24 11:22:46.701426 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.701444 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.701453 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.701473 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.701497 [ 1096 ] {} <Debug> RaftInstance: commit upto 20, current idx 19
2021.03.24 11:22:46.701508 [ 1096 ] {} <Trace> RaftInstance: commit upto 20, current idx 20
2021.03.24 11:22:46.701545 [ 1096 ] {} <Debug> RaftInstance: notify cb 20 0x7faaa6f93920
2021.03.24 11:22:46.701558 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 20, current idx 20
2021.03.24 11:22:46.701565 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.701598 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.701609 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 20 wake up (7822 us), return value (nil)
2021.03.24 11:22:46.708275 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.708331 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=22
2021.03.24 11:22:46.708343 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.708354 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 22
2021.03.24 11:22:46.708362 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.708371 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 20, new matched idx: 21
2021.03.24 11:22:46.708384 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 21 21 18 
2021.03.24 11:22:46.708392 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 21
2021.03.24 11:22:46.708402 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.708410 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.708440 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 21, starting_idx: 1, cur_nxt_idx: 22
2021.03.24 11:22:46.708453 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=21, LastLogTerm=1, EntriesLength=0, CommitIndex=21, Term=1, peer_last_sent_idx 21
2021.03.24 11:22:46.708460 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.708469 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.708508 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.708547 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.708562 [ 1092 ] {} <Trace> RaftInstance: local log idx 21, target_commit_idx 21, quick_commit_index_ 21, state_->get_commit_idx() 20
2021.03.24 11:22:46.708569 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.708600 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.708601 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=20
2021.03.24 11:22:46.708617 [ 1096 ] {} <Debug> RaftInstance: commit upto 21, current idx 20
2021.03.24 11:22:46.708634 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.708647 [ 1096 ] {} <Trace> RaftInstance: commit upto 21, current idx 21
2021.03.24 11:22:46.708664 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 20
2021.03.24 11:22:46.708687 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.708693 [ 1096 ] {} <Debug> RaftInstance: notify cb 21 0x7faa1bb5d020
2021.03.24 11:22:46.708707 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 21, current idx 21
2021.03.24 11:22:46.708696 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 18, new matched idx: 19
2021.03.24 11:22:46.708720 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.708784 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 21 21 19 
2021.03.24 11:22:46.708800 [ 1093 ] {} <Trace> RaftInstance: local log idx 21, target_commit_idx 21, quick_commit_index_ 21, state_->get_commit_idx() 21
2021.03.24 11:22:46.708808 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.708817 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.708825 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 19, starting_idx: 1, cur_nxt_idx: 22
2021.03.24 11:22:46.708837 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=19, LastLogTerm=1, EntriesLength=2, CommitIndex=21, Term=1, peer_last_sent_idx 19
2021.03.24 11:22:46.708844 [ 1093 ] {} <Debug> RaftInstance: idx range: 20-21
2021.03.24 11:22:46.708853 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.708898 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.708925 [ 1164 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.708938 [ 1164 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 21 wake up (14480 us), return value 0x7faaa9a69ae0
2021.03.24 11:22:46.708982 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.709013 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 22
2021.03.24 11:22:46.709036 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 22 0x7faa0daae120 sleep
2021.03.24 11:22:46.709069 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.709124 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.715142 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.715194 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=22
2021.03.24 11:22:46.715230 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.715242 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 22
2021.03.24 11:22:46.715250 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.715259 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 21, new matched idx: 21
2021.03.24 11:22:46.715271 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 22 21 19 
2021.03.24 11:22:46.715280 [ 1094 ] {} <Trace> RaftInstance: local log idx 22, target_commit_idx 21, quick_commit_index_ 21, state_->get_commit_idx() 21
2021.03.24 11:22:46.715287 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.715296 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.715304 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 21, starting_idx: 1, cur_nxt_idx: 23
2021.03.24 11:22:46.715317 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=21, LastLogTerm=1, EntriesLength=1, CommitIndex=21, Term=1, peer_last_sent_idx 22
2021.03.24 11:22:46.715324 [ 1094 ] {} <Debug> RaftInstance: idx: 22
2021.03.24 11:22:46.715333 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.715425 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.721698 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.721764 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 23
2021.03.24 11:22:46.721788 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 23 0x7faaa6f93a20 sleep
2021.03.24 11:22:46.721817 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.721837 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.722077 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.722103 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=23
2021.03.24 11:22:46.722114 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.722124 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 23
2021.03.24 11:22:46.722132 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.722140 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 21, new matched idx: 22
2021.03.24 11:22:46.722151 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 23 22 19 
2021.03.24 11:22:46.722159 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 22
2021.03.24 11:22:46.722167 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.722175 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.722183 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 22, starting_idx: 1, cur_nxt_idx: 24
2021.03.24 11:22:46.722195 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=22, LastLogTerm=1, EntriesLength=1, CommitIndex=22, Term=1, peer_last_sent_idx 22
2021.03.24 11:22:46.722202 [ 1092 ] {} <Debug> RaftInstance: idx: 23
2021.03.24 11:22:46.722211 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.722281 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.722294 [ 1092 ] {} <Trace> RaftInstance: local log idx 23, target_commit_idx 22, quick_commit_index_ 22, state_->get_commit_idx() 21
2021.03.24 11:22:46.722302 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.722313 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.722321 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.722349 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.722391 [ 1096 ] {} <Debug> RaftInstance: commit upto 22, current idx 21
2021.03.24 11:22:46.722401 [ 1096 ] {} <Trace> RaftInstance: commit upto 22, current idx 22
2021.03.24 11:22:46.722439 [ 1096 ] {} <Debug> RaftInstance: notify cb 22 0x7faa0daae120
2021.03.24 11:22:46.722453 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 22, current idx 22
2021.03.24 11:22:46.722461 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.722587 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.722626 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 22 wake up (13541 us), return value (nil)
2021.03.24 11:22:46.722670 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.722707 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 24
2021.03.24 11:22:46.722727 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 24 0x7faa0daae020 sleep
2021.03.24 11:22:46.722746 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.722762 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.729317 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.729354 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=24
2021.03.24 11:22:46.729367 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.729367 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.729378 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 24
2021.03.24 11:22:46.729446 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.729455 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 22, new matched idx: 23
2021.03.24 11:22:46.729468 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 24 23 19 
2021.03.24 11:22:46.729475 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 23
2021.03.24 11:22:46.729484 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.729493 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 19, starting_idx: 1, cur_nxt_idx: 25
2021.03.24 11:22:46.729501 [ 1092 ] {} <Debug> RaftInstance: last sent log (20) to peer 3 is not applied, cnt 1
2021.03.24 11:22:46.729514 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=19, LastLogTerm=1, EntriesLength=5, CommitIndex=23, Term=1, peer_last_sent_idx 20
2021.03.24 11:22:46.729522 [ 1092 ] {} <Debug> RaftInstance: idx range: 20-24
2021.03.24 11:22:46.729532 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.729604 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.729617 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.729624 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 23, starting_idx: 1, cur_nxt_idx: 25
2021.03.24 11:22:46.729634 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=23, LastLogTerm=1, EntriesLength=1, CommitIndex=23, Term=1, peer_last_sent_idx 23
2021.03.24 11:22:46.729641 [ 1092 ] {} <Debug> RaftInstance: idx: 24
2021.03.24 11:22:46.729650 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.729682 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.729693 [ 1092 ] {} <Trace> RaftInstance: local log idx 24, target_commit_idx 23, quick_commit_index_ 23, state_->get_commit_idx() 22
2021.03.24 11:22:46.729701 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.729718 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.729728 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.729745 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.729787 [ 1096 ] {} <Debug> RaftInstance: commit upto 23, current idx 22
2021.03.24 11:22:46.729799 [ 1096 ] {} <Trace> RaftInstance: commit upto 23, current idx 23
2021.03.24 11:22:46.729825 [ 1096 ] {} <Debug> RaftInstance: notify cb 23 0x7faaa6f93a20
2021.03.24 11:22:46.729848 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 23, current idx 23
2021.03.24 11:22:46.729856 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.729870 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.729893 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 23 wake up (8085 us), return value 0x7faaa9a69af0
2021.03.24 11:22:46.729873 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=22
2021.03.24 11:22:46.729936 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.729949 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 22
2021.03.24 11:22:46.729957 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.729965 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 19, new matched idx: 21
2021.03.24 11:22:46.729976 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 24 23 21 
2021.03.24 11:22:46.729984 [ 1094 ] {} <Trace> RaftInstance: local log idx 24, target_commit_idx 23, quick_commit_index_ 23, state_->get_commit_idx() 23
2021.03.24 11:22:46.729993 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.730001 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.733655 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.733688 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 25
2021.03.24 11:22:46.733703 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 25 0x7faaa6f93520 sleep
2021.03.24 11:22:46.733731 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.733764 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.736582 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.736610 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=25
2021.03.24 11:22:46.736620 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.736631 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 25
2021.03.24 11:22:46.736639 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.736648 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 23, new matched idx: 24
2021.03.24 11:22:46.736659 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 25 24 21 
2021.03.24 11:22:46.736667 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 24
2021.03.24 11:22:46.736675 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.736683 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.736691 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 24, starting_idx: 1, cur_nxt_idx: 26
2021.03.24 11:22:46.736703 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=24, LastLogTerm=1, EntriesLength=1, CommitIndex=24, Term=1, peer_last_sent_idx 24
2021.03.24 11:22:46.736710 [ 1095 ] {} <Debug> RaftInstance: idx: 25
2021.03.24 11:22:46.736719 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.736792 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.736805 [ 1095 ] {} <Trace> RaftInstance: local log idx 25, target_commit_idx 24, quick_commit_index_ 24, state_->get_commit_idx() 23
2021.03.24 11:22:46.736836 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.736853 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.736862 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.736872 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.736894 [ 1096 ] {} <Debug> RaftInstance: commit upto 24, current idx 23
2021.03.24 11:22:46.736905 [ 1096 ] {} <Trace> RaftInstance: commit upto 24, current idx 24
2021.03.24 11:22:46.736943 [ 1096 ] {} <Debug> RaftInstance: notify cb 24 0x7faa0daae020
2021.03.24 11:22:46.736957 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 24, current idx 24
2021.03.24 11:22:46.736965 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.737003 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.737016 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 24 wake up (14280 us), return value (nil)
2021.03.24 11:22:46.737049 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.737071 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 26
2021.03.24 11:22:46.737109 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 26 0x7faa0daae120 sleep
2021.03.24 11:22:46.737127 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.737137 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.742428 [ 1094 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:46.742475 [ 1094 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:46.742486 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.743575 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.743598 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=26
2021.03.24 11:22:46.743609 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.743619 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 26
2021.03.24 11:22:46.743628 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.743636 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 24, new matched idx: 25
2021.03.24 11:22:46.743647 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 26 25 21 
2021.03.24 11:22:46.743655 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 25
2021.03.24 11:22:46.743663 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.743671 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.743680 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 25, starting_idx: 1, cur_nxt_idx: 27
2021.03.24 11:22:46.743692 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=25, LastLogTerm=1, EntriesLength=1, CommitIndex=25, Term=1, peer_last_sent_idx 25
2021.03.24 11:22:46.743699 [ 1094 ] {} <Debug> RaftInstance: idx: 26
2021.03.24 11:22:46.743708 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.743779 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.743792 [ 1094 ] {} <Trace> RaftInstance: local log idx 26, target_commit_idx 25, quick_commit_index_ 25, state_->get_commit_idx() 24
2021.03.24 11:22:46.743800 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.743811 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.743820 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.743847 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.743859 [ 1096 ] {} <Debug> RaftInstance: commit upto 25, current idx 24
2021.03.24 11:22:46.743868 [ 1096 ] {} <Trace> RaftInstance: commit upto 25, current idx 25
2021.03.24 11:22:46.743922 [ 1096 ] {} <Debug> RaftInstance: notify cb 25 0x7faaa6f93520
2021.03.24 11:22:46.743943 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 25, current idx 25
2021.03.24 11:22:46.743951 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.743978 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.744003 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 25 wake up (10269 us), return value 0x7faaa9a69b00
2021.03.24 11:22:46.749875 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.749943 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 27
2021.03.24 11:22:46.749966 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 27 0x7faa1f09b020 sleep
2021.03.24 11:22:46.749990 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.750008 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.750293 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.750311 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=25
2021.03.24 11:22:46.750322 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.750332 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 25
2021.03.24 11:22:46.750340 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.750348 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 21, new matched idx: 24
2021.03.24 11:22:46.750358 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 27 25 24 
2021.03.24 11:22:46.750368 [ 1092 ] {} <Trace> RaftInstance: local log idx 27, target_commit_idx 25, quick_commit_index_ 25, state_->get_commit_idx() 25
2021.03.24 11:22:46.750376 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.750383 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.750392 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 24, starting_idx: 1, cur_nxt_idx: 28
2021.03.24 11:22:46.750403 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=24, LastLogTerm=1, EntriesLength=3, CommitIndex=25, Term=1, peer_last_sent_idx 20
2021.03.24 11:22:46.750410 [ 1092 ] {} <Debug> RaftInstance: idx range: 25-27
2021.03.24 11:22:46.750419 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.750504 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.750542 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.750555 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=27
2021.03.24 11:22:46.750564 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.750573 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 27
2021.03.24 11:22:46.750580 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.750587 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 25, new matched idx: 26
2021.03.24 11:22:46.750596 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 27 26 24 
2021.03.24 11:22:46.750604 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 26
2021.03.24 11:22:46.750611 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.750619 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.750626 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 26, starting_idx: 1, cur_nxt_idx: 28
2021.03.24 11:22:46.750635 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=26, LastLogTerm=1, EntriesLength=1, CommitIndex=26, Term=1, peer_last_sent_idx 26
2021.03.24 11:22:46.750666 [ 1094 ] {} <Debug> RaftInstance: idx: 27
2021.03.24 11:22:46.750675 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.750706 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.750717 [ 1094 ] {} <Trace> RaftInstance: local log idx 27, target_commit_idx 26, quick_commit_index_ 26, state_->get_commit_idx() 25
2021.03.24 11:22:46.750724 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.750736 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.750744 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.750803 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.750818 [ 1096 ] {} <Debug> RaftInstance: commit upto 26, current idx 25
2021.03.24 11:22:46.750840 [ 1096 ] {} <Trace> RaftInstance: commit upto 26, current idx 26
2021.03.24 11:22:46.750874 [ 1096 ] {} <Debug> RaftInstance: notify cb 26 0x7faa0daae120
2021.03.24 11:22:46.750889 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 26, current idx 26
2021.03.24 11:22:46.750897 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.750916 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.750928 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 26 wake up (13809 us), return value (nil)
2021.03.24 11:22:46.750999 [ 1165 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.751032 [ 1165 ] {} <Debug> RaftInstance: append at log_idx 28
2021.03.24 11:22:46.751062 [ 1165 ] {} <Debug> RaftInstance: commit_ret_cv 28 0x7faa0db3d020 sleep
2021.03.24 11:22:46.751083 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.751099 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.757442 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.757468 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=28
2021.03.24 11:22:46.757477 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.757487 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 28
2021.03.24 11:22:46.757495 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.757504 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 26, new matched idx: 27
2021.03.24 11:22:46.757515 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 28 27 24 
2021.03.24 11:22:46.757523 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 27
2021.03.24 11:22:46.757531 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.757539 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.757548 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 27, starting_idx: 1, cur_nxt_idx: 29
2021.03.24 11:22:46.757558 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=27, LastLogTerm=1, EntriesLength=1, CommitIndex=27, Term=1, peer_last_sent_idx 27
2021.03.24 11:22:46.757566 [ 1093 ] {} <Debug> RaftInstance: idx: 28
2021.03.24 11:22:46.757574 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.757626 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.757639 [ 1093 ] {} <Trace> RaftInstance: local log idx 28, target_commit_idx 27, quick_commit_index_ 27, state_->get_commit_idx() 26
2021.03.24 11:22:46.757647 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.757662 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.757671 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.757706 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.757760 [ 1096 ] {} <Debug> RaftInstance: commit upto 27, current idx 26
2021.03.24 11:22:46.757771 [ 1096 ] {} <Trace> RaftInstance: commit upto 27, current idx 27
2021.03.24 11:22:46.757840 [ 1096 ] {} <Debug> RaftInstance: notify cb 27 0x7faa1f09b020
2021.03.24 11:22:46.757872 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 27, current idx 27
2021.03.24 11:22:46.757881 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.757925 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.757948 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 27 wake up (7959 us), return value (nil)
2021.03.24 11:22:46.759052 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.759083 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 29
2021.03.24 11:22:46.759099 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 29 0x7faaa6f93e20 sleep
2021.03.24 11:22:46.759119 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.759129 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.764507 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.764528 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=29
2021.03.24 11:22:46.764536 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.764545 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 29
2021.03.24 11:22:46.764553 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.764560 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 27, new matched idx: 28
2021.03.24 11:22:46.764570 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 29 28 24 
2021.03.24 11:22:46.764577 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 28
2021.03.24 11:22:46.764585 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.764593 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.764601 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 28, starting_idx: 1, cur_nxt_idx: 30
2021.03.24 11:22:46.764611 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=28, LastLogTerm=1, EntriesLength=1, CommitIndex=28, Term=1, peer_last_sent_idx 28
2021.03.24 11:22:46.764618 [ 1093 ] {} <Debug> RaftInstance: idx: 29
2021.03.24 11:22:46.764626 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.764668 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.764681 [ 1093 ] {} <Trace> RaftInstance: local log idx 29, target_commit_idx 28, quick_commit_index_ 28, state_->get_commit_idx() 27
2021.03.24 11:22:46.764689 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.764702 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.764711 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.764743 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.764787 [ 1096 ] {} <Debug> RaftInstance: commit upto 28, current idx 27
2021.03.24 11:22:46.764799 [ 1096 ] {} <Trace> RaftInstance: commit upto 28, current idx 28
2021.03.24 11:22:46.764846 [ 1096 ] {} <Debug> RaftInstance: notify cb 28 0x7faa0db3d020
2021.03.24 11:22:46.764869 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 28, current idx 28
2021.03.24 11:22:46.764877 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.764887 [ 1165 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.764903 [ 1165 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 28 wake up (13831 us), return value 0x7faaa9a69b10
2021.03.24 11:22:46.765016 [ 1166 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.765039 [ 1166 ] {} <Debug> RaftInstance: append at log_idx 30
2021.03.24 11:22:46.765061 [ 1166 ] {} <Debug> RaftInstance: commit_ret_cv 30 0x7faaa6fb9f20 sleep
2021.03.24 11:22:46.765113 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.765142 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.770945 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.770998 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=28
2021.03.24 11:22:46.771008 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.771020 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 28
2021.03.24 11:22:46.771028 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.771036 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 24, new matched idx: 27
2021.03.24 11:22:46.771048 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 30 28 27 
2021.03.24 11:22:46.771056 [ 1095 ] {} <Trace> RaftInstance: local log idx 30, target_commit_idx 28, quick_commit_index_ 28, state_->get_commit_idx() 28
2021.03.24 11:22:46.771065 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.771073 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.771081 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 27, starting_idx: 1, cur_nxt_idx: 31
2021.03.24 11:22:46.771094 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=27, LastLogTerm=1, EntriesLength=3, CommitIndex=28, Term=1, peer_last_sent_idx 25
2021.03.24 11:22:46.771102 [ 1095 ] {} <Debug> RaftInstance: idx range: 28-30
2021.03.24 11:22:46.771112 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.771196 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.771468 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.771499 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=30
2021.03.24 11:22:46.771510 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.771519 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 30
2021.03.24 11:22:46.771527 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.771535 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 28, new matched idx: 29
2021.03.24 11:22:46.771544 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 30 29 27 
2021.03.24 11:22:46.771559 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 29
2021.03.24 11:22:46.771568 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.771576 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.771584 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 29, starting_idx: 1, cur_nxt_idx: 31
2021.03.24 11:22:46.771594 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=29, LastLogTerm=1, EntriesLength=1, CommitIndex=29, Term=1, peer_last_sent_idx 29
2021.03.24 11:22:46.771625 [ 1092 ] {} <Debug> RaftInstance: idx: 30
2021.03.24 11:22:46.771635 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.771681 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.771695 [ 1092 ] {} <Trace> RaftInstance: local log idx 30, target_commit_idx 29, quick_commit_index_ 29, state_->get_commit_idx() 28
2021.03.24 11:22:46.771703 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.771715 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.771724 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.771743 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.771755 [ 1096 ] {} <Debug> RaftInstance: commit upto 29, current idx 28
2021.03.24 11:22:46.771764 [ 1096 ] {} <Trace> RaftInstance: commit upto 29, current idx 29
2021.03.24 11:22:46.771797 [ 1096 ] {} <Debug> RaftInstance: notify cb 29 0x7faaa6f93e20
2021.03.24 11:22:46.771812 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 29, current idx 29
2021.03.24 11:22:46.771820 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.771839 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.771861 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 29 wake up (12739 us), return value (nil)
2021.03.24 11:22:46.777958 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.777988 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 31
2021.03.24 11:22:46.778007 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 31 0x7faaa6f93120 sleep
2021.03.24 11:22:46.778054 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.778107 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.778455 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.778479 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=31
2021.03.24 11:22:46.778490 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.778500 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 31
2021.03.24 11:22:46.778509 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.778516 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 29, new matched idx: 30
2021.03.24 11:22:46.778527 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 31 30 27 
2021.03.24 11:22:46.778535 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 30
2021.03.24 11:22:46.778543 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.778551 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.778559 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 30, starting_idx: 1, cur_nxt_idx: 32
2021.03.24 11:22:46.778571 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=30, LastLogTerm=1, EntriesLength=1, CommitIndex=30, Term=1, peer_last_sent_idx 30
2021.03.24 11:22:46.778578 [ 1092 ] {} <Debug> RaftInstance: idx: 31
2021.03.24 11:22:46.778587 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.778693 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.778707 [ 1092 ] {} <Trace> RaftInstance: local log idx 31, target_commit_idx 30, quick_commit_index_ 30, state_->get_commit_idx() 29
2021.03.24 11:22:46.778715 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.778729 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.778738 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.778760 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.778819 [ 1096 ] {} <Debug> RaftInstance: commit upto 30, current idx 29
2021.03.24 11:22:46.778830 [ 1096 ] {} <Trace> RaftInstance: commit upto 30, current idx 30
2021.03.24 11:22:46.778857 [ 1096 ] {} <Debug> RaftInstance: notify cb 30 0x7faaa6fb9f20
2021.03.24 11:22:46.778876 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 30, current idx 30
2021.03.24 11:22:46.778885 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.778898 [ 1166 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.778919 [ 1166 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 30 wake up (13839 us), return value 0x7faaa9a69b20
2021.03.24 11:22:46.778971 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.779003 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 32
2021.03.24 11:22:46.779019 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 32 0x7faa0daae020 sleep
2021.03.24 11:22:46.779037 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.779046 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.785299 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.785329 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=32
2021.03.24 11:22:46.785339 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.785350 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 32
2021.03.24 11:22:46.785358 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.785366 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 30, new matched idx: 31
2021.03.24 11:22:46.785377 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 32 31 27 
2021.03.24 11:22:46.785385 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 31
2021.03.24 11:22:46.785393 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.785401 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.785410 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 31, starting_idx: 1, cur_nxt_idx: 33
2021.03.24 11:22:46.785422 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=31, LastLogTerm=1, EntriesLength=1, CommitIndex=31, Term=1, peer_last_sent_idx 31
2021.03.24 11:22:46.785430 [ 1093 ] {} <Debug> RaftInstance: idx: 32
2021.03.24 11:22:46.785439 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.785489 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.785502 [ 1093 ] {} <Trace> RaftInstance: local log idx 32, target_commit_idx 31, quick_commit_index_ 31, state_->get_commit_idx() 30
2021.03.24 11:22:46.785510 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.785520 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.785528 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.785566 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.785624 [ 1096 ] {} <Debug> RaftInstance: commit upto 31, current idx 30
2021.03.24 11:22:46.785637 [ 1096 ] {} <Trace> RaftInstance: commit upto 31, current idx 31
2021.03.24 11:22:46.785677 [ 1096 ] {} <Debug> RaftInstance: notify cb 31 0x7faaa6f93120
2021.03.24 11:22:46.785699 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 31, current idx 31
2021.03.24 11:22:46.785709 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.785733 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.785754 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 31 wake up (7728 us), return value (nil)
2021.03.24 11:22:46.786453 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.786482 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 33
2021.03.24 11:22:46.786502 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 33 0x7faaa6f93a20 sleep
2021.03.24 11:22:46.786529 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.786546 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.791904 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.791928 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=31
2021.03.24 11:22:46.791937 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.791946 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 31
2021.03.24 11:22:46.791954 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.791971 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 27, new matched idx: 30
2021.03.24 11:22:46.791981 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 33 31 30 
2021.03.24 11:22:46.791989 [ 1092 ] {} <Trace> RaftInstance: local log idx 33, target_commit_idx 31, quick_commit_index_ 31, state_->get_commit_idx() 31
2021.03.24 11:22:46.791997 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.792005 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.792012 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 30, starting_idx: 1, cur_nxt_idx: 34
2021.03.24 11:22:46.792023 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=30, LastLogTerm=1, EntriesLength=3, CommitIndex=31, Term=1, peer_last_sent_idx 28
2021.03.24 11:22:46.792031 [ 1092 ] {} <Debug> RaftInstance: idx range: 31-33
2021.03.24 11:22:46.792039 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.792082 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.792167 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.792186 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=33
2021.03.24 11:22:46.792207 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.792217 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 33
2021.03.24 11:22:46.792225 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.792232 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 31, new matched idx: 32
2021.03.24 11:22:46.792241 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 33 32 30 
2021.03.24 11:22:46.792249 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 32
2021.03.24 11:22:46.792257 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.792264 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.792272 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 32, starting_idx: 1, cur_nxt_idx: 34
2021.03.24 11:22:46.792282 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=32, LastLogTerm=1, EntriesLength=1, CommitIndex=32, Term=1, peer_last_sent_idx 32
2021.03.24 11:22:46.792290 [ 1095 ] {} <Debug> RaftInstance: idx: 33
2021.03.24 11:22:46.792298 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.792339 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.792352 [ 1095 ] {} <Trace> RaftInstance: local log idx 33, target_commit_idx 32, quick_commit_index_ 32, state_->get_commit_idx() 31
2021.03.24 11:22:46.792359 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.792384 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.792393 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.792489 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.792556 [ 1096 ] {} <Debug> RaftInstance: commit upto 32, current idx 31
2021.03.24 11:22:46.792568 [ 1096 ] {} <Trace> RaftInstance: commit upto 32, current idx 32
2021.03.24 11:22:46.792622 [ 1096 ] {} <Debug> RaftInstance: notify cb 32 0x7faa0daae020
2021.03.24 11:22:46.792640 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 32, current idx 32
2021.03.24 11:22:46.792647 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.792671 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.792689 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 32 wake up (13650 us), return value (nil)
2021.03.24 11:22:46.792725 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.792752 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 34
2021.03.24 11:22:46.792771 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 34 0x7faa0daae120 sleep
2021.03.24 11:22:46.792835 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.792857 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.799304 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.799329 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=34
2021.03.24 11:22:46.799338 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.799348 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 34
2021.03.24 11:22:46.799355 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.799363 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 32, new matched idx: 33
2021.03.24 11:22:46.799373 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 34 33 30 
2021.03.24 11:22:46.799380 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 33
2021.03.24 11:22:46.799388 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.799396 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.799407 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 33, starting_idx: 1, cur_nxt_idx: 35
2021.03.24 11:22:46.799418 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=33, LastLogTerm=1, EntriesLength=1, CommitIndex=33, Term=1, peer_last_sent_idx 33
2021.03.24 11:22:46.799425 [ 1092 ] {} <Debug> RaftInstance: idx: 34
2021.03.24 11:22:46.799433 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.799480 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.799494 [ 1092 ] {} <Trace> RaftInstance: local log idx 34, target_commit_idx 33, quick_commit_index_ 33, state_->get_commit_idx() 32
2021.03.24 11:22:46.799505 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.799521 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.799530 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.799589 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.799643 [ 1096 ] {} <Debug> RaftInstance: commit upto 33, current idx 32
2021.03.24 11:22:46.799656 [ 1096 ] {} <Trace> RaftInstance: commit upto 33, current idx 33
2021.03.24 11:22:46.799724 [ 1096 ] {} <Debug> RaftInstance: notify cb 33 0x7faaa6f93a20
2021.03.24 11:22:46.799767 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 33, current idx 33
2021.03.24 11:22:46.799776 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.799783 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.799801 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 33 wake up (13284 us), return value (nil)
2021.03.24 11:22:46.806220 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.806240 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=35
2021.03.24 11:22:46.806249 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.806257 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 35
2021.03.24 11:22:46.806265 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.806279 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 33, new matched idx: 34
2021.03.24 11:22:46.806289 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 34 34 30 
2021.03.24 11:22:46.806296 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 34
2021.03.24 11:22:46.806308 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.806316 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.806324 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 34, starting_idx: 1, cur_nxt_idx: 35
2021.03.24 11:22:46.806333 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=34, LastLogTerm=1, EntriesLength=0, CommitIndex=34, Term=1, peer_last_sent_idx 34
2021.03.24 11:22:46.806341 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.806350 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.806387 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.806400 [ 1092 ] {} <Trace> RaftInstance: local log idx 34, target_commit_idx 34, quick_commit_index_ 34, state_->get_commit_idx() 33
2021.03.24 11:22:46.806407 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.806494 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.806549 [ 1096 ] {} <Debug> RaftInstance: commit upto 34, current idx 33
2021.03.24 11:22:46.806563 [ 1096 ] {} <Trace> RaftInstance: commit upto 34, current idx 34
2021.03.24 11:22:46.806614 [ 1096 ] {} <Debug> RaftInstance: notify cb 34 0x7faa0daae120
2021.03.24 11:22:46.806632 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 34, current idx 34
2021.03.24 11:22:46.806640 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.806666 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.806681 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 34 wake up (13899 us), return value (nil)
2021.03.24 11:22:46.806709 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.806736 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 35
2021.03.24 11:22:46.806755 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 35 0x7faa0daae020 sleep
2021.03.24 11:22:46.806802 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.806819 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.812575 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.812631 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=34
2021.03.24 11:22:46.812643 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.812655 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 34
2021.03.24 11:22:46.812663 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.812696 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 30, new matched idx: 33
2021.03.24 11:22:46.812709 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 35 34 33 
2021.03.24 11:22:46.812718 [ 1095 ] {} <Trace> RaftInstance: local log idx 35, target_commit_idx 34, quick_commit_index_ 34, state_->get_commit_idx() 34
2021.03.24 11:22:46.812726 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.812735 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.812743 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 33, starting_idx: 1, cur_nxt_idx: 36
2021.03.24 11:22:46.812755 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=33, LastLogTerm=1, EntriesLength=2, CommitIndex=34, Term=1, peer_last_sent_idx 31
2021.03.24 11:22:46.812763 [ 1095 ] {} <Debug> RaftInstance: idx range: 34-35
2021.03.24 11:22:46.812773 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.812859 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.812975 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.812998 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=35
2021.03.24 11:22:46.813008 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.813017 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 35
2021.03.24 11:22:46.813025 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.813033 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 34, new matched idx: 34
2021.03.24 11:22:46.813042 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 35 34 33 
2021.03.24 11:22:46.813050 [ 1094 ] {} <Trace> RaftInstance: local log idx 35, target_commit_idx 34, quick_commit_index_ 34, state_->get_commit_idx() 34
2021.03.24 11:22:46.813058 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.813066 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.813101 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 34, starting_idx: 1, cur_nxt_idx: 36
2021.03.24 11:22:46.813117 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=34, LastLogTerm=1, EntriesLength=1, CommitIndex=34, Term=1, peer_last_sent_idx 35
2021.03.24 11:22:46.813124 [ 1094 ] {} <Debug> RaftInstance: idx: 35
2021.03.24 11:22:46.813135 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.813183 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.819981 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.820015 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=36
2021.03.24 11:22:46.820024 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.820034 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 36
2021.03.24 11:22:46.820041 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.820050 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 34, new matched idx: 35
2021.03.24 11:22:46.820061 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 35 35 33 
2021.03.24 11:22:46.820069 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 35
2021.03.24 11:22:46.820077 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.820085 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.820093 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 35, starting_idx: 1, cur_nxt_idx: 36
2021.03.24 11:22:46.820103 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=35, LastLogTerm=1, EntriesLength=0, CommitIndex=35, Term=1, peer_last_sent_idx 35
2021.03.24 11:22:46.820130 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.820139 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.820231 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.820246 [ 1093 ] {} <Trace> RaftInstance: local log idx 35, target_commit_idx 35, quick_commit_index_ 35, state_->get_commit_idx() 34
2021.03.24 11:22:46.820254 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.820337 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.820391 [ 1096 ] {} <Debug> RaftInstance: commit upto 35, current idx 34
2021.03.24 11:22:46.820404 [ 1096 ] {} <Trace> RaftInstance: commit upto 35, current idx 35
2021.03.24 11:22:46.820457 [ 1096 ] {} <Debug> RaftInstance: notify cb 35 0x7faa0daae020
2021.03.24 11:22:46.820475 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 35, current idx 35
2021.03.24 11:22:46.820484 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.820509 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.820526 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 35 wake up (13756 us), return value (nil)
2021.03.24 11:22:46.820560 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.820582 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 36
2021.03.24 11:22:46.820600 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 36 0x7faa0daae120 sleep
2021.03.24 11:22:46.820631 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.820649 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.826955 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.826984 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=36
2021.03.24 11:22:46.826994 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.827003 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 36
2021.03.24 11:22:46.827011 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.827019 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 35, new matched idx: 35
2021.03.24 11:22:46.827030 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 36 35 33 
2021.03.24 11:22:46.827038 [ 1093 ] {} <Trace> RaftInstance: local log idx 36, target_commit_idx 35, quick_commit_index_ 35, state_->get_commit_idx() 35
2021.03.24 11:22:46.827046 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.827055 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.827063 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 35, starting_idx: 1, cur_nxt_idx: 37
2021.03.24 11:22:46.827074 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=35, LastLogTerm=1, EntriesLength=1, CommitIndex=35, Term=1, peer_last_sent_idx 36
2021.03.24 11:22:46.827082 [ 1093 ] {} <Debug> RaftInstance: idx: 36
2021.03.24 11:22:46.827091 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.827144 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.833143 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.833175 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 37
2021.03.24 11:22:46.833196 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 37 0x7faaa6f93a20 sleep
2021.03.24 11:22:46.833291 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.833346 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.833534 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.833583 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=36
2021.03.24 11:22:46.833593 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.833604 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 36
2021.03.24 11:22:46.833612 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.833620 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 33, new matched idx: 35
2021.03.24 11:22:46.833630 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 37 35 35 
2021.03.24 11:22:46.833640 [ 1095 ] {} <Trace> RaftInstance: local log idx 37, target_commit_idx 35, quick_commit_index_ 35, state_->get_commit_idx() 35
2021.03.24 11:22:46.833648 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.833656 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.833664 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 35, starting_idx: 1, cur_nxt_idx: 38
2021.03.24 11:22:46.833676 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=35, LastLogTerm=1, EntriesLength=2, CommitIndex=35, Term=1, peer_last_sent_idx 34
2021.03.24 11:22:46.833683 [ 1095 ] {} <Debug> RaftInstance: idx range: 36-37
2021.03.24 11:22:46.833692 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.833778 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.833811 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.833822 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=37
2021.03.24 11:22:46.833830 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.833838 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 37
2021.03.24 11:22:46.833845 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.833852 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 35, new matched idx: 36
2021.03.24 11:22:46.833861 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 37 36 35 
2021.03.24 11:22:46.833868 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 36
2021.03.24 11:22:46.833876 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.833883 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.833890 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 36, starting_idx: 1, cur_nxt_idx: 38
2021.03.24 11:22:46.833900 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=36, LastLogTerm=1, EntriesLength=1, CommitIndex=36, Term=1, peer_last_sent_idx 36
2021.03.24 11:22:46.833906 [ 1095 ] {} <Debug> RaftInstance: idx: 37
2021.03.24 11:22:46.833915 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.833945 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.833957 [ 1095 ] {} <Trace> RaftInstance: local log idx 37, target_commit_idx 36, quick_commit_index_ 36, state_->get_commit_idx() 35
2021.03.24 11:22:46.833964 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.833982 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.833991 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.834013 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.834036 [ 1096 ] {} <Debug> RaftInstance: commit upto 36, current idx 35
2021.03.24 11:22:46.834047 [ 1096 ] {} <Trace> RaftInstance: commit upto 36, current idx 36
2021.03.24 11:22:46.834083 [ 1096 ] {} <Debug> RaftInstance: notify cb 36 0x7faa0daae120
2021.03.24 11:22:46.834101 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 36, current idx 36
2021.03.24 11:22:46.834120 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.834149 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.834167 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 36 wake up (13551 us), return value (nil)
2021.03.24 11:22:46.834198 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.834214 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 38
2021.03.24 11:22:46.834233 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 38 0x7faa0daae020 sleep
2021.03.24 11:22:46.834257 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.834272 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.841369 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.841428 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=38
2021.03.24 11:22:46.841439 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.841450 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 38
2021.03.24 11:22:46.841458 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.841466 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 36, new matched idx: 37
2021.03.24 11:22:46.841478 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 38 37 35 
2021.03.24 11:22:46.841486 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 37
2021.03.24 11:22:46.841495 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.841504 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.841512 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 37, starting_idx: 1, cur_nxt_idx: 39
2021.03.24 11:22:46.841524 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=37, LastLogTerm=1, EntriesLength=1, CommitIndex=37, Term=1, peer_last_sent_idx 37
2021.03.24 11:22:46.841531 [ 1095 ] {} <Debug> RaftInstance: idx: 38
2021.03.24 11:22:46.841540 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.841612 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.841625 [ 1095 ] {} <Trace> RaftInstance: local log idx 38, target_commit_idx 37, quick_commit_index_ 37, state_->get_commit_idx() 36
2021.03.24 11:22:46.841633 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.841655 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.841664 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.841696 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.841727 [ 1096 ] {} <Debug> RaftInstance: commit upto 37, current idx 36
2021.03.24 11:22:46.841738 [ 1096 ] {} <Trace> RaftInstance: commit upto 37, current idx 37
2021.03.24 11:22:46.841786 [ 1096 ] {} <Debug> RaftInstance: notify cb 37 0x7faaa6f93a20
2021.03.24 11:22:46.841800 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 37, current idx 37
2021.03.24 11:22:46.841807 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.841835 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.841863 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 37 wake up (8638 us), return value (nil)
2021.03.24 11:22:46.848236 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.848288 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=39
2021.03.24 11:22:46.848298 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.848309 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 39
2021.03.24 11:22:46.848343 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.848353 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 37, new matched idx: 38
2021.03.24 11:22:46.848364 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 38 38 35 
2021.03.24 11:22:46.848372 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 38
2021.03.24 11:22:46.848381 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.848389 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.848397 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 38, starting_idx: 1, cur_nxt_idx: 39
2021.03.24 11:22:46.848408 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=38, LastLogTerm=1, EntriesLength=0, CommitIndex=38, Term=1, peer_last_sent_idx 38
2021.03.24 11:22:46.848415 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.848423 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.848493 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.848506 [ 1095 ] {} <Trace> RaftInstance: local log idx 38, target_commit_idx 38, quick_commit_index_ 38, state_->get_commit_idx() 37
2021.03.24 11:22:46.848514 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.848549 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.848571 [ 1096 ] {} <Debug> RaftInstance: commit upto 38, current idx 37
2021.03.24 11:22:46.848582 [ 1096 ] {} <Trace> RaftInstance: commit upto 38, current idx 38
2021.03.24 11:22:46.848618 [ 1096 ] {} <Debug> RaftInstance: notify cb 38 0x7faa0daae020
2021.03.24 11:22:46.848637 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 38, current idx 38
2021.03.24 11:22:46.848646 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.848677 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.848690 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 38 wake up (14449 us), return value (nil)
2021.03.24 11:22:46.848758 [ 1167 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.848797 [ 1167 ] {} <Debug> RaftInstance: append at log_idx 39
2021.03.24 11:22:46.848841 [ 1167 ] {} <Debug> RaftInstance: commit_ret_cv 39 0x7faa1ad97020 sleep
2021.03.24 11:22:46.848880 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.848900 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.854461 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.854499 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=38
2021.03.24 11:22:46.854510 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.854521 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 38
2021.03.24 11:22:46.854537 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.854545 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 35, new matched idx: 37
2021.03.24 11:22:46.854555 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 39 38 37 
2021.03.24 11:22:46.854564 [ 1092 ] {} <Trace> RaftInstance: local log idx 39, target_commit_idx 38, quick_commit_index_ 38, state_->get_commit_idx() 38
2021.03.24 11:22:46.854572 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.854580 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.854588 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 37, starting_idx: 1, cur_nxt_idx: 40
2021.03.24 11:22:46.854599 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=37, LastLogTerm=1, EntriesLength=2, CommitIndex=38, Term=1, peer_last_sent_idx 36
2021.03.24 11:22:46.854625 [ 1092 ] {} <Debug> RaftInstance: idx range: 38-39
2021.03.24 11:22:46.854634 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.854697 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.855154 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.855178 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=39
2021.03.24 11:22:46.855188 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.855197 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 39
2021.03.24 11:22:46.855205 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.855213 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 38, new matched idx: 38
2021.03.24 11:22:46.855226 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 39 38 37 
2021.03.24 11:22:46.855234 [ 1094 ] {} <Trace> RaftInstance: local log idx 39, target_commit_idx 38, quick_commit_index_ 38, state_->get_commit_idx() 38
2021.03.24 11:22:46.855242 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.855250 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.855258 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 38, starting_idx: 1, cur_nxt_idx: 40
2021.03.24 11:22:46.855269 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=38, LastLogTerm=1, EntriesLength=1, CommitIndex=38, Term=1, peer_last_sent_idx 39
2021.03.24 11:22:46.855276 [ 1094 ] {} <Debug> RaftInstance: idx: 39
2021.03.24 11:22:46.855284 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.855327 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.861689 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.861760 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 40
2021.03.24 11:22:46.861781 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 40 0x7faaa6f93a20 sleep
2021.03.24 11:22:46.861795 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.861826 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 41
2021.03.24 11:22:46.861840 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 41 0x7faaa6f93920 sleep
2021.03.24 11:22:46.861850 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.861855 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.861864 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.861907 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=40
2021.03.24 11:22:46.861921 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.861931 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 40
2021.03.24 11:22:46.861940 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.861948 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 38, new matched idx: 39
2021.03.24 11:22:46.861959 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 41 39 37 
2021.03.24 11:22:46.861967 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 39
2021.03.24 11:22:46.861975 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.861983 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.861992 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 39, starting_idx: 1, cur_nxt_idx: 42
2021.03.24 11:22:46.862003 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=39, LastLogTerm=1, EntriesLength=2, CommitIndex=39, Term=1, peer_last_sent_idx 39
2021.03.24 11:22:46.862037 [ 1095 ] {} <Debug> RaftInstance: idx range: 40-41
2021.03.24 11:22:46.862046 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.862119 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.862132 [ 1095 ] {} <Trace> RaftInstance: local log idx 41, target_commit_idx 39, quick_commit_index_ 39, state_->get_commit_idx() 38
2021.03.24 11:22:46.862140 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.862156 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.862166 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.862234 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.862283 [ 1096 ] {} <Debug> RaftInstance: commit upto 39, current idx 38
2021.03.24 11:22:46.862295 [ 1096 ] {} <Trace> RaftInstance: commit upto 39, current idx 39
2021.03.24 11:22:46.862323 [ 1096 ] {} <Debug> RaftInstance: notify cb 39 0x7faa1ad97020
2021.03.24 11:22:46.862351 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 39, current idx 39
2021.03.24 11:22:46.862359 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.862375 [ 1167 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:46.862394 [ 1167 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 39 wake up (13537 us), return value 0x7faaa9a69b30
2021.03.24 11:22:46.862444 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.862474 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 42
2021.03.24 11:22:46.862495 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 42 0x7faa0daae120 sleep
2021.03.24 11:22:46.862517 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.862530 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.868928 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.868974 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=42
2021.03.24 11:22:46.868985 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.868995 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 42
2021.03.24 11:22:46.869004 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.869012 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 39, new matched idx: 41
2021.03.24 11:22:46.869023 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 42 41 37 
2021.03.24 11:22:46.869031 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 41
2021.03.24 11:22:46.869040 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.869048 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.869056 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 41, starting_idx: 1, cur_nxt_idx: 43
2021.03.24 11:22:46.869068 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=41, LastLogTerm=1, EntriesLength=1, CommitIndex=41, Term=1, peer_last_sent_idx 40
2021.03.24 11:22:46.869106 [ 1094 ] {} <Debug> RaftInstance: idx: 42
2021.03.24 11:22:46.869122 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.869178 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.869190 [ 1094 ] {} <Trace> RaftInstance: local log idx 42, target_commit_idx 41, quick_commit_index_ 41, state_->get_commit_idx() 39
2021.03.24 11:22:46.869198 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.869215 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.869262 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.869236 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.869310 [ 1096 ] {} <Debug> RaftInstance: commit upto 41, current idx 39
2021.03.24 11:22:46.869321 [ 1096 ] {} <Trace> RaftInstance: commit upto 41, current idx 40
2021.03.24 11:22:46.869341 [ 1096 ] {} <Debug> RaftInstance: notify cb 40 0x7faaa6f93a20
2021.03.24 11:22:46.869364 [ 1096 ] {} <Trace> RaftInstance: commit upto 41, current idx 41
2021.03.24 11:22:46.869383 [ 1096 ] {} <Debug> RaftInstance: notify cb 41 0x7faaa6f93920
2021.03.24 11:22:46.869413 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 41, current idx 41
2021.03.24 11:22:46.869422 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.869428 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:46.869449 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 41 wake up (7581 us), return value 0x7faaa9a69b50
2021.03.24 11:22:46.869545 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.869569 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 40 wake up (7769 us), return value 0x7faaa9a69b40
2021.03.24 11:22:46.875367 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.875409 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=40
2021.03.24 11:22:46.875419 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.875435 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 40
2021.03.24 11:22:46.875444 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.875452 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 37, new matched idx: 39
2021.03.24 11:22:46.875463 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 42 41 39 
2021.03.24 11:22:46.875472 [ 1094 ] {} <Trace> RaftInstance: local log idx 42, target_commit_idx 41, quick_commit_index_ 41, state_->get_commit_idx() 41
2021.03.24 11:22:46.875480 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.875488 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.875496 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 39, starting_idx: 1, cur_nxt_idx: 43
2021.03.24 11:22:46.875508 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=39, LastLogTerm=1, EntriesLength=3, CommitIndex=41, Term=1, peer_last_sent_idx 38
2021.03.24 11:22:46.875516 [ 1094 ] {} <Debug> RaftInstance: idx range: 40-42
2021.03.24 11:22:46.875525 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.875587 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.876001 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.876032 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=43
2021.03.24 11:22:46.876041 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.876058 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 43
2021.03.24 11:22:46.876066 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.876074 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 41, new matched idx: 42
2021.03.24 11:22:46.876085 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 42 42 39 
2021.03.24 11:22:46.876093 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 42
2021.03.24 11:22:46.876102 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.876133 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.876142 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 42, starting_idx: 1, cur_nxt_idx: 43
2021.03.24 11:22:46.876152 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=42, LastLogTerm=1, EntriesLength=0, CommitIndex=42, Term=1, peer_last_sent_idx 42
2021.03.24 11:22:46.876160 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.876168 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.876215 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.876228 [ 1092 ] {} <Trace> RaftInstance: local log idx 42, target_commit_idx 42, quick_commit_index_ 42, state_->get_commit_idx() 41
2021.03.24 11:22:46.876239 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.876274 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.876294 [ 1096 ] {} <Debug> RaftInstance: commit upto 42, current idx 41
2021.03.24 11:22:46.876304 [ 1096 ] {} <Trace> RaftInstance: commit upto 42, current idx 42
2021.03.24 11:22:46.876364 [ 1096 ] {} <Debug> RaftInstance: notify cb 42 0x7faa0daae120
2021.03.24 11:22:46.876380 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 42, current idx 42
2021.03.24 11:22:46.876388 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.876406 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.876415 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 42 wake up (13918 us), return value (nil)
2021.03.24 11:22:46.876445 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.876465 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 43
2021.03.24 11:22:46.876484 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 43 0x7faa0daae020 sleep
2021.03.24 11:22:46.876505 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.876524 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.877889 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.877914 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 44
2021.03.24 11:22:46.877934 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 44 0x7faaa6f93120 sleep
2021.03.24 11:22:46.877951 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.877968 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.882879 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.882906 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=43
2021.03.24 11:22:46.882916 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.882925 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 43
2021.03.24 11:22:46.882933 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.882941 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 42, new matched idx: 42
2021.03.24 11:22:46.882951 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 44 42 39 
2021.03.24 11:22:46.882959 [ 1093 ] {} <Trace> RaftInstance: local log idx 44, target_commit_idx 42, quick_commit_index_ 42, state_->get_commit_idx() 42
2021.03.24 11:22:46.882967 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.882975 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.882983 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 42, starting_idx: 1, cur_nxt_idx: 45
2021.03.24 11:22:46.883001 [ 1093 ] {} <Debug> RaftInstance: last sent log (43) to peer 1 is not applied, cnt 1
2021.03.24 11:22:46.883012 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=42, LastLogTerm=1, EntriesLength=2, CommitIndex=42, Term=1, peer_last_sent_idx 43
2021.03.24 11:22:46.883022 [ 1093 ] {} <Debug> RaftInstance: idx range: 43-44
2021.03.24 11:22:46.883031 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.883084 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.889945 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.889982 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=45
2021.03.24 11:22:46.889992 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.890002 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 45
2021.03.24 11:22:46.890010 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.890023 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 42, new matched idx: 44
2021.03.24 11:22:46.890035 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 44 44 39 
2021.03.24 11:22:46.890042 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 44
2021.03.24 11:22:46.890051 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.890058 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.890067 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 44, starting_idx: 1, cur_nxt_idx: 45
2021.03.24 11:22:46.890076 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=44, LastLogTerm=1, EntriesLength=0, CommitIndex=44, Term=1, peer_last_sent_idx 43
2021.03.24 11:22:46.890083 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.890091 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.890147 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.890160 [ 1093 ] {} <Trace> RaftInstance: local log idx 44, target_commit_idx 44, quick_commit_index_ 44, state_->get_commit_idx() 42
2021.03.24 11:22:46.890168 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.890223 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.890246 [ 1096 ] {} <Debug> RaftInstance: commit upto 44, current idx 42
2021.03.24 11:22:46.890256 [ 1096 ] {} <Trace> RaftInstance: commit upto 44, current idx 43
2021.03.24 11:22:46.890303 [ 1096 ] {} <Debug> RaftInstance: notify cb 43 0x7faa0daae020
2021.03.24 11:22:46.890319 [ 1096 ] {} <Trace> RaftInstance: commit upto 44, current idx 44
2021.03.24 11:22:46.890342 [ 1096 ] {} <Debug> RaftInstance: notify cb 44 0x7faaa6f93120
2021.03.24 11:22:46.890362 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 44, current idx 44
2021.03.24 11:22:46.890370 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.890376 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.890399 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.890410 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 43 wake up (13921 us), return value (nil)
2021.03.24 11:22:46.890432 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.890454 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 45
2021.03.24 11:22:46.890468 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 45 0x7faa0daae120 sleep
2021.03.24 11:22:46.890487 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.890515 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.890529 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 44 wake up (12436 us), return value 0x7faaa9a69b60
2021.03.24 11:22:46.896370 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.896424 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=43
2021.03.24 11:22:46.896435 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.896447 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 43
2021.03.24 11:22:46.896455 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.896463 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 39, new matched idx: 42
2021.03.24 11:22:46.896476 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 45 44 42 
2021.03.24 11:22:46.896485 [ 1094 ] {} <Trace> RaftInstance: local log idx 45, target_commit_idx 44, quick_commit_index_ 44, state_->get_commit_idx() 44
2021.03.24 11:22:46.896498 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.896507 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.896515 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 42, starting_idx: 1, cur_nxt_idx: 46
2021.03.24 11:22:46.896527 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=42, LastLogTerm=1, EntriesLength=3, CommitIndex=44, Term=1, peer_last_sent_idx 40
2021.03.24 11:22:46.896535 [ 1094 ] {} <Debug> RaftInstance: idx range: 43-45
2021.03.24 11:22:46.896544 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.896639 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.896719 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.896749 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=45
2021.03.24 11:22:46.896759 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.896769 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 45
2021.03.24 11:22:46.896777 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.896785 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 44, new matched idx: 44
2021.03.24 11:22:46.896796 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 45 44 42 
2021.03.24 11:22:46.896804 [ 1092 ] {} <Trace> RaftInstance: local log idx 45, target_commit_idx 44, quick_commit_index_ 44, state_->get_commit_idx() 44
2021.03.24 11:22:46.896818 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.896827 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.896835 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 44, starting_idx: 1, cur_nxt_idx: 46
2021.03.24 11:22:46.896846 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=44, LastLogTerm=1, EntriesLength=1, CommitIndex=44, Term=1, peer_last_sent_idx 45
2021.03.24 11:22:46.896853 [ 1092 ] {} <Debug> RaftInstance: idx: 45
2021.03.24 11:22:46.896862 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.896921 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.898838 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.898870 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 46
2021.03.24 11:22:46.898889 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 46 0x7faaa6f93a20 sleep
2021.03.24 11:22:46.898910 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.898920 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.903741 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.903821 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=46
2021.03.24 11:22:46.903849 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.903861 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 46
2021.03.24 11:22:46.903869 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.903877 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 44, new matched idx: 45
2021.03.24 11:22:46.903890 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 46 45 42 
2021.03.24 11:22:46.903898 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 45
2021.03.24 11:22:46.903907 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.903915 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.903923 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 45, starting_idx: 1, cur_nxt_idx: 47
2021.03.24 11:22:46.903936 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=45, LastLogTerm=1, EntriesLength=1, CommitIndex=45, Term=1, peer_last_sent_idx 45
2021.03.24 11:22:46.903943 [ 1094 ] {} <Debug> RaftInstance: idx: 46
2021.03.24 11:22:46.903952 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.904037 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.904051 [ 1094 ] {} <Trace> RaftInstance: local log idx 46, target_commit_idx 45, quick_commit_index_ 45, state_->get_commit_idx() 44
2021.03.24 11:22:46.904059 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.904079 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.904088 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.904183 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.904226 [ 1096 ] {} <Debug> RaftInstance: commit upto 45, current idx 44
2021.03.24 11:22:46.904238 [ 1096 ] {} <Trace> RaftInstance: commit upto 45, current idx 45
2021.03.24 11:22:46.904277 [ 1096 ] {} <Debug> RaftInstance: notify cb 45 0x7faa0daae120
2021.03.24 11:22:46.904295 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 45, current idx 45
2021.03.24 11:22:46.904304 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.904350 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.904363 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 45 wake up (13881 us), return value (nil)
2021.03.24 11:22:46.905470 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.905507 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 47
2021.03.24 11:22:46.905530 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 47 0x7faa0daae020 sleep
2021.03.24 11:22:46.905553 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.905574 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.905787 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.905856 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 48
2021.03.24 11:22:46.905880 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 48 0x7faaa6f93520 sleep
2021.03.24 11:22:46.905898 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.905914 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.910881 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.910934 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=47
2021.03.24 11:22:46.910970 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.910983 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 47
2021.03.24 11:22:46.910991 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.910999 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 45, new matched idx: 46
2021.03.24 11:22:46.911011 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 48 46 42 
2021.03.24 11:22:46.911019 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 46
2021.03.24 11:22:46.911028 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.911036 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.911044 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 46, starting_idx: 1, cur_nxt_idx: 49
2021.03.24 11:22:46.911055 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=46, LastLogTerm=1, EntriesLength=2, CommitIndex=46, Term=1, peer_last_sent_idx 46
2021.03.24 11:22:46.911063 [ 1092 ] {} <Debug> RaftInstance: idx range: 47-48
2021.03.24 11:22:46.911072 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.911130 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.911143 [ 1092 ] {} <Trace> RaftInstance: local log idx 48, target_commit_idx 46, quick_commit_index_ 46, state_->get_commit_idx() 45
2021.03.24 11:22:46.911151 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.911169 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.911178 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.911205 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.911227 [ 1096 ] {} <Debug> RaftInstance: commit upto 46, current idx 45
2021.03.24 11:22:46.911237 [ 1096 ] {} <Trace> RaftInstance: commit upto 46, current idx 46
2021.03.24 11:22:46.911272 [ 1096 ] {} <Debug> RaftInstance: notify cb 46 0x7faaa6f93a20
2021.03.24 11:22:46.911290 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 46, current idx 46
2021.03.24 11:22:46.911299 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.911323 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:46.911349 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 46 wake up (12432 us), return value (nil)
2021.03.24 11:22:46.917380 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.917451 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=46
2021.03.24 11:22:46.917464 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.917475 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 46
2021.03.24 11:22:46.917488 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.917496 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 42, new matched idx: 45
2021.03.24 11:22:46.917508 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 48 46 45 
2021.03.24 11:22:46.917516 [ 1092 ] {} <Trace> RaftInstance: local log idx 48, target_commit_idx 46, quick_commit_index_ 46, state_->get_commit_idx() 46
2021.03.24 11:22:46.917524 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.917533 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.917541 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 45, starting_idx: 1, cur_nxt_idx: 49
2021.03.24 11:22:46.917553 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=45, LastLogTerm=1, EntriesLength=3, CommitIndex=46, Term=1, peer_last_sent_idx 43
2021.03.24 11:22:46.917561 [ 1092 ] {} <Debug> RaftInstance: idx range: 46-48
2021.03.24 11:22:46.917570 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.917662 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.918027 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.918053 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=49
2021.03.24 11:22:46.918064 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.918081 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 49
2021.03.24 11:22:46.918089 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.918097 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 46, new matched idx: 48
2021.03.24 11:22:46.918109 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 48 48 45 
2021.03.24 11:22:46.918117 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 48
2021.03.24 11:22:46.918126 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.918159 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.918171 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 48, starting_idx: 1, cur_nxt_idx: 49
2021.03.24 11:22:46.918182 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=48, LastLogTerm=1, EntriesLength=0, CommitIndex=48, Term=1, peer_last_sent_idx 47
2021.03.24 11:22:46.918190 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.918198 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.918245 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.918258 [ 1093 ] {} <Trace> RaftInstance: local log idx 48, target_commit_idx 48, quick_commit_index_ 48, state_->get_commit_idx() 46
2021.03.24 11:22:46.918269 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.918323 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.918345 [ 1096 ] {} <Debug> RaftInstance: commit upto 48, current idx 46
2021.03.24 11:22:46.918355 [ 1096 ] {} <Trace> RaftInstance: commit upto 48, current idx 47
2021.03.24 11:22:46.918419 [ 1096 ] {} <Debug> RaftInstance: notify cb 47 0x7faa0daae020
2021.03.24 11:22:46.918453 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.918465 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 47 wake up (12931 us), return value (nil)
2021.03.24 11:22:46.918487 [ 1096 ] {} <Trace> RaftInstance: commit upto 48, current idx 48
2021.03.24 11:22:46.918509 [ 1096 ] {} <Debug> RaftInstance: notify cb 48 0x7faaa6f93520
2021.03.24 11:22:46.918527 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 48, current idx 48
2021.03.24 11:22:46.918535 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.918576 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.918598 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 48 wake up (12699 us), return value (nil)
2021.03.24 11:22:46.919858 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.919894 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 49
2021.03.24 11:22:46.919915 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 49 0x7faa1f09b020 sleep
2021.03.24 11:22:46.919938 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.919960 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.924985 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.925007 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=49
2021.03.24 11:22:46.925017 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.925044 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 49
2021.03.24 11:22:46.925053 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.925061 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 48, new matched idx: 48
2021.03.24 11:22:46.925071 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 49 48 45 
2021.03.24 11:22:46.925102 [ 1093 ] {} <Trace> RaftInstance: local log idx 49, target_commit_idx 48, quick_commit_index_ 48, state_->get_commit_idx() 48
2021.03.24 11:22:46.925116 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.925124 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.925132 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 48, starting_idx: 1, cur_nxt_idx: 50
2021.03.24 11:22:46.925144 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=48, LastLogTerm=1, EntriesLength=1, CommitIndex=48, Term=1, peer_last_sent_idx 49
2021.03.24 11:22:46.925151 [ 1093 ] {} <Debug> RaftInstance: idx: 49
2021.03.24 11:22:46.925161 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.925208 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.932082 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.932140 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=50
2021.03.24 11:22:46.932152 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.932163 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 50
2021.03.24 11:22:46.932172 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.932180 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 48, new matched idx: 49
2021.03.24 11:22:46.932193 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 49 49 45 
2021.03.24 11:22:46.932201 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 49
2021.03.24 11:22:46.932211 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.932225 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.932234 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 49, starting_idx: 1, cur_nxt_idx: 50
2021.03.24 11:22:46.932245 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=49, LastLogTerm=1, EntriesLength=0, CommitIndex=49, Term=1, peer_last_sent_idx 49
2021.03.24 11:22:46.932253 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.932261 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.932336 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.932350 [ 1094 ] {} <Trace> RaftInstance: local log idx 49, target_commit_idx 49, quick_commit_index_ 49, state_->get_commit_idx() 48
2021.03.24 11:22:46.932358 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.932400 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.932423 [ 1096 ] {} <Debug> RaftInstance: commit upto 49, current idx 48
2021.03.24 11:22:46.932434 [ 1096 ] {} <Trace> RaftInstance: commit upto 49, current idx 49
2021.03.24 11:22:46.932472 [ 1096 ] {} <Debug> RaftInstance: notify cb 49 0x7faa1f09b020
2021.03.24 11:22:46.932491 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 49, current idx 49
2021.03.24 11:22:46.932501 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.932533 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.932562 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 49 wake up (12614 us), return value (nil)
2021.03.24 11:22:46.938228 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.938281 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=49
2021.03.24 11:22:46.938334 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.938349 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 49
2021.03.24 11:22:46.938357 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.938366 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 45, new matched idx: 48
2021.03.24 11:22:46.938367 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.938377 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 49 49 48 
2021.03.24 11:22:46.938427 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 50
2021.03.24 11:22:46.938450 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 50 0x7faaa6f93920 sleep
2021.03.24 11:22:46.938431 [ 1093 ] {} <Trace> RaftInstance: local log idx 50, target_commit_idx 49, quick_commit_index_ 49, state_->get_commit_idx() 49
2021.03.24 11:22:46.938477 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.938487 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.938496 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 48, starting_idx: 1, cur_nxt_idx: 51
2021.03.24 11:22:46.938507 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=48, LastLogTerm=1, EntriesLength=2, CommitIndex=49, Term=1, peer_last_sent_idx 46
2021.03.24 11:22:46.938515 [ 1093 ] {} <Debug> RaftInstance: idx range: 49-50
2021.03.24 11:22:46.938524 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.938602 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.938625 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.938635 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.938872 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.938901 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=50
2021.03.24 11:22:46.938916 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.938927 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 50
2021.03.24 11:22:46.938935 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.938943 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 49, new matched idx: 49
2021.03.24 11:22:46.938955 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 50 49 48 
2021.03.24 11:22:46.938964 [ 1092 ] {} <Trace> RaftInstance: local log idx 50, target_commit_idx 49, quick_commit_index_ 49, state_->get_commit_idx() 49
2021.03.24 11:22:46.938972 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.938980 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.938988 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 49, starting_idx: 1, cur_nxt_idx: 51
2021.03.24 11:22:46.939000 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=49, LastLogTerm=1, EntriesLength=1, CommitIndex=49, Term=1, peer_last_sent_idx 50
2021.03.24 11:22:46.939007 [ 1092 ] {} <Debug> RaftInstance: idx: 50
2021.03.24 11:22:46.939016 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.939066 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.940969 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.941006 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 51
2021.03.24 11:22:46.941034 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.941044 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.941056 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 51 0x7faaa6f93120 sleep
2021.03.24 11:22:46.945680 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.945713 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=51
2021.03.24 11:22:46.945723 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.945733 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 51
2021.03.24 11:22:46.945741 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.945760 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 49, new matched idx: 50
2021.03.24 11:22:46.945772 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 51 50 48 
2021.03.24 11:22:46.945780 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 50
2021.03.24 11:22:46.945789 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.945798 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.945805 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 50, starting_idx: 1, cur_nxt_idx: 52
2021.03.24 11:22:46.945817 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=50, LastLogTerm=1, EntriesLength=1, CommitIndex=50, Term=1, peer_last_sent_idx 50
2021.03.24 11:22:46.945825 [ 1092 ] {} <Debug> RaftInstance: idx: 51
2021.03.24 11:22:46.945833 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.945879 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.945891 [ 1092 ] {} <Trace> RaftInstance: local log idx 51, target_commit_idx 50, quick_commit_index_ 50, state_->get_commit_idx() 49
2021.03.24 11:22:46.945898 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.945920 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.945930 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.945965 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.946012 [ 1096 ] {} <Debug> RaftInstance: commit upto 50, current idx 49
2021.03.24 11:22:46.946024 [ 1096 ] {} <Trace> RaftInstance: commit upto 50, current idx 50
2021.03.24 11:22:46.946065 [ 1096 ] {} <Debug> RaftInstance: notify cb 50 0x7faaa6f93920
2021.03.24 11:22:46.946107 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 50, current idx 50
2021.03.24 11:22:46.946116 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.946167 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.946188 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 50 wake up (7716 us), return value (nil)
2021.03.24 11:22:46.952786 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.952813 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=52
2021.03.24 11:22:46.952839 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.952850 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 52
2021.03.24 11:22:46.952858 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.952866 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 50, new matched idx: 51
2021.03.24 11:22:46.952876 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 51 51 48 
2021.03.24 11:22:46.952884 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 51
2021.03.24 11:22:46.952892 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.952900 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.952908 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 51, starting_idx: 1, cur_nxt_idx: 52
2021.03.24 11:22:46.952930 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=51, LastLogTerm=1, EntriesLength=0, CommitIndex=51, Term=1, peer_last_sent_idx 51
2021.03.24 11:22:46.952963 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.952973 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.953053 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.953068 [ 1092 ] {} <Trace> RaftInstance: local log idx 51, target_commit_idx 51, quick_commit_index_ 51, state_->get_commit_idx() 50
2021.03.24 11:22:46.953109 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.953184 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.953231 [ 1096 ] {} <Debug> RaftInstance: commit upto 51, current idx 50
2021.03.24 11:22:46.953243 [ 1096 ] {} <Trace> RaftInstance: commit upto 51, current idx 51
2021.03.24 11:22:46.953297 [ 1096 ] {} <Debug> RaftInstance: notify cb 51 0x7faaa6f93120
2021.03.24 11:22:46.953314 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 51, current idx 51
2021.03.24 11:22:46.953323 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.953359 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.953381 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 51 wake up (12332 us), return value (nil)
2021.03.24 11:22:46.959239 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.959296 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=51
2021.03.24 11:22:46.959307 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.959318 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 51
2021.03.24 11:22:46.959326 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.959334 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 48, new matched idx: 50
2021.03.24 11:22:46.959345 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 51 51 50 
2021.03.24 11:22:46.959360 [ 1094 ] {} <Trace> RaftInstance: local log idx 51, target_commit_idx 51, quick_commit_index_ 51, state_->get_commit_idx() 51
2021.03.24 11:22:46.959369 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.959377 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.959385 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 50, starting_idx: 1, cur_nxt_idx: 52
2021.03.24 11:22:46.959397 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=50, LastLogTerm=1, EntriesLength=1, CommitIndex=51, Term=1, peer_last_sent_idx 49
2021.03.24 11:22:46.959404 [ 1094 ] {} <Debug> RaftInstance: idx: 51
2021.03.24 11:22:46.959412 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.959504 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.959709 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.959743 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=52
2021.03.24 11:22:46.959754 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.959764 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 52
2021.03.24 11:22:46.959773 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.959781 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 51, new matched idx: 51
2021.03.24 11:22:46.959792 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 51 51 50 
2021.03.24 11:22:46.959805 [ 1093 ] {} <Trace> RaftInstance: local log idx 51, target_commit_idx 51, quick_commit_index_ 51, state_->get_commit_idx() 51
2021.03.24 11:22:46.962167 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.962215 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 52
2021.03.24 11:22:46.962235 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 52 0x7faa1f09b020 sleep
2021.03.24 11:22:46.962265 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.962293 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.962303 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 51, starting_idx: 1, cur_nxt_idx: 53
2021.03.24 11:22:46.962315 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=51, LastLogTerm=1, EntriesLength=1, CommitIndex=51, Term=1, peer_last_sent_idx 52
2021.03.24 11:22:46.962324 [ 1097 ] {} <Debug> RaftInstance: idx: 52
2021.03.24 11:22:46.962334 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.962405 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.966271 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.966319 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 53
2021.03.24 11:22:46.966340 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 53 0x7faaa6f93a20 sleep
2021.03.24 11:22:46.966357 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.966377 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.967759 [ 1168 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.967776 [ 1168 ] {} <Debug> RaftInstance: append at log_idx 54
2021.03.24 11:22:46.967814 [ 1168 ] {} <Debug> RaftInstance: commit_ret_cv 54 0x7faa1adc8020 sleep
2021.03.24 11:22:46.967855 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.967875 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.969130 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.969156 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=53
2021.03.24 11:22:46.969166 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.969182 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 53
2021.03.24 11:22:46.969190 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.969198 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 51, new matched idx: 52
2021.03.24 11:22:46.969209 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 54 52 50 
2021.03.24 11:22:46.969217 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 52
2021.03.24 11:22:46.969226 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.969234 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.969242 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 52, starting_idx: 1, cur_nxt_idx: 55
2021.03.24 11:22:46.969254 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=52, LastLogTerm=1, EntriesLength=2, CommitIndex=52, Term=1, peer_last_sent_idx 52
2021.03.24 11:22:46.969262 [ 1095 ] {} <Debug> RaftInstance: idx range: 53-54
2021.03.24 11:22:46.969271 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.969322 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.969351 [ 1095 ] {} <Trace> RaftInstance: local log idx 54, target_commit_idx 52, quick_commit_index_ 52, state_->get_commit_idx() 51
2021.03.24 11:22:46.969360 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.969377 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.969387 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.969396 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.969419 [ 1096 ] {} <Debug> RaftInstance: commit upto 52, current idx 51
2021.03.24 11:22:46.969429 [ 1096 ] {} <Trace> RaftInstance: commit upto 52, current idx 52
2021.03.24 11:22:46.969493 [ 1096 ] {} <Debug> RaftInstance: notify cb 52 0x7faa1f09b020
2021.03.24 11:22:46.969512 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 52, current idx 52
2021.03.24 11:22:46.969521 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.969566 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:46.969589 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 52 wake up (7331 us), return value (nil)
2021.03.24 11:22:46.976213 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.976256 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=55
2021.03.24 11:22:46.976267 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.976279 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 55
2021.03.24 11:22:46.976287 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.976295 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 52, new matched idx: 54
2021.03.24 11:22:46.976307 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 54 54 50 
2021.03.24 11:22:46.976315 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 54
2021.03.24 11:22:46.976325 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.976334 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.976342 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 54, starting_idx: 1, cur_nxt_idx: 55
2021.03.24 11:22:46.976353 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=54, LastLogTerm=1, EntriesLength=0, CommitIndex=54, Term=1, peer_last_sent_idx 53
2021.03.24 11:22:46.976361 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.976369 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.976430 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.976443 [ 1095 ] {} <Trace> RaftInstance: local log idx 54, target_commit_idx 54, quick_commit_index_ 54, state_->get_commit_idx() 52
2021.03.24 11:22:46.976451 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.976517 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.976592 [ 1096 ] {} <Debug> RaftInstance: commit upto 54, current idx 52
2021.03.24 11:22:46.976611 [ 1096 ] {} <Trace> RaftInstance: commit upto 54, current idx 53
2021.03.24 11:22:46.976639 [ 1096 ] {} <Debug> RaftInstance: notify cb 53 0x7faaa6f93a20
2021.03.24 11:22:46.976665 [ 1096 ] {} <Trace> RaftInstance: commit upto 54, current idx 54
2021.03.24 11:22:46.976691 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.976723 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 53 wake up (10350 us), return value 0x7faaa9a69b70
2021.03.24 11:22:46.976736 [ 1096 ] {} <Debug> RaftInstance: notify cb 54 0x7faa1adc8020
2021.03.24 11:22:46.976785 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 54, current idx 54
2021.03.24 11:22:46.976794 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.976832 [ 1168 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.976847 [ 1168 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 54 wake up (9034 us), return value 0x7faaa9a69b80
2021.03.24 11:22:46.976933 [ 1169 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.976982 [ 1169 ] {} <Debug> RaftInstance: append at log_idx 55
2021.03.24 11:22:46.977043 [ 1169 ] {} <Debug> RaftInstance: commit_ret_cv 55 0x7faa1bb69020 sleep
2021.03.24 11:22:46.977063 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.977115 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.977909 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.977942 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 56
2021.03.24 11:22:46.977964 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 56 0x7faa1f09b020 sleep
2021.03.24 11:22:46.978003 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.978027 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.980136 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:46.980156 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=52
2021.03.24 11:22:46.980166 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.980176 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 52
2021.03.24 11:22:46.980184 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:46.980192 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 50, new matched idx: 51
2021.03.24 11:22:46.980201 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 56 54 51 
2021.03.24 11:22:46.980209 [ 1092 ] {} <Trace> RaftInstance: local log idx 56, target_commit_idx 54, quick_commit_index_ 54, state_->get_commit_idx() 54
2021.03.24 11:22:46.980217 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:46.980225 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:46.980233 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 51, starting_idx: 1, cur_nxt_idx: 57
2021.03.24 11:22:46.980245 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=51, LastLogTerm=1, EntriesLength=5, CommitIndex=54, Term=1, peer_last_sent_idx 51
2021.03.24 11:22:46.980253 [ 1092 ] {} <Debug> RaftInstance: idx range: 52-56
2021.03.24 11:22:46.980262 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:46.980306 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.982943 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.982993 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=55
2021.03.24 11:22:46.983004 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.983016 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 55
2021.03.24 11:22:46.983024 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.983032 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 54, new matched idx: 54
2021.03.24 11:22:46.983045 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 56 54 51 
2021.03.24 11:22:46.983054 [ 1094 ] {} <Trace> RaftInstance: local log idx 56, target_commit_idx 54, quick_commit_index_ 54, state_->get_commit_idx() 54
2021.03.24 11:22:46.983086 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.983095 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.983104 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 54, starting_idx: 1, cur_nxt_idx: 57
2021.03.24 11:22:46.983112 [ 1094 ] {} <Debug> RaftInstance: last sent log (55) to peer 1 is not applied, cnt 1
2021.03.24 11:22:46.983124 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=54, LastLogTerm=1, EntriesLength=2, CommitIndex=54, Term=1, peer_last_sent_idx 55
2021.03.24 11:22:46.983132 [ 1094 ] {} <Debug> RaftInstance: idx range: 55-56
2021.03.24 11:22:46.983141 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.983216 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.989926 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.989966 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=57
2021.03.24 11:22:46.989977 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.989988 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 57
2021.03.24 11:22:46.989996 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.990003 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 54, new matched idx: 56
2021.03.24 11:22:46.990015 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 56 56 51 
2021.03.24 11:22:46.990022 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 56
2021.03.24 11:22:46.990031 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.990039 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.990047 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 56, starting_idx: 1, cur_nxt_idx: 57
2021.03.24 11:22:46.990058 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=56, LastLogTerm=1, EntriesLength=0, CommitIndex=56, Term=1, peer_last_sent_idx 55
2021.03.24 11:22:46.990065 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:46.990073 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.990134 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.990147 [ 1092 ] {} <Trace> RaftInstance: local log idx 56, target_commit_idx 56, quick_commit_index_ 56, state_->get_commit_idx() 54
2021.03.24 11:22:46.990155 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:46.990203 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:46.990243 [ 1096 ] {} <Debug> RaftInstance: commit upto 56, current idx 54
2021.03.24 11:22:46.990255 [ 1096 ] {} <Trace> RaftInstance: commit upto 56, current idx 55
2021.03.24 11:22:46.990282 [ 1096 ] {} <Debug> RaftInstance: notify cb 55 0x7faa1bb69020
2021.03.24 11:22:46.990306 [ 1096 ] {} <Trace> RaftInstance: commit upto 56, current idx 56
2021.03.24 11:22:46.990325 [ 1096 ] {} <Debug> RaftInstance: notify cb 56 0x7faa1f09b020
2021.03.24 11:22:46.990343 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 56, current idx 56
2021.03.24 11:22:46.990351 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:46.990364 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:46.990385 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 56 wake up (12403 us), return value 0x7faaa9a69ba0
2021.03.24 11:22:46.990451 [ 1169 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:46.990467 [ 1169 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 55 wake up (13417 us), return value 0x7faaa9a69b90
2021.03.24 11:22:46.990540 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.990574 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 57
2021.03.24 11:22:46.990597 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 57 0x7faa0daae120 sleep
2021.03.24 11:22:46.990622 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.990641 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:46.996705 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:46.996713 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:46.996764 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 58
2021.03.24 11:22:46.996789 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=57
2021.03.24 11:22:46.996869 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:46.996807 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 58 0x7faa1f070a20 sleep
2021.03.24 11:22:46.996889 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 57
2021.03.24 11:22:46.996936 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:46.996946 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 56, new matched idx: 56
2021.03.24 11:22:46.996958 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 58 56 51 
2021.03.24 11:22:46.996966 [ 1095 ] {} <Trace> RaftInstance: local log idx 58, target_commit_idx 56, quick_commit_index_ 56, state_->get_commit_idx() 56
2021.03.24 11:22:46.996975 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:46.996982 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:46.996991 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 56, starting_idx: 1, cur_nxt_idx: 59
2021.03.24 11:22:46.996998 [ 1095 ] {} <Debug> RaftInstance: last sent log (57) to peer 1 is not applied, cnt 1
2021.03.24 11:22:46.997010 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=56, LastLogTerm=1, EntriesLength=2, CommitIndex=56, Term=1, peer_last_sent_idx 57
2021.03.24 11:22:46.997018 [ 1095 ] {} <Debug> RaftInstance: idx range: 57-58
2021.03.24 11:22:46.997027 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:46.997109 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:46.997131 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:46.997140 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.000934 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.000976 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=57
2021.03.24 11:22:47.000988 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.000999 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 57
2021.03.24 11:22:47.001007 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.001015 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 51, new matched idx: 56
2021.03.24 11:22:47.001027 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 58 56 56 
2021.03.24 11:22:47.001036 [ 1093 ] {} <Trace> RaftInstance: local log idx 58, target_commit_idx 56, quick_commit_index_ 56, state_->get_commit_idx() 56
2021.03.24 11:22:47.001049 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.001058 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.001114 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 56, starting_idx: 1, cur_nxt_idx: 59
2021.03.24 11:22:47.001130 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=56, LastLogTerm=1, EntriesLength=2, CommitIndex=56, Term=1, peer_last_sent_idx 52
2021.03.24 11:22:47.001138 [ 1093 ] {} <Debug> RaftInstance: idx range: 57-58
2021.03.24 11:22:47.001147 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.001216 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.003877 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.003907 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=59
2021.03.24 11:22:47.003917 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.003927 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 59
2021.03.24 11:22:47.003934 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.003948 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 56, new matched idx: 58
2021.03.24 11:22:47.003958 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 58 58 56 
2021.03.24 11:22:47.003972 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 58
2021.03.24 11:22:47.003983 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.003991 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.003999 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 58, starting_idx: 1, cur_nxt_idx: 59
2021.03.24 11:22:47.004008 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=58, LastLogTerm=1, EntriesLength=0, CommitIndex=58, Term=1, peer_last_sent_idx 57
2021.03.24 11:22:47.004016 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.004023 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.004064 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.004076 [ 1095 ] {} <Trace> RaftInstance: local log idx 58, target_commit_idx 58, quick_commit_index_ 58, state_->get_commit_idx() 56
2021.03.24 11:22:47.004084 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.004140 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.004166 [ 1096 ] {} <Debug> RaftInstance: commit upto 58, current idx 56
2021.03.24 11:22:47.004176 [ 1096 ] {} <Trace> RaftInstance: commit upto 58, current idx 57
2021.03.24 11:22:47.004216 [ 1096 ] {} <Debug> RaftInstance: notify cb 57 0x7faa0daae120
2021.03.24 11:22:47.004251 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.004264 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 57 wake up (13663 us), return value (nil)
2021.03.24 11:22:47.004287 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.004309 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 59
2021.03.24 11:22:47.004329 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 59 0x7faa0daae020 sleep
2021.03.24 11:22:47.004347 [ 1096 ] {} <Trace> RaftInstance: commit upto 58, current idx 58
2021.03.24 11:22:47.004348 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.004376 [ 1096 ] {} <Debug> RaftInstance: notify cb 58 0x7faa1f070a20
2021.03.24 11:22:47.004410 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.004423 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 58, current idx 58
2021.03.24 11:22:47.004434 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.004443 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.004462 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 58 wake up (7639 us), return value (nil)
2021.03.24 11:22:47.007264 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.007300 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 60
2021.03.24 11:22:47.007325 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 60 0x7faaa6f93520 sleep
2021.03.24 11:22:47.007348 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.007368 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.010664 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.010686 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=59
2021.03.24 11:22:47.010696 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.010705 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 59
2021.03.24 11:22:47.010713 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.010721 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 58, new matched idx: 58
2021.03.24 11:22:47.010731 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 60 58 56 
2021.03.24 11:22:47.010739 [ 1092 ] {} <Trace> RaftInstance: local log idx 60, target_commit_idx 58, quick_commit_index_ 58, state_->get_commit_idx() 58
2021.03.24 11:22:47.010747 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.010755 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.010763 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 58, starting_idx: 1, cur_nxt_idx: 61
2021.03.24 11:22:47.010771 [ 1092 ] {} <Debug> RaftInstance: last sent log (59) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.010801 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=58, LastLogTerm=1, EntriesLength=2, CommitIndex=58, Term=1, peer_last_sent_idx 59
2021.03.24 11:22:47.010809 [ 1092 ] {} <Debug> RaftInstance: idx range: 59-60
2021.03.24 11:22:47.010819 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.010866 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.017655 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.017678 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=61
2021.03.24 11:22:47.017687 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.017697 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 61
2021.03.24 11:22:47.017705 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.017713 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 58, new matched idx: 60
2021.03.24 11:22:47.017723 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 60 60 56 
2021.03.24 11:22:47.017730 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 60
2021.03.24 11:22:47.017739 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.017746 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.017754 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 60, starting_idx: 1, cur_nxt_idx: 61
2021.03.24 11:22:47.017764 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=60, LastLogTerm=1, EntriesLength=0, CommitIndex=60, Term=1, peer_last_sent_idx 59
2021.03.24 11:22:47.017771 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.017779 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.017821 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.017835 [ 1092 ] {} <Trace> RaftInstance: local log idx 60, target_commit_idx 60, quick_commit_index_ 60, state_->get_commit_idx() 58
2021.03.24 11:22:47.017843 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.017930 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.017980 [ 1096 ] {} <Debug> RaftInstance: commit upto 60, current idx 58
2021.03.24 11:22:47.017992 [ 1096 ] {} <Trace> RaftInstance: commit upto 60, current idx 59
2021.03.24 11:22:47.018032 [ 1096 ] {} <Debug> RaftInstance: notify cb 59 0x7faa0daae020
2021.03.24 11:22:47.018068 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.018082 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 59 wake up (13745 us), return value (nil)
2021.03.24 11:22:47.018108 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.018153 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 61
2021.03.24 11:22:47.018174 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 61 0x7faa0daae120 sleep
2021.03.24 11:22:47.018192 [ 1096 ] {} <Trace> RaftInstance: commit upto 60, current idx 60
2021.03.24 11:22:47.018200 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.018213 [ 1096 ] {} <Debug> RaftInstance: notify cb 60 0x7faaa6f93520
2021.03.24 11:22:47.018232 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.018252 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 60, current idx 60
2021.03.24 11:22:47.018276 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.018307 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 60 wake up (10958 us), return value (nil)
2021.03.24 11:22:47.018280 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.021972 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.022015 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=59
2021.03.24 11:22:47.022026 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.022043 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 59
2021.03.24 11:22:47.022051 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.022060 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 56, new matched idx: 58
2021.03.24 11:22:47.022071 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 61 60 58 
2021.03.24 11:22:47.022080 [ 1094 ] {} <Trace> RaftInstance: local log idx 61, target_commit_idx 60, quick_commit_index_ 60, state_->get_commit_idx() 60
2021.03.24 11:22:47.022088 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.022097 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.022106 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 58, starting_idx: 1, cur_nxt_idx: 62
2021.03.24 11:22:47.022117 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=58, LastLogTerm=1, EntriesLength=3, CommitIndex=60, Term=1, peer_last_sent_idx 57
2021.03.24 11:22:47.022125 [ 1094 ] {} <Debug> RaftInstance: idx range: 59-61
2021.03.24 11:22:47.022135 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.022229 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.024334 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.024375 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 62
2021.03.24 11:22:47.024403 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.024413 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.024425 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 62 0x7faaa6f93120 sleep
2021.03.24 11:22:47.024486 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.024501 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=61
2021.03.24 11:22:47.024522 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.024531 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 61
2021.03.24 11:22:47.024539 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.024547 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 60, new matched idx: 60
2021.03.24 11:22:47.024557 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 62 60 58 
2021.03.24 11:22:47.024565 [ 1092 ] {} <Trace> RaftInstance: local log idx 62, target_commit_idx 60, quick_commit_index_ 60, state_->get_commit_idx() 60
2021.03.24 11:22:47.024573 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.024580 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.024589 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 60, starting_idx: 1, cur_nxt_idx: 63
2021.03.24 11:22:47.024597 [ 1092 ] {} <Debug> RaftInstance: last sent log (61) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.024608 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=60, LastLogTerm=1, EntriesLength=2, CommitIndex=60, Term=1, peer_last_sent_idx 61
2021.03.24 11:22:47.024615 [ 1092 ] {} <Debug> RaftInstance: idx range: 61-62
2021.03.24 11:22:47.024624 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.024674 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.031257 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.031295 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=63
2021.03.24 11:22:47.031306 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.031317 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 63
2021.03.24 11:22:47.031324 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.031333 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 60, new matched idx: 62
2021.03.24 11:22:47.031350 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 62 62 58 
2021.03.24 11:22:47.031358 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 62
2021.03.24 11:22:47.031368 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.031376 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.031384 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 62, starting_idx: 1, cur_nxt_idx: 63
2021.03.24 11:22:47.031396 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=62, LastLogTerm=1, EntriesLength=0, CommitIndex=62, Term=1, peer_last_sent_idx 61
2021.03.24 11:22:47.031403 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.031411 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.031457 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.031470 [ 1093 ] {} <Trace> RaftInstance: local log idx 62, target_commit_idx 62, quick_commit_index_ 62, state_->get_commit_idx() 60
2021.03.24 11:22:47.031478 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.031516 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.031538 [ 1096 ] {} <Debug> RaftInstance: commit upto 62, current idx 60
2021.03.24 11:22:47.031549 [ 1096 ] {} <Trace> RaftInstance: commit upto 62, current idx 61
2021.03.24 11:22:47.031616 [ 1096 ] {} <Debug> RaftInstance: notify cb 61 0x7faa0daae120
2021.03.24 11:22:47.031650 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.031662 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 61 wake up (13482 us), return value (nil)
2021.03.24 11:22:47.031703 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.031725 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 63
2021.03.24 11:22:47.031745 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 63 0x7faa0daae020 sleep
2021.03.24 11:22:47.031763 [ 1096 ] {} <Trace> RaftInstance: commit upto 62, current idx 62
2021.03.24 11:22:47.031776 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.031826 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.031798 [ 1096 ] {} <Debug> RaftInstance: notify cb 62 0x7faaa6f93120
2021.03.24 11:22:47.031870 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 62, current idx 62
2021.03.24 11:22:47.031880 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.031886 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.031930 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 62 wake up (7493 us), return value 0x7faaa9a69a60
2021.03.24 11:22:47.033702 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.033732 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 64
2021.03.24 11:22:47.033760 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.033771 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.033783 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 64 0x7faaa6f93920 sleep
2021.03.24 11:22:47.037929 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.037966 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=63
2021.03.24 11:22:47.037975 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.037985 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 63
2021.03.24 11:22:47.037993 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.038001 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 62, new matched idx: 62
2021.03.24 11:22:47.038012 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 64 62 58 
2021.03.24 11:22:47.038021 [ 1093 ] {} <Trace> RaftInstance: local log idx 64, target_commit_idx 62, quick_commit_index_ 62, state_->get_commit_idx() 62
2021.03.24 11:22:47.038029 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.038037 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.038045 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 62, starting_idx: 1, cur_nxt_idx: 65
2021.03.24 11:22:47.038053 [ 1093 ] {} <Debug> RaftInstance: last sent log (63) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.038065 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=62, LastLogTerm=1, EntriesLength=2, CommitIndex=62, Term=1, peer_last_sent_idx 63
2021.03.24 11:22:47.038072 [ 1093 ] {} <Debug> RaftInstance: idx range: 63-64
2021.03.24 11:22:47.038081 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.038146 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.042610 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.042654 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=62
2021.03.24 11:22:47.042665 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.042676 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 62
2021.03.24 11:22:47.042708 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.042717 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 58, new matched idx: 61
2021.03.24 11:22:47.042728 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 64 62 61 
2021.03.24 11:22:47.042737 [ 1094 ] {} <Trace> RaftInstance: local log idx 64, target_commit_idx 62, quick_commit_index_ 62, state_->get_commit_idx() 62
2021.03.24 11:22:47.042746 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.042754 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.042762 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 61, starting_idx: 1, cur_nxt_idx: 65
2021.03.24 11:22:47.042788 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=61, LastLogTerm=1, EntriesLength=3, CommitIndex=62, Term=1, peer_last_sent_idx 59
2021.03.24 11:22:47.042799 [ 1094 ] {} <Debug> RaftInstance: idx range: 62-64
2021.03.24 11:22:47.042808 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.042873 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.043650 [ 1093 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:47.043695 [ 1093 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:47.043705 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.044610 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.044629 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=65
2021.03.24 11:22:47.044637 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.044646 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 65
2021.03.24 11:22:47.044654 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.044661 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 62, new matched idx: 64
2021.03.24 11:22:47.044670 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 64 64 61 
2021.03.24 11:22:47.044678 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 64
2021.03.24 11:22:47.044686 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.044693 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.044701 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 64, starting_idx: 1, cur_nxt_idx: 65
2021.03.24 11:22:47.044711 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=64, LastLogTerm=1, EntriesLength=0, CommitIndex=64, Term=1, peer_last_sent_idx 63
2021.03.24 11:22:47.044718 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.044726 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.044767 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.044780 [ 1094 ] {} <Trace> RaftInstance: local log idx 64, target_commit_idx 64, quick_commit_index_ 64, state_->get_commit_idx() 62
2021.03.24 11:22:47.044788 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.044822 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.044843 [ 1096 ] {} <Debug> RaftInstance: commit upto 64, current idx 62
2021.03.24 11:22:47.044863 [ 1096 ] {} <Trace> RaftInstance: commit upto 64, current idx 63
2021.03.24 11:22:47.044911 [ 1096 ] {} <Debug> RaftInstance: notify cb 63 0x7faa0daae020
2021.03.24 11:22:47.044943 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.044955 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 63 wake up (13205 us), return value (nil)
2021.03.24 11:22:47.044977 [ 1096 ] {} <Trace> RaftInstance: commit upto 64, current idx 64
2021.03.24 11:22:47.045018 [ 1096 ] {} <Debug> RaftInstance: notify cb 64 0x7faaa6f93920
2021.03.24 11:22:47.045034 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 64, current idx 64
2021.03.24 11:22:47.045043 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.045058 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.045100 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 64 wake up (11303 us), return value (nil)
2021.03.24 11:22:47.051810 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.051839 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=65
2021.03.24 11:22:47.051850 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.051866 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 65
2021.03.24 11:22:47.051875 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.051883 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 64, new matched idx: 64
2021.03.24 11:22:47.051895 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 64 64 61 
2021.03.24 11:22:47.051904 [ 1095 ] {} <Trace> RaftInstance: local log idx 64, target_commit_idx 64, quick_commit_index_ 64, state_->get_commit_idx() 64
2021.03.24 11:22:47.053019 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.053095 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 65
2021.03.24 11:22:47.053120 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 65 0x7faaa6f93a20 sleep
2021.03.24 11:22:47.053142 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.053160 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.053169 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 64, starting_idx: 1, cur_nxt_idx: 66
2021.03.24 11:22:47.053183 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=64, LastLogTerm=1, EntriesLength=1, CommitIndex=64, Term=1, peer_last_sent_idx 65
2021.03.24 11:22:47.053191 [ 1097 ] {} <Debug> RaftInstance: idx: 65
2021.03.24 11:22:47.053201 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.053251 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.060103 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.060136 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=66
2021.03.24 11:22:47.060147 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.060158 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 66
2021.03.24 11:22:47.060167 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.060175 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 64, new matched idx: 65
2021.03.24 11:22:47.060186 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 65 65 61 
2021.03.24 11:22:47.060194 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 65
2021.03.24 11:22:47.060204 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.060212 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.060220 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 65, starting_idx: 1, cur_nxt_idx: 66
2021.03.24 11:22:47.060231 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=65, LastLogTerm=1, EntriesLength=0, CommitIndex=65, Term=1, peer_last_sent_idx 65
2021.03.24 11:22:47.060238 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.060246 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.060299 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.060312 [ 1092 ] {} <Trace> RaftInstance: local log idx 65, target_commit_idx 65, quick_commit_index_ 65, state_->get_commit_idx() 64
2021.03.24 11:22:47.060348 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.060410 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.060464 [ 1096 ] {} <Debug> RaftInstance: commit upto 65, current idx 64
2021.03.24 11:22:47.060477 [ 1096 ] {} <Trace> RaftInstance: commit upto 65, current idx 65
2021.03.24 11:22:47.060537 [ 1096 ] {} <Debug> RaftInstance: notify cb 65 0x7faaa6f93a20
2021.03.24 11:22:47.060558 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 65, current idx 65
2021.03.24 11:22:47.060567 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.060594 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.060615 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 65 wake up (7474 us), return value (nil)
2021.03.24 11:22:47.063160 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.063191 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=65
2021.03.24 11:22:47.063201 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.063212 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 65
2021.03.24 11:22:47.063220 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.063228 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 61, new matched idx: 64
2021.03.24 11:22:47.063239 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 65 65 64 
2021.03.24 11:22:47.063248 [ 1093 ] {} <Trace> RaftInstance: local log idx 65, target_commit_idx 65, quick_commit_index_ 65, state_->get_commit_idx() 65
2021.03.24 11:22:47.063256 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.063264 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.063273 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 64, starting_idx: 1, cur_nxt_idx: 66
2021.03.24 11:22:47.063284 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=64, LastLogTerm=1, EntriesLength=1, CommitIndex=65, Term=1, peer_last_sent_idx 62
2021.03.24 11:22:47.063291 [ 1093 ] {} <Debug> RaftInstance: idx: 65
2021.03.24 11:22:47.063300 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.063350 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.066951 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.066992 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=66
2021.03.24 11:22:47.067003 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.067014 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 66
2021.03.24 11:22:47.067028 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.067036 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 65, new matched idx: 65
2021.03.24 11:22:47.067054 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 65 65 64 
2021.03.24 11:22:47.067063 [ 1095 ] {} <Trace> RaftInstance: local log idx 65, target_commit_idx 65, quick_commit_index_ 65, state_->get_commit_idx() 65
2021.03.24 11:22:47.080393 [ 1171 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.080418 [ 1171 ] {} <Debug> RaftInstance: append at log_idx 66
2021.03.24 11:22:47.080462 [ 1171 ] {} <Debug> RaftInstance: commit_ret_cv 66 0x7faaa6f0ef20 sleep
2021.03.24 11:22:47.080507 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.080551 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.080562 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 65, starting_idx: 1, cur_nxt_idx: 67
2021.03.24 11:22:47.080578 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=65, LastLogTerm=1, EntriesLength=1, CommitIndex=65, Term=1, peer_last_sent_idx 66
2021.03.24 11:22:47.080587 [ 1097 ] {} <Debug> RaftInstance: idx: 66
2021.03.24 11:22:47.080599 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.080681 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.080707 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 67
2021.03.24 11:22:47.080742 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.080767 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.080776 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.080788 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 67 0x7faa1f070c20 sleep
2021.03.24 11:22:47.083814 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.083838 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=66
2021.03.24 11:22:47.083849 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.083859 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 66
2021.03.24 11:22:47.083868 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.083876 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 64, new matched idx: 65
2021.03.24 11:22:47.083887 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 67 65 65 
2021.03.24 11:22:47.083896 [ 1094 ] {} <Trace> RaftInstance: local log idx 67, target_commit_idx 65, quick_commit_index_ 65, state_->get_commit_idx() 65
2021.03.24 11:22:47.083904 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.083912 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.083920 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 65, starting_idx: 1, cur_nxt_idx: 68
2021.03.24 11:22:47.083931 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=65, LastLogTerm=1, EntriesLength=2, CommitIndex=65, Term=1, peer_last_sent_idx 65
2021.03.24 11:22:47.083938 [ 1094 ] {} <Debug> RaftInstance: idx range: 66-67
2021.03.24 11:22:47.083947 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.083991 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.086931 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.086957 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 68
2021.03.24 11:22:47.086981 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.086990 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.087001 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 68 0x7faa1f070820 sleep
2021.03.24 11:22:47.087134 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.087153 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=67
2021.03.24 11:22:47.087176 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.087185 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 67
2021.03.24 11:22:47.087193 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.087201 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 65, new matched idx: 66
2021.03.24 11:22:47.087210 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 68 66 65 
2021.03.24 11:22:47.087217 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 66
2021.03.24 11:22:47.087225 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.087244 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.087253 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 66, starting_idx: 1, cur_nxt_idx: 69
2021.03.24 11:22:47.087263 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=66, LastLogTerm=1, EntriesLength=2, CommitIndex=66, Term=1, peer_last_sent_idx 66
2021.03.24 11:22:47.087271 [ 1094 ] {} <Debug> RaftInstance: idx range: 67-68
2021.03.24 11:22:47.087279 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.087336 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.087348 [ 1094 ] {} <Trace> RaftInstance: local log idx 68, target_commit_idx 66, quick_commit_index_ 66, state_->get_commit_idx() 65
2021.03.24 11:22:47.087356 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.087386 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.087396 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.087428 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.087479 [ 1096 ] {} <Debug> RaftInstance: commit upto 66, current idx 65
2021.03.24 11:22:47.087491 [ 1096 ] {} <Trace> RaftInstance: commit upto 66, current idx 66
2021.03.24 11:22:47.087523 [ 1096 ] {} <Debug> RaftInstance: notify cb 66 0x7faaa6f0ef20
2021.03.24 11:22:47.087540 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 66, current idx 66
2021.03.24 11:22:47.087547 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.087570 [ 1171 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.087581 [ 1171 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 66 wake up (7117 us), return value 0x7faaa9a69bb0
2021.03.24 11:22:47.087653 [ 1172 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.087675 [ 1172 ] {} <Debug> RaftInstance: append at log_idx 69
2021.03.24 11:22:47.087702 [ 1172 ] {} <Debug> RaftInstance: commit_ret_cv 69 0x7faa07c25020 sleep
2021.03.24 11:22:47.087718 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.087736 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.094042 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.094073 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=69
2021.03.24 11:22:47.094084 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.094093 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 69
2021.03.24 11:22:47.094101 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.094109 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 66, new matched idx: 68
2021.03.24 11:22:47.094118 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 69 68 65 
2021.03.24 11:22:47.094125 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 68
2021.03.24 11:22:47.094143 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.094151 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.094159 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 68, starting_idx: 1, cur_nxt_idx: 70
2021.03.24 11:22:47.094169 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=68, LastLogTerm=1, EntriesLength=1, CommitIndex=68, Term=1, peer_last_sent_idx 67
2021.03.24 11:22:47.094176 [ 1093 ] {} <Debug> RaftInstance: idx: 69
2021.03.24 11:22:47.094184 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.094234 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.094247 [ 1093 ] {} <Trace> RaftInstance: local log idx 69, target_commit_idx 68, quick_commit_index_ 68, state_->get_commit_idx() 66
2021.03.24 11:22:47.094255 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.094285 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.094294 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.094334 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.094378 [ 1096 ] {} <Debug> RaftInstance: commit upto 68, current idx 66
2021.03.24 11:22:47.094391 [ 1096 ] {} <Trace> RaftInstance: commit upto 68, current idx 67
2021.03.24 11:22:47.094417 [ 1096 ] {} <Debug> RaftInstance: notify cb 67 0x7faa1f070c20
2021.03.24 11:22:47.094442 [ 1096 ] {} <Trace> RaftInstance: commit upto 68, current idx 68
2021.03.24 11:22:47.094461 [ 1096 ] {} <Debug> RaftInstance: notify cb 68 0x7faa1f070820
2021.03.24 11:22:47.094481 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 68, current idx 68
2021.03.24 11:22:47.094489 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.094501 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.094525 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 68 wake up (7528 us), return value 0x7faaa9a69bd0
2021.03.24 11:22:47.094589 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.094603 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 67 wake up (13867 us), return value 0x7faaa9a69bc0
2021.03.24 11:22:47.100960 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.100982 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=70
2021.03.24 11:22:47.100991 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.101000 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 70
2021.03.24 11:22:47.101008 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.101015 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 68, new matched idx: 69
2021.03.24 11:22:47.101025 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 69 69 65 
2021.03.24 11:22:47.101033 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 69
2021.03.24 11:22:47.101040 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.101048 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.101055 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 69, starting_idx: 1, cur_nxt_idx: 70
2021.03.24 11:22:47.101065 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=69, LastLogTerm=1, EntriesLength=0, CommitIndex=69, Term=1, peer_last_sent_idx 69
2021.03.24 11:22:47.101072 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.101099 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.101140 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.101175 [ 1094 ] {} <Trace> RaftInstance: local log idx 69, target_commit_idx 69, quick_commit_index_ 69, state_->get_commit_idx() 68
2021.03.24 11:22:47.101183 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.101236 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.101281 [ 1096 ] {} <Debug> RaftInstance: commit upto 69, current idx 68
2021.03.24 11:22:47.101292 [ 1096 ] {} <Trace> RaftInstance: commit upto 69, current idx 69
2021.03.24 11:22:47.101319 [ 1096 ] {} <Debug> RaftInstance: notify cb 69 0x7faa07c25020
2021.03.24 11:22:47.101342 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 69, current idx 69
2021.03.24 11:22:47.101350 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.101361 [ 1172 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.101378 [ 1172 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 69 wake up (13665 us), return value 0x7faaa9a69be0
2021.03.24 11:22:47.101427 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.101459 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 70
2021.03.24 11:22:47.101482 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 70 0x7faa0daae120 sleep
2021.03.24 11:22:47.101506 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.101526 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.102918 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.102943 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 71
2021.03.24 11:22:47.102966 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.102975 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.102986 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 71 0x7faa1f070820 sleep
2021.03.24 11:22:47.104395 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.104413 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=68
2021.03.24 11:22:47.104421 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.104430 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 68
2021.03.24 11:22:47.104437 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.104445 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 65, new matched idx: 67
2021.03.24 11:22:47.104454 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 71 69 67 
2021.03.24 11:22:47.104462 [ 1093 ] {} <Trace> RaftInstance: local log idx 71, target_commit_idx 69, quick_commit_index_ 69, state_->get_commit_idx() 69
2021.03.24 11:22:47.104470 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.104477 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.104485 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 67, starting_idx: 1, cur_nxt_idx: 72
2021.03.24 11:22:47.104496 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=67, LastLogTerm=1, EntriesLength=4, CommitIndex=69, Term=1, peer_last_sent_idx 66
2021.03.24 11:22:47.104503 [ 1093 ] {} <Debug> RaftInstance: idx range: 68-71
2021.03.24 11:22:47.104512 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.104555 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.107976 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.108019 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=70
2021.03.24 11:22:47.108029 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.108037 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 70
2021.03.24 11:22:47.108045 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.108052 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 69, new matched idx: 69
2021.03.24 11:22:47.108073 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 71 69 67 
2021.03.24 11:22:47.108082 [ 1094 ] {} <Trace> RaftInstance: local log idx 71, target_commit_idx 69, quick_commit_index_ 69, state_->get_commit_idx() 69
2021.03.24 11:22:47.108090 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.108097 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.108105 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 69, starting_idx: 1, cur_nxt_idx: 72
2021.03.24 11:22:47.108112 [ 1094 ] {} <Debug> RaftInstance: last sent log (70) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.108122 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=69, LastLogTerm=1, EntriesLength=2, CommitIndex=69, Term=1, peer_last_sent_idx 70
2021.03.24 11:22:47.108142 [ 1094 ] {} <Debug> RaftInstance: idx range: 70-71
2021.03.24 11:22:47.108150 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.108190 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.114389 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.114427 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 72
2021.03.24 11:22:47.114449 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 72 0x7faaa6f93a20 sleep
2021.03.24 11:22:47.114488 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.114508 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.114860 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.114881 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=72
2021.03.24 11:22:47.114890 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.114899 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 72
2021.03.24 11:22:47.114907 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.114916 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 69, new matched idx: 71
2021.03.24 11:22:47.114926 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 72 71 67 
2021.03.24 11:22:47.114933 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 71
2021.03.24 11:22:47.114941 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.114949 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.114957 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 71, starting_idx: 1, cur_nxt_idx: 73
2021.03.24 11:22:47.114967 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=71, LastLogTerm=1, EntriesLength=1, CommitIndex=71, Term=1, peer_last_sent_idx 70
2021.03.24 11:22:47.114975 [ 1094 ] {} <Debug> RaftInstance: idx: 72
2021.03.24 11:22:47.114983 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.115029 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.115042 [ 1094 ] {} <Trace> RaftInstance: local log idx 72, target_commit_idx 71, quick_commit_index_ 71, state_->get_commit_idx() 69
2021.03.24 11:22:47.115050 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.115067 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.115088 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.115136 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.115194 [ 1096 ] {} <Debug> RaftInstance: commit upto 71, current idx 69
2021.03.24 11:22:47.115207 [ 1096 ] {} <Trace> RaftInstance: commit upto 71, current idx 70
2021.03.24 11:22:47.115247 [ 1096 ] {} <Debug> RaftInstance: notify cb 70 0x7faa0daae120
2021.03.24 11:22:47.115266 [ 1096 ] {} <Trace> RaftInstance: commit upto 71, current idx 71
2021.03.24 11:22:47.115300 [ 1096 ] {} <Debug> RaftInstance: notify cb 71 0x7faa1f070820
2021.03.24 11:22:47.115316 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 71, current idx 71
2021.03.24 11:22:47.115325 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.115338 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.115357 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 70 wake up (13853 us), return value (nil)
2021.03.24 11:22:47.115381 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.115397 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 73
2021.03.24 11:22:47.115412 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 73 0x7faa0daae020 sleep
2021.03.24 11:22:47.115427 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.115436 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.115451 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.115461 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 71 wake up (12497 us), return value 0x7faaa9a69bf0
2021.03.24 11:22:47.121733 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.121767 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=73
2021.03.24 11:22:47.121778 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.121788 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 73
2021.03.24 11:22:47.121796 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.121804 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 71, new matched idx: 72
2021.03.24 11:22:47.121816 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 73 72 67 
2021.03.24 11:22:47.121824 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 72
2021.03.24 11:22:47.121833 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.121841 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.121849 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 72, starting_idx: 1, cur_nxt_idx: 74
2021.03.24 11:22:47.121859 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=72, LastLogTerm=1, EntriesLength=1, CommitIndex=72, Term=1, peer_last_sent_idx 72
2021.03.24 11:22:47.121867 [ 1094 ] {} <Debug> RaftInstance: idx: 73
2021.03.24 11:22:47.121876 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.121925 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.121938 [ 1094 ] {} <Trace> RaftInstance: local log idx 73, target_commit_idx 72, quick_commit_index_ 72, state_->get_commit_idx() 71
2021.03.24 11:22:47.121945 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.121962 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.121972 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.122005 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.122055 [ 1096 ] {} <Debug> RaftInstance: commit upto 72, current idx 71
2021.03.24 11:22:47.122091 [ 1096 ] {} <Trace> RaftInstance: commit upto 72, current idx 72
2021.03.24 11:22:47.122148 [ 1096 ] {} <Debug> RaftInstance: notify cb 72 0x7faaa6f93a20
2021.03.24 11:22:47.122169 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 72, current idx 72
2021.03.24 11:22:47.122177 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.122208 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.122230 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 72 wake up (7762 us), return value (nil)
2021.03.24 11:22:47.123758 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.123789 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 74
2021.03.24 11:22:47.123817 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.123827 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.123838 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 74 0x7faa1f070820 sleep
2021.03.24 11:22:47.125160 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.125184 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=72
2021.03.24 11:22:47.125194 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.125204 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 72
2021.03.24 11:22:47.125212 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.125221 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 67, new matched idx: 71
2021.03.24 11:22:47.125231 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 74 72 71 
2021.03.24 11:22:47.125239 [ 1092 ] {} <Trace> RaftInstance: local log idx 74, target_commit_idx 72, quick_commit_index_ 72, state_->get_commit_idx() 72
2021.03.24 11:22:47.125247 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.125255 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.125264 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 71, starting_idx: 1, cur_nxt_idx: 75
2021.03.24 11:22:47.125275 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=71, LastLogTerm=1, EntriesLength=3, CommitIndex=72, Term=1, peer_last_sent_idx 68
2021.03.24 11:22:47.125282 [ 1092 ] {} <Debug> RaftInstance: idx range: 72-74
2021.03.24 11:22:47.125291 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.125343 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.128697 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.128718 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=74
2021.03.24 11:22:47.128727 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.128736 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 74
2021.03.24 11:22:47.128744 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.128752 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 72, new matched idx: 73
2021.03.24 11:22:47.128761 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 74 73 71 
2021.03.24 11:22:47.128769 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 73
2021.03.24 11:22:47.128777 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.128785 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.128793 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 73, starting_idx: 1, cur_nxt_idx: 75
2021.03.24 11:22:47.128803 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=73, LastLogTerm=1, EntriesLength=1, CommitIndex=73, Term=1, peer_last_sent_idx 73
2021.03.24 11:22:47.128825 [ 1094 ] {} <Debug> RaftInstance: idx: 74
2021.03.24 11:22:47.128834 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.128879 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.128892 [ 1094 ] {} <Trace> RaftInstance: local log idx 74, target_commit_idx 73, quick_commit_index_ 73, state_->get_commit_idx() 72
2021.03.24 11:22:47.128899 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.128915 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.128924 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.128967 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.129016 [ 1096 ] {} <Debug> RaftInstance: commit upto 73, current idx 72
2021.03.24 11:22:47.129028 [ 1096 ] {} <Trace> RaftInstance: commit upto 73, current idx 73
2021.03.24 11:22:47.129066 [ 1096 ] {} <Debug> RaftInstance: notify cb 73 0x7faa0daae020
2021.03.24 11:22:47.129127 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 73, current idx 73
2021.03.24 11:22:47.129140 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.129142 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.129162 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 73 wake up (13728 us), return value (nil)
2021.03.24 11:22:47.129187 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.129207 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 75
2021.03.24 11:22:47.129221 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 75 0x7faa0daae120 sleep
2021.03.24 11:22:47.129273 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.129285 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.135639 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.135692 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=75
2021.03.24 11:22:47.135704 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.135714 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 75
2021.03.24 11:22:47.135722 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.135730 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 73, new matched idx: 74
2021.03.24 11:22:47.135742 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 75 74 71 
2021.03.24 11:22:47.135749 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 74
2021.03.24 11:22:47.135757 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.135765 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.135773 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 74, starting_idx: 1, cur_nxt_idx: 76
2021.03.24 11:22:47.135784 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=74, LastLogTerm=1, EntriesLength=1, CommitIndex=74, Term=1, peer_last_sent_idx 74
2021.03.24 11:22:47.135791 [ 1093 ] {} <Debug> RaftInstance: idx: 75
2021.03.24 11:22:47.135800 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.135851 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.135864 [ 1093 ] {} <Trace> RaftInstance: local log idx 75, target_commit_idx 74, quick_commit_index_ 74, state_->get_commit_idx() 73
2021.03.24 11:22:47.135872 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.135888 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.135897 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.135978 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.136035 [ 1096 ] {} <Debug> RaftInstance: commit upto 74, current idx 73
2021.03.24 11:22:47.136048 [ 1096 ] {} <Trace> RaftInstance: commit upto 74, current idx 74
2021.03.24 11:22:47.136087 [ 1096 ] {} <Debug> RaftInstance: notify cb 74 0x7faa1f070820
2021.03.24 11:22:47.136106 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 74, current idx 74
2021.03.24 11:22:47.136115 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.136134 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.136167 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 74 wake up (12326 us), return value (nil)
2021.03.24 11:22:47.142121 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.142169 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 76
2021.03.24 11:22:47.142193 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 76 0x7faa1f09b020 sleep
2021.03.24 11:22:47.142220 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.142241 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.142643 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.142690 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=76
2021.03.24 11:22:47.142701 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.142712 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 76
2021.03.24 11:22:47.142721 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.142729 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 74, new matched idx: 75
2021.03.24 11:22:47.142741 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 76 75 71 
2021.03.24 11:22:47.142749 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 75
2021.03.24 11:22:47.142758 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.142767 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.142798 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 75, starting_idx: 1, cur_nxt_idx: 77
2021.03.24 11:22:47.142815 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=75, LastLogTerm=1, EntriesLength=1, CommitIndex=75, Term=1, peer_last_sent_idx 75
2021.03.24 11:22:47.142823 [ 1093 ] {} <Debug> RaftInstance: idx: 76
2021.03.24 11:22:47.142832 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.142905 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.142918 [ 1093 ] {} <Trace> RaftInstance: local log idx 76, target_commit_idx 75, quick_commit_index_ 75, state_->get_commit_idx() 74
2021.03.24 11:22:47.142926 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.142942 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.142955 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.142979 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.143002 [ 1096 ] {} <Debug> RaftInstance: commit upto 75, current idx 74
2021.03.24 11:22:47.143013 [ 1096 ] {} <Trace> RaftInstance: commit upto 75, current idx 75
2021.03.24 11:22:47.143078 [ 1096 ] {} <Debug> RaftInstance: notify cb 75 0x7faa0daae120
2021.03.24 11:22:47.143093 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 75, current idx 75
2021.03.24 11:22:47.143102 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.143127 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.143161 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 75 wake up (13902 us), return value (nil)
2021.03.24 11:22:47.143187 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.143203 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 77
2021.03.24 11:22:47.143217 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 77 0x7faa0daae020 sleep
2021.03.24 11:22:47.143234 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.143243 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.144512 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.144539 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 78
2021.03.24 11:22:47.144564 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.144574 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.144585 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 78 0x7faa1f070820 sleep
2021.03.24 11:22:47.145810 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.145833 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=75
2021.03.24 11:22:47.145843 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.145852 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 75
2021.03.24 11:22:47.145860 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.145868 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 71, new matched idx: 74
2021.03.24 11:22:47.145879 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 78 75 74 
2021.03.24 11:22:47.145887 [ 1092 ] {} <Trace> RaftInstance: local log idx 78, target_commit_idx 75, quick_commit_index_ 75, state_->get_commit_idx() 75
2021.03.24 11:22:47.145895 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.145903 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.145911 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 74, starting_idx: 1, cur_nxt_idx: 79
2021.03.24 11:22:47.145923 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=74, LastLogTerm=1, EntriesLength=4, CommitIndex=75, Term=1, peer_last_sent_idx 72
2021.03.24 11:22:47.145931 [ 1092 ] {} <Debug> RaftInstance: idx range: 75-78
2021.03.24 11:22:47.145940 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.145990 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.149724 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.149744 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=77
2021.03.24 11:22:47.149753 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.149762 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 77
2021.03.24 11:22:47.149770 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.149777 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 75, new matched idx: 76
2021.03.24 11:22:47.149787 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 78 76 74 
2021.03.24 11:22:47.149794 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 76
2021.03.24 11:22:47.149802 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.149810 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.149817 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 76, starting_idx: 1, cur_nxt_idx: 79
2021.03.24 11:22:47.149827 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=76, LastLogTerm=1, EntriesLength=2, CommitIndex=76, Term=1, peer_last_sent_idx 76
2021.03.24 11:22:47.149846 [ 1092 ] {} <Debug> RaftInstance: idx range: 77-78
2021.03.24 11:22:47.149855 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.149900 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.149913 [ 1092 ] {} <Trace> RaftInstance: local log idx 78, target_commit_idx 76, quick_commit_index_ 76, state_->get_commit_idx() 75
2021.03.24 11:22:47.149921 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.149937 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.149946 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.150041 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.150102 [ 1096 ] {} <Debug> RaftInstance: commit upto 76, current idx 75
2021.03.24 11:22:47.150116 [ 1096 ] {} <Trace> RaftInstance: commit upto 76, current idx 76
2021.03.24 11:22:47.150164 [ 1096 ] {} <Debug> RaftInstance: notify cb 76 0x7faa1f09b020
2021.03.24 11:22:47.150186 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 76, current idx 76
2021.03.24 11:22:47.150195 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.150206 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.150222 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 76 wake up (8019 us), return value (nil)
2021.03.24 11:22:47.156609 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.156639 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=79
2021.03.24 11:22:47.156649 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.156659 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 79
2021.03.24 11:22:47.156667 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.156674 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 76, new matched idx: 78
2021.03.24 11:22:47.156685 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 78 78 74 
2021.03.24 11:22:47.156693 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 78
2021.03.24 11:22:47.156701 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.156708 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.156717 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 78, starting_idx: 1, cur_nxt_idx: 79
2021.03.24 11:22:47.156726 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=78, LastLogTerm=1, EntriesLength=0, CommitIndex=78, Term=1, peer_last_sent_idx 77
2021.03.24 11:22:47.156734 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.156741 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.156791 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.156811 [ 1093 ] {} <Trace> RaftInstance: local log idx 78, target_commit_idx 78, quick_commit_index_ 78, state_->get_commit_idx() 76
2021.03.24 11:22:47.156819 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.156941 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.157000 [ 1096 ] {} <Debug> RaftInstance: commit upto 78, current idx 76
2021.03.24 11:22:47.157013 [ 1096 ] {} <Trace> RaftInstance: commit upto 78, current idx 77
2021.03.24 11:22:47.157110 [ 1096 ] {} <Debug> RaftInstance: notify cb 77 0x7faa0daae020
2021.03.24 11:22:47.157132 [ 1096 ] {} <Trace> RaftInstance: commit upto 78, current idx 78
2021.03.24 11:22:47.157150 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.157187 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 77 wake up (13930 us), return value (nil)
2021.03.24 11:22:47.157221 [ 1096 ] {} <Debug> RaftInstance: notify cb 78 0x7faa1f070820
2021.03.24 11:22:47.157239 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 78, current idx 78
2021.03.24 11:22:47.157247 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.157249 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.157266 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 78 wake up (12695 us), return value (nil)
2021.03.24 11:22:47.163413 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.163436 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=79
2021.03.24 11:22:47.163446 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.163456 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 79
2021.03.24 11:22:47.163464 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.163472 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 78, new matched idx: 78
2021.03.24 11:22:47.163482 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 78 78 74 
2021.03.24 11:22:47.163490 [ 1094 ] {} <Trace> RaftInstance: local log idx 78, target_commit_idx 78, quick_commit_index_ 78, state_->get_commit_idx() 78
2021.03.24 11:22:47.164262 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.164292 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 79
2021.03.24 11:22:47.164309 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 79 0x7faa0daae120 sleep
2021.03.24 11:22:47.164330 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.164350 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.164360 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 78, starting_idx: 1, cur_nxt_idx: 80
2021.03.24 11:22:47.164372 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=78, LastLogTerm=1, EntriesLength=1, CommitIndex=78, Term=1, peer_last_sent_idx 79
2021.03.24 11:22:47.164380 [ 1097 ] {} <Debug> RaftInstance: idx: 79
2021.03.24 11:22:47.164389 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.164432 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.165694 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.165720 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 80
2021.03.24 11:22:47.165739 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 80 0x7faa1f070c20 sleep
2021.03.24 11:22:47.165764 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.165782 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.166566 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.166587 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=79
2021.03.24 11:22:47.166596 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.166606 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 79
2021.03.24 11:22:47.166624 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.166633 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 74, new matched idx: 78
2021.03.24 11:22:47.166643 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 80 78 78 
2021.03.24 11:22:47.166651 [ 1092 ] {} <Trace> RaftInstance: local log idx 80, target_commit_idx 78, quick_commit_index_ 78, state_->get_commit_idx() 78
2021.03.24 11:22:47.166660 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.166679 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.166688 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 78, starting_idx: 1, cur_nxt_idx: 81
2021.03.24 11:22:47.166699 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=78, LastLogTerm=1, EntriesLength=2, CommitIndex=78, Term=1, peer_last_sent_idx 75
2021.03.24 11:22:47.166706 [ 1092 ] {} <Debug> RaftInstance: idx range: 79-80
2021.03.24 11:22:47.166715 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.166792 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.170441 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.170475 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 81
2021.03.24 11:22:47.170518 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.170528 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.170540 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 81 0x7faa1f070820 sleep
2021.03.24 11:22:47.171132 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.171191 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=80
2021.03.24 11:22:47.171203 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.171213 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 80
2021.03.24 11:22:47.171222 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.171231 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 78, new matched idx: 79
2021.03.24 11:22:47.171244 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 81 79 78 
2021.03.24 11:22:47.171252 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 79
2021.03.24 11:22:47.171261 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.171270 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.171278 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 79, starting_idx: 1, cur_nxt_idx: 82
2021.03.24 11:22:47.171291 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=79, LastLogTerm=1, EntriesLength=2, CommitIndex=79, Term=1, peer_last_sent_idx 79
2021.03.24 11:22:47.171299 [ 1094 ] {} <Debug> RaftInstance: idx range: 80-81
2021.03.24 11:22:47.171308 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.171396 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.171410 [ 1094 ] {} <Trace> RaftInstance: local log idx 81, target_commit_idx 79, quick_commit_index_ 79, state_->get_commit_idx() 78
2021.03.24 11:22:47.171419 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.171439 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.171448 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.171483 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.171515 [ 1096 ] {} <Debug> RaftInstance: commit upto 79, current idx 78
2021.03.24 11:22:47.171527 [ 1096 ] {} <Trace> RaftInstance: commit upto 79, current idx 79
2021.03.24 11:22:47.171559 [ 1096 ] {} <Debug> RaftInstance: notify cb 79 0x7faa0daae120
2021.03.24 11:22:47.171579 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 79, current idx 79
2021.03.24 11:22:47.171588 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.171600 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.171618 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 79 wake up (7290 us), return value (nil)
2021.03.24 11:22:47.172581 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.172619 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 82
2021.03.24 11:22:47.172634 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 82 0x7faa0daae020 sleep
2021.03.24 11:22:47.172652 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.172661 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.178120 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.178148 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=82
2021.03.24 11:22:47.178159 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.178170 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 82
2021.03.24 11:22:47.178178 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.178187 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 79, new matched idx: 81
2021.03.24 11:22:47.178197 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 82 81 78 
2021.03.24 11:22:47.178205 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 81
2021.03.24 11:22:47.178214 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.178222 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.178230 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 81, starting_idx: 1, cur_nxt_idx: 83
2021.03.24 11:22:47.178243 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=81, LastLogTerm=1, EntriesLength=1, CommitIndex=81, Term=1, peer_last_sent_idx 80
2021.03.24 11:22:47.178250 [ 1092 ] {} <Debug> RaftInstance: idx: 82
2021.03.24 11:22:47.178260 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.178320 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.178333 [ 1092 ] {} <Trace> RaftInstance: local log idx 82, target_commit_idx 81, quick_commit_index_ 81, state_->get_commit_idx() 79
2021.03.24 11:22:47.178341 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.178357 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.178370 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.178440 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.178492 [ 1096 ] {} <Debug> RaftInstance: commit upto 81, current idx 79
2021.03.24 11:22:47.178504 [ 1096 ] {} <Trace> RaftInstance: commit upto 81, current idx 80
2021.03.24 11:22:47.178547 [ 1096 ] {} <Debug> RaftInstance: notify cb 80 0x7faa1f070c20
2021.03.24 11:22:47.178569 [ 1096 ] {} <Trace> RaftInstance: commit upto 81, current idx 81
2021.03.24 11:22:47.178598 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.178627 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 80 wake up (12858 us), return value (nil)
2021.03.24 11:22:47.178651 [ 1096 ] {} <Debug> RaftInstance: notify cb 81 0x7faa1f070820
2021.03.24 11:22:47.178668 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 81, current idx 81
2021.03.24 11:22:47.178677 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.178692 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.178708 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 81 wake up (8200 us), return value (nil)
2021.03.24 11:22:47.185026 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.185121 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=83
2021.03.24 11:22:47.185136 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.185171 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 83
2021.03.24 11:22:47.185179 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.185187 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 81, new matched idx: 82
2021.03.24 11:22:47.185199 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 82 82 78 
2021.03.24 11:22:47.185207 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 82
2021.03.24 11:22:47.185216 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.185224 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.185232 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 82, starting_idx: 1, cur_nxt_idx: 83
2021.03.24 11:22:47.185243 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=82, LastLogTerm=1, EntriesLength=0, CommitIndex=82, Term=1, peer_last_sent_idx 82
2021.03.24 11:22:47.185251 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.185259 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.185349 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.185363 [ 1094 ] {} <Trace> RaftInstance: local log idx 82, target_commit_idx 82, quick_commit_index_ 82, state_->get_commit_idx() 81
2021.03.24 11:22:47.185372 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.185448 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.185473 [ 1096 ] {} <Debug> RaftInstance: commit upto 82, current idx 81
2021.03.24 11:22:47.185484 [ 1096 ] {} <Trace> RaftInstance: commit upto 82, current idx 82
2021.03.24 11:22:47.185544 [ 1096 ] {} <Debug> RaftInstance: notify cb 82 0x7faa0daae020
2021.03.24 11:22:47.185565 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 82, current idx 82
2021.03.24 11:22:47.185574 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.185669 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.185696 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 82 wake up (13029 us), return value (nil)
2021.03.24 11:22:47.187048 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.187082 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 83
2021.03.24 11:22:47.187113 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.187124 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.187136 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 83 0x7faaa6f93c20 sleep
2021.03.24 11:22:47.187221 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.187243 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=81
2021.03.24 11:22:47.187253 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.187263 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 81
2021.03.24 11:22:47.187271 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.187279 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 78, new matched idx: 80
2021.03.24 11:22:47.187290 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 83 82 80 
2021.03.24 11:22:47.187298 [ 1093 ] {} <Trace> RaftInstance: local log idx 83, target_commit_idx 82, quick_commit_index_ 82, state_->get_commit_idx() 82
2021.03.24 11:22:47.187307 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.187315 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.187324 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 80, starting_idx: 1, cur_nxt_idx: 84
2021.03.24 11:22:47.187352 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=80, LastLogTerm=1, EntriesLength=3, CommitIndex=82, Term=1, peer_last_sent_idx 79
2021.03.24 11:22:47.187360 [ 1093 ] {} <Debug> RaftInstance: idx range: 81-83
2021.03.24 11:22:47.187370 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.187426 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.191878 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.191925 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=83
2021.03.24 11:22:47.191935 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.191946 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 83
2021.03.24 11:22:47.191954 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.191962 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 82, new matched idx: 82
2021.03.24 11:22:47.191974 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 83 82 80 
2021.03.24 11:22:47.191983 [ 1094 ] {} <Trace> RaftInstance: local log idx 83, target_commit_idx 82, quick_commit_index_ 82, state_->get_commit_idx() 82
2021.03.24 11:22:47.191991 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.191999 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.192007 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 82, starting_idx: 1, cur_nxt_idx: 84
2021.03.24 11:22:47.192018 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=82, LastLogTerm=1, EntriesLength=1, CommitIndex=82, Term=1, peer_last_sent_idx 83
2021.03.24 11:22:47.192026 [ 1094 ] {} <Debug> RaftInstance: idx: 83
2021.03.24 11:22:47.192035 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.192107 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.198652 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.198708 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 84
2021.03.24 11:22:47.198732 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 84 0x7faa1f09b620 sleep
2021.03.24 11:22:47.198758 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.198769 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.198832 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.198844 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 82, starting_idx: 1, cur_nxt_idx: 85
2021.03.24 11:22:47.198854 [ 1097 ] {} <Debug> RaftInstance: last sent log (83) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.198867 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=82, LastLogTerm=1, EntriesLength=2, CommitIndex=82, Term=1, peer_last_sent_idx 83
2021.03.24 11:22:47.198876 [ 1097 ] {} <Debug> RaftInstance: idx range: 83-84
2021.03.24 11:22:47.198887 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.198952 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.198982 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=84
2021.03.24 11:22:47.198999 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.199009 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 84
2021.03.24 11:22:47.199018 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.199026 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 82, new matched idx: 83
2021.03.24 11:22:47.199037 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 84 83 80 
2021.03.24 11:22:47.199052 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 83
2021.03.24 11:22:47.199079 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.199088 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.199096 [ 1093 ] {} <Trace> RaftInstance: local log idx 84, target_commit_idx 83, quick_commit_index_ 83, state_->get_commit_idx() 82
2021.03.24 11:22:47.199104 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.199120 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.199129 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.199186 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.199213 [ 1096 ] {} <Debug> RaftInstance: commit upto 83, current idx 82
2021.03.24 11:22:47.199225 [ 1096 ] {} <Trace> RaftInstance: commit upto 83, current idx 83
2021.03.24 11:22:47.199270 [ 1096 ] {} <Debug> RaftInstance: notify cb 83 0x7faaa6f93c20
2021.03.24 11:22:47.199294 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 83, current idx 83
2021.03.24 11:22:47.199303 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.199318 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.199336 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 83 wake up (12218 us), return value (nil)
2021.03.24 11:22:47.205537 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.205561 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=85
2021.03.24 11:22:47.205570 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.205579 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 85
2021.03.24 11:22:47.205586 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.205594 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 83, new matched idx: 84
2021.03.24 11:22:47.205604 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 84 84 80 
2021.03.24 11:22:47.205611 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 84
2021.03.24 11:22:47.205619 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.205628 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.205635 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 84, starting_idx: 1, cur_nxt_idx: 85
2021.03.24 11:22:47.205645 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=84, LastLogTerm=1, EntriesLength=0, CommitIndex=84, Term=1, peer_last_sent_idx 83
2021.03.24 11:22:47.205652 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.205659 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.205707 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.205719 [ 1093 ] {} <Trace> RaftInstance: local log idx 84, target_commit_idx 84, quick_commit_index_ 84, state_->get_commit_idx() 83
2021.03.24 11:22:47.205727 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.205824 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.205869 [ 1096 ] {} <Debug> RaftInstance: commit upto 84, current idx 83
2021.03.24 11:22:47.205880 [ 1096 ] {} <Trace> RaftInstance: commit upto 84, current idx 84
2021.03.24 11:22:47.205931 [ 1096 ] {} <Debug> RaftInstance: notify cb 84 0x7faa1f09b620
2021.03.24 11:22:47.205968 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 84, current idx 84
2021.03.24 11:22:47.205978 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.206022 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.206046 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 84 wake up (7292 us), return value 0x7faaa9a69c00
2021.03.24 11:22:47.207814 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.207848 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 85
2021.03.24 11:22:47.207874 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.207883 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.207895 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 85 0x7faaa6f93c20 sleep
2021.03.24 11:22:47.208104 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.208123 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=84
2021.03.24 11:22:47.208132 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.208141 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 84
2021.03.24 11:22:47.208148 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.208156 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 80, new matched idx: 83
2021.03.24 11:22:47.208170 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 85 84 83 
2021.03.24 11:22:47.208179 [ 1095 ] {} <Trace> RaftInstance: local log idx 85, target_commit_idx 84, quick_commit_index_ 84, state_->get_commit_idx() 84
2021.03.24 11:22:47.208187 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.208194 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.208202 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 83, starting_idx: 1, cur_nxt_idx: 86
2021.03.24 11:22:47.208213 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=83, LastLogTerm=1, EntriesLength=2, CommitIndex=84, Term=1, peer_last_sent_idx 81
2021.03.24 11:22:47.208220 [ 1095 ] {} <Debug> RaftInstance: idx range: 84-85
2021.03.24 11:22:47.208228 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.208277 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.212212 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.212244 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=85
2021.03.24 11:22:47.212266 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.212276 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 85
2021.03.24 11:22:47.212284 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.212291 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 84, new matched idx: 84
2021.03.24 11:22:47.212301 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 85 84 83 
2021.03.24 11:22:47.212309 [ 1093 ] {} <Trace> RaftInstance: local log idx 85, target_commit_idx 84, quick_commit_index_ 84, state_->get_commit_idx() 84
2021.03.24 11:22:47.212317 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.212325 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.212333 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 84, starting_idx: 1, cur_nxt_idx: 86
2021.03.24 11:22:47.212343 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=84, LastLogTerm=1, EntriesLength=1, CommitIndex=84, Term=1, peer_last_sent_idx 85
2021.03.24 11:22:47.212350 [ 1093 ] {} <Debug> RaftInstance: idx: 85
2021.03.24 11:22:47.212359 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.212401 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.219129 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.219156 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=86
2021.03.24 11:22:47.219179 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.219189 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 86
2021.03.24 11:22:47.219197 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.219204 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 84, new matched idx: 85
2021.03.24 11:22:47.219214 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 85 85 83 
2021.03.24 11:22:47.219222 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 85
2021.03.24 11:22:47.219231 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.219238 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.219246 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 85, starting_idx: 1, cur_nxt_idx: 86
2021.03.24 11:22:47.219255 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=85, LastLogTerm=1, EntriesLength=0, CommitIndex=85, Term=1, peer_last_sent_idx 85
2021.03.24 11:22:47.219263 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.219270 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.219316 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.219329 [ 1093 ] {} <Trace> RaftInstance: local log idx 85, target_commit_idx 85, quick_commit_index_ 85, state_->get_commit_idx() 84
2021.03.24 11:22:47.219336 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.219420 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.219490 [ 1096 ] {} <Debug> RaftInstance: commit upto 85, current idx 84
2021.03.24 11:22:47.219504 [ 1096 ] {} <Trace> RaftInstance: commit upto 85, current idx 85
2021.03.24 11:22:47.219534 [ 1096 ] {} <Debug> RaftInstance: notify cb 85 0x7faaa6f93c20
2021.03.24 11:22:47.219570 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 85, current idx 85
2021.03.24 11:22:47.219586 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.219615 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.219638 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 85 wake up (11748 us), return value 0x7faaa9a69c10
2021.03.24 11:22:47.226148 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.226170 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.226180 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 86
2021.03.24 11:22:47.226236 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=86
2021.03.24 11:22:47.226275 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 86 0x7faa1f09b820 sleep
2021.03.24 11:22:47.226276 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.226311 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 86
2021.03.24 11:22:47.226319 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.226327 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 85, new matched idx: 85
2021.03.24 11:22:47.226339 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 86 85 83 
2021.03.24 11:22:47.226348 [ 1094 ] {} <Trace> RaftInstance: local log idx 86, target_commit_idx 85, quick_commit_index_ 85, state_->get_commit_idx() 85
2021.03.24 11:22:47.226357 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.226365 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.226378 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 85, starting_idx: 1, cur_nxt_idx: 87
2021.03.24 11:22:47.226415 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=85, LastLogTerm=1, EntriesLength=1, CommitIndex=85, Term=1, peer_last_sent_idx 86
2021.03.24 11:22:47.226423 [ 1094 ] {} <Debug> RaftInstance: idx: 86
2021.03.24 11:22:47.226432 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.226516 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.226570 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.226591 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.228628 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.228656 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=86
2021.03.24 11:22:47.228665 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.228674 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 86
2021.03.24 11:22:47.228682 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.228690 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 83, new matched idx: 85
2021.03.24 11:22:47.228700 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 86 85 85 
2021.03.24 11:22:47.228708 [ 1092 ] {} <Trace> RaftInstance: local log idx 86, target_commit_idx 85, quick_commit_index_ 85, state_->get_commit_idx() 85
2021.03.24 11:22:47.228716 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.228724 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.228732 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 85, starting_idx: 1, cur_nxt_idx: 87
2021.03.24 11:22:47.228743 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=85, LastLogTerm=1, EntriesLength=1, CommitIndex=85, Term=1, peer_last_sent_idx 84
2021.03.24 11:22:47.228750 [ 1092 ] {} <Debug> RaftInstance: idx: 86
2021.03.24 11:22:47.228759 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.228813 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.233261 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.233292 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=87
2021.03.24 11:22:47.233303 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.233325 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 87
2021.03.24 11:22:47.233333 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.233341 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 85, new matched idx: 86
2021.03.24 11:22:47.233351 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 86 86 85 
2021.03.24 11:22:47.233358 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 86
2021.03.24 11:22:47.233366 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.233387 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.233395 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 86, starting_idx: 1, cur_nxt_idx: 87
2021.03.24 11:22:47.233405 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=86, LastLogTerm=1, EntriesLength=0, CommitIndex=86, Term=1, peer_last_sent_idx 86
2021.03.24 11:22:47.233416 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.233423 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.233482 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.233495 [ 1095 ] {} <Trace> RaftInstance: local log idx 86, target_commit_idx 86, quick_commit_index_ 86, state_->get_commit_idx() 85
2021.03.24 11:22:47.233502 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.233593 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.233659 [ 1096 ] {} <Debug> RaftInstance: commit upto 86, current idx 85
2021.03.24 11:22:47.233671 [ 1096 ] {} <Trace> RaftInstance: commit upto 86, current idx 86
2021.03.24 11:22:47.233699 [ 1096 ] {} <Debug> RaftInstance: notify cb 86 0x7faa1f09b820
2021.03.24 11:22:47.233721 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 86, current idx 86
2021.03.24 11:22:47.233730 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.233743 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.233760 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 86 wake up (7475 us), return value 0x7faaa9a69c20
2021.03.24 11:22:47.234596 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.234628 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 87
2021.03.24 11:22:47.234656 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.234666 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.234678 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 87 0x7faa1f070820 sleep
2021.03.24 11:22:47.240314 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.240389 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=87
2021.03.24 11:22:47.240399 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.240410 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 87
2021.03.24 11:22:47.240418 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.240431 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 86, new matched idx: 86
2021.03.24 11:22:47.240443 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 87 86 85 
2021.03.24 11:22:47.240452 [ 1094 ] {} <Trace> RaftInstance: local log idx 87, target_commit_idx 86, quick_commit_index_ 86, state_->get_commit_idx() 86
2021.03.24 11:22:47.240460 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.240468 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.240476 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 86, starting_idx: 1, cur_nxt_idx: 88
2021.03.24 11:22:47.240488 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=86, LastLogTerm=1, EntriesLength=1, CommitIndex=86, Term=1, peer_last_sent_idx 87
2021.03.24 11:22:47.240496 [ 1094 ] {} <Debug> RaftInstance: idx: 87
2021.03.24 11:22:47.240505 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.240597 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.242550 [ 1093 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:47.242572 [ 1093 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:47.242581 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.247309 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.247352 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=88
2021.03.24 11:22:47.247361 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.247372 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 88
2021.03.24 11:22:47.247380 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.247388 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 86, new matched idx: 87
2021.03.24 11:22:47.247400 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 87 87 85 
2021.03.24 11:22:47.247441 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 87
2021.03.24 11:22:47.247450 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.247458 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.247467 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 87, starting_idx: 1, cur_nxt_idx: 88
2021.03.24 11:22:47.247479 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=87, LastLogTerm=1, EntriesLength=0, CommitIndex=87, Term=1, peer_last_sent_idx 87
2021.03.24 11:22:47.247486 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.247494 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.247568 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.247581 [ 1094 ] {} <Trace> RaftInstance: local log idx 87, target_commit_idx 87, quick_commit_index_ 87, state_->get_commit_idx() 86
2021.03.24 11:22:47.247590 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.247653 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.247678 [ 1096 ] {} <Debug> RaftInstance: commit upto 87, current idx 86
2021.03.24 11:22:47.247690 [ 1096 ] {} <Trace> RaftInstance: commit upto 87, current idx 87
2021.03.24 11:22:47.247721 [ 1096 ] {} <Debug> RaftInstance: notify cb 87 0x7faa1f070820
2021.03.24 11:22:47.247743 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 87, current idx 87
2021.03.24 11:22:47.247753 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.247778 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.247802 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 87 wake up (13132 us), return value (nil)
2021.03.24 11:22:47.249373 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.249394 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=87
2021.03.24 11:22:47.249404 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.249413 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 87
2021.03.24 11:22:47.249426 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.249435 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 85, new matched idx: 86
2021.03.24 11:22:47.249445 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 87 87 86 
2021.03.24 11:22:47.249453 [ 1095 ] {} <Trace> RaftInstance: local log idx 87, target_commit_idx 87, quick_commit_index_ 87, state_->get_commit_idx() 87
2021.03.24 11:22:47.249461 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.249469 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.249477 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 86, starting_idx: 1, cur_nxt_idx: 88
2021.03.24 11:22:47.249489 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=86, LastLogTerm=1, EntriesLength=1, CommitIndex=87, Term=1, peer_last_sent_idx 86
2021.03.24 11:22:47.249496 [ 1095 ] {} <Debug> RaftInstance: idx: 87
2021.03.24 11:22:47.249505 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.249591 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.253706 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.253764 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 88
2021.03.24 11:22:47.253788 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 88 0x7faa1f09b920 sleep
2021.03.24 11:22:47.253822 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.253843 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.254261 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.254297 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=88
2021.03.24 11:22:47.254320 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.254330 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 88
2021.03.24 11:22:47.254338 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.254346 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 87, new matched idx: 87
2021.03.24 11:22:47.254355 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 88 87 86 
2021.03.24 11:22:47.254363 [ 1093 ] {} <Trace> RaftInstance: local log idx 88, target_commit_idx 87, quick_commit_index_ 87, state_->get_commit_idx() 87
2021.03.24 11:22:47.254371 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.254379 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.254387 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 87, starting_idx: 1, cur_nxt_idx: 89
2021.03.24 11:22:47.254397 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=87, LastLogTerm=1, EntriesLength=1, CommitIndex=87, Term=1, peer_last_sent_idx 88
2021.03.24 11:22:47.254405 [ 1093 ] {} <Debug> RaftInstance: idx: 88
2021.03.24 11:22:47.254414 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.254455 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.256118 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.256148 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 89
2021.03.24 11:22:47.256173 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.256183 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.256195 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 89 0x7faa1f070820 sleep
2021.03.24 11:22:47.261032 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.261054 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=89
2021.03.24 11:22:47.261063 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.261072 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 89
2021.03.24 11:22:47.261079 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.261124 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 87, new matched idx: 88
2021.03.24 11:22:47.261136 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 89 88 86 
2021.03.24 11:22:47.261144 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 88
2021.03.24 11:22:47.261152 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.261160 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.261168 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 88, starting_idx: 1, cur_nxt_idx: 90
2021.03.24 11:22:47.261178 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=88, LastLogTerm=1, EntriesLength=1, CommitIndex=88, Term=1, peer_last_sent_idx 88
2021.03.24 11:22:47.261185 [ 1093 ] {} <Debug> RaftInstance: idx: 89
2021.03.24 11:22:47.261194 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.261231 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.261242 [ 1093 ] {} <Trace> RaftInstance: local log idx 89, target_commit_idx 88, quick_commit_index_ 88, state_->get_commit_idx() 87
2021.03.24 11:22:47.261250 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.261265 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.261274 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.261367 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.261425 [ 1096 ] {} <Debug> RaftInstance: commit upto 88, current idx 87
2021.03.24 11:22:47.261437 [ 1096 ] {} <Trace> RaftInstance: commit upto 88, current idx 88
2021.03.24 11:22:47.261473 [ 1096 ] {} <Debug> RaftInstance: notify cb 88 0x7faa1f09b920
2021.03.24 11:22:47.261507 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 88, current idx 88
2021.03.24 11:22:47.261517 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.261543 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.261610 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 88 wake up (7758 us), return value (nil)
2021.03.24 11:22:47.267906 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.267931 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=90
2021.03.24 11:22:47.267940 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.267949 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 90
2021.03.24 11:22:47.267957 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.267964 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 88, new matched idx: 89
2021.03.24 11:22:47.267974 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 89 89 86 
2021.03.24 11:22:47.267981 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 89
2021.03.24 11:22:47.267989 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.267996 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.268004 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 89, starting_idx: 1, cur_nxt_idx: 90
2021.03.24 11:22:47.268013 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=89, LastLogTerm=1, EntriesLength=0, CommitIndex=89, Term=1, peer_last_sent_idx 89
2021.03.24 11:22:47.268021 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.268028 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.268068 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.268081 [ 1093 ] {} <Trace> RaftInstance: local log idx 89, target_commit_idx 89, quick_commit_index_ 89, state_->get_commit_idx() 88
2021.03.24 11:22:47.268089 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.268170 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.268229 [ 1096 ] {} <Debug> RaftInstance: commit upto 89, current idx 88
2021.03.24 11:22:47.268241 [ 1096 ] {} <Trace> RaftInstance: commit upto 89, current idx 89
2021.03.24 11:22:47.268271 [ 1096 ] {} <Debug> RaftInstance: notify cb 89 0x7faa1f070820
2021.03.24 11:22:47.268301 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 89, current idx 89
2021.03.24 11:22:47.268310 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.268335 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.268355 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 89 wake up (12168 us), return value 0x7faaa9a69c30
2021.03.24 11:22:47.270057 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.270090 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=88
2021.03.24 11:22:47.270101 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.270112 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 88
2021.03.24 11:22:47.270120 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.270128 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 86, new matched idx: 87
2021.03.24 11:22:47.270171 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 89 89 87 
2021.03.24 11:22:47.270180 [ 1094 ] {} <Trace> RaftInstance: local log idx 89, target_commit_idx 89, quick_commit_index_ 89, state_->get_commit_idx() 89
2021.03.24 11:22:47.270188 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.270201 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.270210 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 87, starting_idx: 1, cur_nxt_idx: 90
2021.03.24 11:22:47.270222 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=87, LastLogTerm=1, EntriesLength=2, CommitIndex=89, Term=1, peer_last_sent_idx 87
2021.03.24 11:22:47.270230 [ 1094 ] {} <Debug> RaftInstance: idx range: 88-89
2021.03.24 11:22:47.270248 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.270332 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.274670 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.274695 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=90
2021.03.24 11:22:47.274706 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.274715 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 90
2021.03.24 11:22:47.274723 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.274730 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 89, new matched idx: 89
2021.03.24 11:22:47.274740 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 89 89 87 
2021.03.24 11:22:47.274749 [ 1095 ] {} <Trace> RaftInstance: local log idx 89, target_commit_idx 89, quick_commit_index_ 89, state_->get_commit_idx() 89
2021.03.24 11:22:47.276769 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.276797 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 90
2021.03.24 11:22:47.276816 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 90 0x7faa1f070b20 sleep
2021.03.24 11:22:47.276832 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.276851 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.276861 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 89, starting_idx: 1, cur_nxt_idx: 91
2021.03.24 11:22:47.276881 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=89, LastLogTerm=1, EntriesLength=1, CommitIndex=89, Term=1, peer_last_sent_idx 90
2021.03.24 11:22:47.276889 [ 1097 ] {} <Debug> RaftInstance: idx: 90
2021.03.24 11:22:47.276899 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.276969 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.281747 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.281780 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 91
2021.03.24 11:22:47.281817 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.281827 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.281840 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 91 0x7faa1f070820 sleep
2021.03.24 11:22:47.283647 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.283705 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=91
2021.03.24 11:22:47.283716 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.283733 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 91
2021.03.24 11:22:47.283741 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.283775 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 89, new matched idx: 90
2021.03.24 11:22:47.283788 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 91 90 87 
2021.03.24 11:22:47.283796 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 90
2021.03.24 11:22:47.283805 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.283813 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.283820 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 90, starting_idx: 1, cur_nxt_idx: 92
2021.03.24 11:22:47.283832 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=90, LastLogTerm=1, EntriesLength=1, CommitIndex=90, Term=1, peer_last_sent_idx 90
2021.03.24 11:22:47.283839 [ 1094 ] {} <Debug> RaftInstance: idx: 91
2021.03.24 11:22:47.283848 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.283939 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.283952 [ 1094 ] {} <Trace> RaftInstance: local log idx 91, target_commit_idx 90, quick_commit_index_ 90, state_->get_commit_idx() 89
2021.03.24 11:22:47.283960 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.283979 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.283989 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.284042 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.284066 [ 1096 ] {} <Debug> RaftInstance: commit upto 90, current idx 89
2021.03.24 11:22:47.284077 [ 1096 ] {} <Trace> RaftInstance: commit upto 90, current idx 90
2021.03.24 11:22:47.284156 [ 1096 ] {} <Debug> RaftInstance: notify cb 90 0x7faa1f070b20
2021.03.24 11:22:47.284179 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 90, current idx 90
2021.03.24 11:22:47.284187 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.284209 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.284230 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 90 wake up (7394 us), return value (nil)
2021.03.24 11:22:47.290773 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.290816 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=92
2021.03.24 11:22:47.290833 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.290809 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.290899 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 92
2021.03.24 11:22:47.290974 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.290989 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 90, new matched idx: 91
2021.03.24 11:22:47.291000 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 91 91 87 
2021.03.24 11:22:47.291020 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 91
2021.03.24 11:22:47.291029 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.291038 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 87, starting_idx: 1, cur_nxt_idx: 92
2021.03.24 11:22:47.291045 [ 1095 ] {} <Debug> RaftInstance: last sent log (88) to peer 3 is not applied, cnt 1
2021.03.24 11:22:47.291057 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=87, LastLogTerm=1, EntriesLength=4, CommitIndex=91, Term=1, peer_last_sent_idx 88
2021.03.24 11:22:47.291065 [ 1095 ] {} <Debug> RaftInstance: idx range: 88-91
2021.03.24 11:22:47.291074 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.291125 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.291154 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.291163 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 91, starting_idx: 1, cur_nxt_idx: 92
2021.03.24 11:22:47.291173 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=91, LastLogTerm=1, EntriesLength=0, CommitIndex=91, Term=1, peer_last_sent_idx 91
2021.03.24 11:22:47.291180 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.291187 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.291222 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.291234 [ 1095 ] {} <Trace> RaftInstance: local log idx 91, target_commit_idx 91, quick_commit_index_ 91, state_->get_commit_idx() 90
2021.03.24 11:22:47.291241 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.291344 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=90
2021.03.24 11:22:47.291372 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.291350 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.291407 [ 1096 ] {} <Debug> RaftInstance: commit upto 91, current idx 90
2021.03.24 11:22:47.291386 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 90
2021.03.24 11:22:47.291426 [ 1096 ] {} <Trace> RaftInstance: commit upto 91, current idx 91
2021.03.24 11:22:47.291427 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.291440 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 87, new matched idx: 89
2021.03.24 11:22:47.291452 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 91 91 89 
2021.03.24 11:22:47.291461 [ 1094 ] {} <Trace> RaftInstance: local log idx 91, target_commit_idx 91, quick_commit_index_ 91, state_->get_commit_idx() 90
2021.03.24 11:22:47.291468 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.291473 [ 1096 ] {} <Debug> RaftInstance: notify cb 91 0x7faa1f070820
2021.03.24 11:22:47.291476 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.291497 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.291498 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 91, current idx 91
2021.03.24 11:22:47.291515 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.291548 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.291574 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 91 wake up (9749 us), return value (nil)
2021.03.24 11:22:47.292517 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.292553 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 92
2021.03.24 11:22:47.292578 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 92 0x7faa1f09b820 sleep
2021.03.24 11:22:47.292599 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.292616 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.297821 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.297867 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=92
2021.03.24 11:22:47.297878 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.297890 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 92
2021.03.24 11:22:47.297898 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.297906 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 91, new matched idx: 91
2021.03.24 11:22:47.297917 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 92 91 89 
2021.03.24 11:22:47.297926 [ 1094 ] {} <Trace> RaftInstance: local log idx 92, target_commit_idx 91, quick_commit_index_ 91, state_->get_commit_idx() 91
2021.03.24 11:22:47.297955 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.297964 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.297972 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 91, starting_idx: 1, cur_nxt_idx: 93
2021.03.24 11:22:47.297984 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=91, LastLogTerm=1, EntriesLength=1, CommitIndex=91, Term=1, peer_last_sent_idx 92
2021.03.24 11:22:47.297991 [ 1094 ] {} <Debug> RaftInstance: idx: 92
2021.03.24 11:22:47.298000 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.298086 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.304739 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.304769 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=93
2021.03.24 11:22:47.304780 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.304790 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 93
2021.03.24 11:22:47.304798 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.304806 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 91, new matched idx: 92
2021.03.24 11:22:47.304816 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 92 92 89 
2021.03.24 11:22:47.304823 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 92
2021.03.24 11:22:47.304832 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.304839 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.304851 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 92, starting_idx: 1, cur_nxt_idx: 93
2021.03.24 11:22:47.304862 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=92, LastLogTerm=1, EntriesLength=0, CommitIndex=92, Term=1, peer_last_sent_idx 92
2021.03.24 11:22:47.304870 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.304878 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.304925 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.304939 [ 1092 ] {} <Trace> RaftInstance: local log idx 92, target_commit_idx 92, quick_commit_index_ 92, state_->get_commit_idx() 91
2021.03.24 11:22:47.304946 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.305006 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.305065 [ 1096 ] {} <Debug> RaftInstance: commit upto 92, current idx 91
2021.03.24 11:22:47.305121 [ 1096 ] {} <Trace> RaftInstance: commit upto 92, current idx 92
2021.03.24 11:22:47.305178 [ 1096 ] {} <Debug> RaftInstance: notify cb 92 0x7faa1f09b820
2021.03.24 11:22:47.305202 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 92, current idx 92
2021.03.24 11:22:47.305211 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.305227 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.305246 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 92 wake up (12650 us), return value (nil)
2021.03.24 11:22:47.311446 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.311474 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=92
2021.03.24 11:22:47.311477 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.311489 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.311518 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 92
2021.03.24 11:22:47.311443 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.311541 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.311627 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 89, new matched idx: 91
2021.03.24 11:22:47.311641 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 92 92 91 
2021.03.24 11:22:47.311650 [ 1092 ] {} <Trace> RaftInstance: local log idx 93, target_commit_idx 92, quick_commit_index_ 92, state_->get_commit_idx() 92
2021.03.24 11:22:47.311659 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.311667 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.311676 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 91, starting_idx: 1, cur_nxt_idx: 93
2021.03.24 11:22:47.311688 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=91, LastLogTerm=1, EntriesLength=1, CommitIndex=92, Term=1, peer_last_sent_idx 88
2021.03.24 11:22:47.311695 [ 1092 ] {} <Debug> RaftInstance: idx: 92
2021.03.24 11:22:47.311711 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.311762 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.311802 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=93
2021.03.24 11:22:47.311610 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 93
2021.03.24 11:22:47.311821 [ 1173 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.311822 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.311888 [ 1173 ] {} <Debug> RaftInstance: append at log_idx 94
2021.03.24 11:22:47.311905 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 93
2021.03.24 11:22:47.311907 [ 1173 ] {} <Debug> RaftInstance: commit_ret_cv 94 0x7faa1f021220 sleep
2021.03.24 11:22:47.311859 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 93 0x7faaa6f93c20 sleep
2021.03.24 11:22:47.311924 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.311990 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 92, new matched idx: 92
2021.03.24 11:22:47.312010 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 94 92 91 
2021.03.24 11:22:47.312020 [ 1095 ] {} <Trace> RaftInstance: local log idx 94, target_commit_idx 92, quick_commit_index_ 92, state_->get_commit_idx() 92
2021.03.24 11:22:47.312028 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.312036 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.312048 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 92, starting_idx: 1, cur_nxt_idx: 95
2021.03.24 11:22:47.312057 [ 1095 ] {} <Debug> RaftInstance: last sent log (93) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.312067 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=92, LastLogTerm=1, EntriesLength=2, CommitIndex=92, Term=1, peer_last_sent_idx 93
2021.03.24 11:22:47.312078 [ 1095 ] {} <Debug> RaftInstance: idx range: 93-94
2021.03.24 11:22:47.312087 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.312130 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.312166 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.312180 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.312202 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.312211 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.313633 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.313673 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 95
2021.03.24 11:22:47.313694 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 95 0x7faa1f070820 sleep
2021.03.24 11:22:47.313723 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.313743 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.318854 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.318919 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=95
2021.03.24 11:22:47.318931 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.318954 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 95
2021.03.24 11:22:47.318962 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.318970 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 92, new matched idx: 94
2021.03.24 11:22:47.318986 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 95 94 91 
2021.03.24 11:22:47.318993 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 94
2021.03.24 11:22:47.319003 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.319023 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.319031 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 94, starting_idx: 1, cur_nxt_idx: 96
2021.03.24 11:22:47.319042 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=94, LastLogTerm=1, EntriesLength=1, CommitIndex=94, Term=1, peer_last_sent_idx 93
2021.03.24 11:22:47.319048 [ 1093 ] {} <Debug> RaftInstance: idx: 95
2021.03.24 11:22:47.319057 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.319133 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.319145 [ 1093 ] {} <Trace> RaftInstance: local log idx 95, target_commit_idx 94, quick_commit_index_ 94, state_->get_commit_idx() 92
2021.03.24 11:22:47.319153 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.319168 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.319193 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.319225 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.319249 [ 1096 ] {} <Debug> RaftInstance: commit upto 94, current idx 92
2021.03.24 11:22:47.319264 [ 1096 ] {} <Trace> RaftInstance: commit upto 94, current idx 93
2021.03.24 11:22:47.319323 [ 1096 ] {} <Debug> RaftInstance: notify cb 93 0x7faaa6f93c20
2021.03.24 11:22:47.319351 [ 1096 ] {} <Trace> RaftInstance: commit upto 94, current idx 94
2021.03.24 11:22:47.319371 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.319410 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 93 wake up (7516 us), return value (nil)
2021.03.24 11:22:47.319412 [ 1096 ] {} <Debug> RaftInstance: notify cb 94 0x7faa1f021220
2021.03.24 11:22:47.319457 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 94, current idx 94
2021.03.24 11:22:47.319467 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.319474 [ 1173 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.319495 [ 1173 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 94 wake up (7560 us), return value 0x7faaa9a69c40
2021.03.24 11:22:47.319574 [ 1174 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.319598 [ 1174 ] {} <Debug> RaftInstance: append at log_idx 96
2021.03.24 11:22:47.319626 [ 1174 ] {} <Debug> RaftInstance: commit_ret_cv 96 0x7faa1bb76020 sleep
2021.03.24 11:22:47.319643 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.319658 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.325963 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.325997 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=96
2021.03.24 11:22:47.326008 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.326019 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 96
2021.03.24 11:22:47.326027 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.326036 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 94, new matched idx: 95
2021.03.24 11:22:47.326047 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 96 95 91 
2021.03.24 11:22:47.326059 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 95
2021.03.24 11:22:47.326068 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.326076 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.326085 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 95, starting_idx: 1, cur_nxt_idx: 97
2021.03.24 11:22:47.326096 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=95, LastLogTerm=1, EntriesLength=1, CommitIndex=95, Term=1, peer_last_sent_idx 95
2021.03.24 11:22:47.326104 [ 1092 ] {} <Debug> RaftInstance: idx: 96
2021.03.24 11:22:47.326113 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.326172 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.326185 [ 1092 ] {} <Trace> RaftInstance: local log idx 96, target_commit_idx 95, quick_commit_index_ 95, state_->get_commit_idx() 94
2021.03.24 11:22:47.326195 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.326212 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.326221 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.326276 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.326325 [ 1096 ] {} <Debug> RaftInstance: commit upto 95, current idx 94
2021.03.24 11:22:47.326337 [ 1096 ] {} <Trace> RaftInstance: commit upto 95, current idx 95
2021.03.24 11:22:47.326384 [ 1096 ] {} <Debug> RaftInstance: notify cb 95 0x7faa1f070820
2021.03.24 11:22:47.326418 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 95, current idx 95
2021.03.24 11:22:47.326439 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.326482 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.326510 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 95 wake up (12748 us), return value (nil)
2021.03.24 11:22:47.332692 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.332719 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.332751 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=93
2021.03.24 11:22:47.332783 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.332801 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 93
2021.03.24 11:22:47.332810 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.332842 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 91, new matched idx: 92
2021.03.24 11:22:47.332854 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 96 95 92 
2021.03.24 11:22:47.332863 [ 1094 ] {} <Trace> RaftInstance: local log idx 96, target_commit_idx 95, quick_commit_index_ 95, state_->get_commit_idx() 95
2021.03.24 11:22:47.332871 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.332880 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.332888 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 92, starting_idx: 1, cur_nxt_idx: 97
2021.03.24 11:22:47.332901 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=92, LastLogTerm=1, EntriesLength=4, CommitIndex=95, Term=1, peer_last_sent_idx 92
2021.03.24 11:22:47.332908 [ 1094 ] {} <Debug> RaftInstance: idx range: 93-96
2021.03.24 11:22:47.332918 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.332989 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.333037 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=97
2021.03.24 11:22:47.333057 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.333068 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 97
2021.03.24 11:22:47.333092 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.333103 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 95, new matched idx: 96
2021.03.24 11:22:47.333113 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 96 96 92 
2021.03.24 11:22:47.333121 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 96
2021.03.24 11:22:47.333142 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.333150 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.333160 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 96, starting_idx: 1, cur_nxt_idx: 97
2021.03.24 11:22:47.333170 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=96, LastLogTerm=1, EntriesLength=0, CommitIndex=96, Term=1, peer_last_sent_idx 96
2021.03.24 11:22:47.333177 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.333185 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.333239 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.333253 [ 1092 ] {} <Trace> RaftInstance: local log idx 96, target_commit_idx 96, quick_commit_index_ 96, state_->get_commit_idx() 95
2021.03.24 11:22:47.333261 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.333303 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.333324 [ 1096 ] {} <Debug> RaftInstance: commit upto 96, current idx 95
2021.03.24 11:22:47.333335 [ 1096 ] {} <Trace> RaftInstance: commit upto 96, current idx 96
2021.03.24 11:22:47.333362 [ 1096 ] {} <Debug> RaftInstance: notify cb 96 0x7faa1bb76020
2021.03.24 11:22:47.333383 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 96, current idx 96
2021.03.24 11:22:47.333392 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.333440 [ 1174 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.333462 [ 1174 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 96 wake up (13816 us), return value 0x7faaa9a69c50
2021.03.24 11:22:47.333540 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.333571 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 97
2021.03.24 11:22:47.333589 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 97 0x7faa0daae120 sleep
2021.03.24 11:22:47.333620 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.333631 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.334981 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.335051 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 98
2021.03.24 11:22:47.335085 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 98 0x7faa1f070820 sleep
2021.03.24 11:22:47.335100 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.335117 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.339331 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.339379 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 99
2021.03.24 11:22:47.339436 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 99 0x7faa1f09b620 sleep
2021.03.24 11:22:47.339456 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.339476 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.339877 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.339896 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=97
2021.03.24 11:22:47.339906 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.339919 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 97
2021.03.24 11:22:47.339928 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.339936 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 96, new matched idx: 96
2021.03.24 11:22:47.339946 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 99 96 92 
2021.03.24 11:22:47.339956 [ 1092 ] {} <Trace> RaftInstance: local log idx 99, target_commit_idx 96, quick_commit_index_ 96, state_->get_commit_idx() 96
2021.03.24 11:22:47.339964 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.339972 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.339980 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 96, starting_idx: 1, cur_nxt_idx: 100
2021.03.24 11:22:47.339988 [ 1092 ] {} <Debug> RaftInstance: last sent log (97) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.339999 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=96, LastLogTerm=1, EntriesLength=3, CommitIndex=96, Term=1, peer_last_sent_idx 97
2021.03.24 11:22:47.340007 [ 1092 ] {} <Debug> RaftInstance: idx range: 97-99
2021.03.24 11:22:47.340018 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.340079 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.346821 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.346868 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=100
2021.03.24 11:22:47.346879 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.346890 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 100
2021.03.24 11:22:47.346914 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.346931 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 96, new matched idx: 99
2021.03.24 11:22:47.346949 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 99 99 92 
2021.03.24 11:22:47.346964 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 99
2021.03.24 11:22:47.346973 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.346981 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.347021 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 99, starting_idx: 1, cur_nxt_idx: 100
2021.03.24 11:22:47.347033 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=99, LastLogTerm=1, EntriesLength=0, CommitIndex=99, Term=1, peer_last_sent_idx 97
2021.03.24 11:22:47.347040 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.347049 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.347114 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.347127 [ 1092 ] {} <Trace> RaftInstance: local log idx 99, target_commit_idx 99, quick_commit_index_ 99, state_->get_commit_idx() 96
2021.03.24 11:22:47.347135 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.347211 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.347261 [ 1096 ] {} <Debug> RaftInstance: commit upto 99, current idx 96
2021.03.24 11:22:47.347274 [ 1096 ] {} <Trace> RaftInstance: commit upto 99, current idx 97
2021.03.24 11:22:47.347369 [ 1096 ] {} <Debug> RaftInstance: notify cb 97 0x7faa0daae120
2021.03.24 11:22:47.347391 [ 1096 ] {} <Trace> RaftInstance: commit upto 99, current idx 98
2021.03.24 11:22:47.347414 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.347432 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 97 wake up (13825 us), return value (nil)
2021.03.24 11:22:47.347461 [ 1096 ] {} <Debug> RaftInstance: notify cb 98 0x7faa1f070820
2021.03.24 11:22:47.347471 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.347480 [ 1096 ] {} <Trace> RaftInstance: commit upto 99, current idx 99
2021.03.24 11:22:47.347496 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 100
2021.03.24 11:22:47.347501 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.347535 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 98 wake up (12428 us), return value (nil)
2021.03.24 11:22:47.347542 [ 1096 ] {} <Debug> RaftInstance: notify cb 99 0x7faa1f09b620
2021.03.24 11:22:47.347571 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 99, current idx 99
2021.03.24 11:22:47.347580 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.347588 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 100 0x7faa0daae020 sleep
2021.03.24 11:22:47.347603 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.347611 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.347646 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.347633 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 99 wake up (8170 us), return value (nil)
2021.03.24 11:22:47.353520 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.353561 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=97
2021.03.24 11:22:47.353571 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.353582 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 97
2021.03.24 11:22:47.353590 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.353598 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 92, new matched idx: 96
2021.03.24 11:22:47.353609 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 100 99 96 
2021.03.24 11:22:47.353618 [ 1092 ] {} <Trace> RaftInstance: local log idx 100, target_commit_idx 99, quick_commit_index_ 99, state_->get_commit_idx() 99
2021.03.24 11:22:47.353626 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.353650 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.353658 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 96, starting_idx: 1, cur_nxt_idx: 101
2021.03.24 11:22:47.353674 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=96, LastLogTerm=1, EntriesLength=4, CommitIndex=99, Term=1, peer_last_sent_idx 93
2021.03.24 11:22:47.353682 [ 1092 ] {} <Debug> RaftInstance: idx range: 97-100
2021.03.24 11:22:47.353691 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.353758 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.353785 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.353796 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=100
2021.03.24 11:22:47.353804 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.353813 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 100
2021.03.24 11:22:47.353820 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.353830 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 99, new matched idx: 99
2021.03.24 11:22:47.353839 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 100 99 96 
2021.03.24 11:22:47.353847 [ 1092 ] {} <Trace> RaftInstance: local log idx 100, target_commit_idx 99, quick_commit_index_ 99, state_->get_commit_idx() 99
2021.03.24 11:22:47.353855 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.353862 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.353870 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 99, starting_idx: 1, cur_nxt_idx: 101
2021.03.24 11:22:47.353880 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=99, LastLogTerm=1, EntriesLength=1, CommitIndex=99, Term=1, peer_last_sent_idx 100
2021.03.24 11:22:47.353887 [ 1092 ] {} <Debug> RaftInstance: idx: 100
2021.03.24 11:22:47.353895 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.353933 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.355769 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.355802 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 101
2021.03.24 11:22:47.355823 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 101 0x7faa1f070820 sleep
2021.03.24 11:22:47.355877 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.355915 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.360625 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.360671 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=101
2021.03.24 11:22:47.360681 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.360691 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 101
2021.03.24 11:22:47.360704 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.360712 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 99, new matched idx: 100
2021.03.24 11:22:47.360723 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 101 100 96 
2021.03.24 11:22:47.360731 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 100
2021.03.24 11:22:47.360740 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.360747 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.360756 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 100, starting_idx: 1, cur_nxt_idx: 102
2021.03.24 11:22:47.360767 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=100, LastLogTerm=1, EntriesLength=1, CommitIndex=100, Term=1, peer_last_sent_idx 100
2021.03.24 11:22:47.360792 [ 1093 ] {} <Debug> RaftInstance: idx: 101
2021.03.24 11:22:47.360804 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.360868 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.360881 [ 1093 ] {} <Trace> RaftInstance: local log idx 101, target_commit_idx 100, quick_commit_index_ 100, state_->get_commit_idx() 99
2021.03.24 11:22:47.360890 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.360905 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.360915 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.360928 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.360950 [ 1096 ] {} <Debug> RaftInstance: commit upto 100, current idx 99
2021.03.24 11:22:47.360966 [ 1096 ] {} <Trace> RaftInstance: commit upto 100, current idx 100
2021.03.24 11:22:47.361054 [ 1096 ] {} <Debug> RaftInstance: notify cb 100 0x7faa0daae020
2021.03.24 11:22:47.361103 [ 1096 ] {} <Information> RaftInstance: creating a snapshot for index 100
2021.03.24 11:22:47.361118 [ 1096 ] {} <Information> RaftInstance: create snapshot idx 100 log_term 1
2021.03.24 11:22:47.361130 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.361149 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 100 wake up (13612 us), return value (nil)
2021.03.24 11:22:47.361178 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.361193 [ 1096 ] {} <Information> RaftInstance: create snapshot idx 100 log_term 1 done: 65 us elapsed
2021.03.24 11:22:47.361215 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 100, current idx 100
2021.03.24 11:22:47.361227 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.361200 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 102
2021.03.24 11:22:47.361265 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 102 0x7faa0daae120 sleep
2021.03.24 11:22:47.361282 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.361298 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.367497 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.367792 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.374433 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.387256 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=102
2021.03.24 11:22:47.387301 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.387314 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 102
2021.03.24 11:22:47.387323 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.387331 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 100, new matched idx: 101
2021.03.24 11:22:47.387366 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 102 101 96 
2021.03.24 11:22:47.387376 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 101
2021.03.24 11:22:47.387385 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.387394 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 96, starting_idx: 1, cur_nxt_idx: 103
2021.03.24 11:22:47.387402 [ 1092 ] {} <Debug> RaftInstance: last sent log (97) to peer 3 is not applied, cnt 1
2021.03.24 11:22:47.387438 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=96, LastLogTerm=1, EntriesLength=6, CommitIndex=101, Term=1, peer_last_sent_idx 97
2021.03.24 11:22:47.387446 [ 1092 ] {} <Debug> RaftInstance: idx range: 97-102
2021.03.24 11:22:47.387456 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.387578 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.387591 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.387599 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 101, starting_idx: 1, cur_nxt_idx: 103
2021.03.24 11:22:47.387609 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=101, LastLogTerm=1, EntriesLength=1, CommitIndex=101, Term=1, peer_last_sent_idx 101
2021.03.24 11:22:47.387616 [ 1092 ] {} <Debug> RaftInstance: idx: 102
2021.03.24 11:22:47.387625 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.387657 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.387668 [ 1092 ] {} <Trace> RaftInstance: local log idx 103, target_commit_idx 101, quick_commit_index_ 101, state_->get_commit_idx() 100
2021.03.24 11:22:47.387676 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.387694 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.387703 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.387731 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.387747 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 103
2021.03.24 11:22:47.387756 [ 1096 ] {} <Debug> RaftInstance: commit upto 101, current idx 100
2021.03.24 11:22:47.387758 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=101
2021.03.24 11:22:47.387797 [ 1096 ] {} <Trace> RaftInstance: commit upto 101, current idx 101
2021.03.24 11:22:47.387804 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.387823 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 101
2021.03.24 11:22:47.387827 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 103 0x7faaa6f93920 sleep
2021.03.24 11:22:47.387845 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.387858 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 96, new matched idx: 100
2021.03.24 11:22:47.387887 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 103 101 100 
2021.03.24 11:22:47.387896 [ 1093 ] {} <Trace> RaftInstance: local log idx 103, target_commit_idx 101, quick_commit_index_ 101, state_->get_commit_idx() 100
2021.03.24 11:22:47.387909 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.387898 [ 1096 ] {} <Debug> RaftInstance: notify cb 101 0x7faa1f070820
2021.03.24 11:22:47.387918 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.387943 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 101, current idx 101
2021.03.24 11:22:47.387951 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.387973 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.387986 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.387987 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.388022 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 101 wake up (32162 us), return value (nil)
2021.03.24 11:22:47.387953 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.392825 [ 1103 ] {} <Debug> RaftInstance: snapshot created, compact the log store
2021.03.24 11:22:47.392835 [ 1103 ] {} <Debug> RaftInstance: log_store_ compact upto 90
2021.03.24 11:22:47.394538 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.394591 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=103
2021.03.24 11:22:47.394603 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.394614 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 103
2021.03.24 11:22:47.394627 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.394635 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 101, new matched idx: 102
2021.03.24 11:22:47.394648 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 103 102 100 
2021.03.24 11:22:47.394656 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 102
2021.03.24 11:22:47.394665 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.394673 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.394682 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 102, starting_idx: 91, cur_nxt_idx: 104
2021.03.24 11:22:47.394694 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=102, LastLogTerm=1, EntriesLength=1, CommitIndex=102, Term=1, peer_last_sent_idx 102
2021.03.24 11:22:47.394701 [ 1092 ] {} <Debug> RaftInstance: idx: 103
2021.03.24 11:22:47.394711 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.394810 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.394826 [ 1092 ] {} <Trace> RaftInstance: local log idx 103, target_commit_idx 102, quick_commit_index_ 102, state_->get_commit_idx() 101
2021.03.24 11:22:47.394834 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.394852 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.394861 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.394881 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.394904 [ 1096 ] {} <Debug> RaftInstance: commit upto 102, current idx 101
2021.03.24 11:22:47.394915 [ 1096 ] {} <Trace> RaftInstance: commit upto 102, current idx 102
2021.03.24 11:22:47.394983 [ 1096 ] {} <Debug> RaftInstance: notify cb 102 0x7faa0daae120
2021.03.24 11:22:47.395005 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 102, current idx 102
2021.03.24 11:22:47.395014 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.395055 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.395073 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 102 wake up (33792 us), return value (nil)
2021.03.24 11:22:47.395104 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.395127 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 104
2021.03.24 11:22:47.395148 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 104 0x7faa0daae020 sleep
2021.03.24 11:22:47.395174 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.395200 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.396272 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.396303 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 105
2021.03.24 11:22:47.396328 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 105 0x7faa1f057320 sleep
2021.03.24 11:22:47.396364 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.396382 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.401695 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.401724 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=104
2021.03.24 11:22:47.401734 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.401745 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 104
2021.03.24 11:22:47.401753 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.401761 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 102, new matched idx: 103
2021.03.24 11:22:47.401781 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 105 103 100 
2021.03.24 11:22:47.401789 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 103
2021.03.24 11:22:47.401798 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.401806 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.401815 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 103, starting_idx: 91, cur_nxt_idx: 106
2021.03.24 11:22:47.401826 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=103, LastLogTerm=1, EntriesLength=2, CommitIndex=103, Term=1, peer_last_sent_idx 103
2021.03.24 11:22:47.401833 [ 1095 ] {} <Debug> RaftInstance: idx range: 104-105
2021.03.24 11:22:47.401843 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.401895 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.401913 [ 1095 ] {} <Trace> RaftInstance: local log idx 105, target_commit_idx 103, quick_commit_index_ 103, state_->get_commit_idx() 102
2021.03.24 11:22:47.401921 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.401938 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.401947 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.401997 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.402020 [ 1096 ] {} <Debug> RaftInstance: commit upto 103, current idx 102
2021.03.24 11:22:47.402037 [ 1096 ] {} <Trace> RaftInstance: commit upto 103, current idx 103
2021.03.24 11:22:47.402077 [ 1096 ] {} <Debug> RaftInstance: notify cb 103 0x7faaa6f93920
2021.03.24 11:22:47.402098 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 103, current idx 103
2021.03.24 11:22:47.402113 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.402174 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.402232 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 103 wake up (14380 us), return value (nil)
2021.03.24 11:22:47.408021 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.408053 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=103
2021.03.24 11:22:47.408062 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.408073 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 103
2021.03.24 11:22:47.408080 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.408088 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 100, new matched idx: 102
2021.03.24 11:22:47.408099 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 105 103 102 
2021.03.24 11:22:47.408108 [ 1095 ] {} <Trace> RaftInstance: local log idx 105, target_commit_idx 103, quick_commit_index_ 103, state_->get_commit_idx() 103
2021.03.24 11:22:47.408116 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.408124 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.408133 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 102, starting_idx: 91, cur_nxt_idx: 106
2021.03.24 11:22:47.408145 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=102, LastLogTerm=1, EntriesLength=3, CommitIndex=103, Term=1, peer_last_sent_idx 97
2021.03.24 11:22:47.408169 [ 1095 ] {} <Debug> RaftInstance: idx range: 103-105
2021.03.24 11:22:47.408179 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.408231 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.408719 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.408773 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=106
2021.03.24 11:22:47.408785 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.408797 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 106
2021.03.24 11:22:47.408806 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.408814 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 103, new matched idx: 105
2021.03.24 11:22:47.408828 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 105 105 102 
2021.03.24 11:22:47.408836 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 105
2021.03.24 11:22:47.408845 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.408854 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.408862 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 105, starting_idx: 91, cur_nxt_idx: 106
2021.03.24 11:22:47.408873 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=105, LastLogTerm=1, EntriesLength=0, CommitIndex=105, Term=1, peer_last_sent_idx 104
2021.03.24 11:22:47.408881 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.408889 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.408965 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.408979 [ 1092 ] {} <Trace> RaftInstance: local log idx 105, target_commit_idx 105, quick_commit_index_ 105, state_->get_commit_idx() 103
2021.03.24 11:22:47.408987 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.409060 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.409105 [ 1096 ] {} <Debug> RaftInstance: commit upto 105, current idx 103
2021.03.24 11:22:47.409118 [ 1096 ] {} <Trace> RaftInstance: commit upto 105, current idx 104
2021.03.24 11:22:47.409195 [ 1096 ] {} <Debug> RaftInstance: notify cb 104 0x7faa0daae020
2021.03.24 11:22:47.409213 [ 1096 ] {} <Trace> RaftInstance: commit upto 105, current idx 105
2021.03.24 11:22:47.409233 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.409254 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 104 wake up (14088 us), return value (nil)
2021.03.24 11:22:47.409264 [ 1096 ] {} <Debug> RaftInstance: notify cb 105 0x7faa1f057320
2021.03.24 11:22:47.409288 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 105, current idx 105
2021.03.24 11:22:47.409306 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.409291 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.409325 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.409340 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 106
2021.03.24 11:22:47.409346 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 105 wake up (13003 us), return value 0x7faaa9a69c60
2021.03.24 11:22:47.409363 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 106 0x7faa0daae120 sleep
2021.03.24 11:22:47.409404 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.409421 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.415723 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.415748 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=106
2021.03.24 11:22:47.415758 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.415768 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 106
2021.03.24 11:22:47.415777 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.415784 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 105, new matched idx: 105
2021.03.24 11:22:47.415794 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 106 105 102 
2021.03.24 11:22:47.415803 [ 1095 ] {} <Trace> RaftInstance: local log idx 106, target_commit_idx 105, quick_commit_index_ 105, state_->get_commit_idx() 105
2021.03.24 11:22:47.415811 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.415822 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.415830 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 105, starting_idx: 91, cur_nxt_idx: 107
2021.03.24 11:22:47.415841 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=105, LastLogTerm=1, EntriesLength=1, CommitIndex=105, Term=1, peer_last_sent_idx 106
2021.03.24 11:22:47.415848 [ 1095 ] {} <Debug> RaftInstance: idx: 106
2021.03.24 11:22:47.415856 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.415904 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.422187 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.422259 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 107
2021.03.24 11:22:47.422282 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 107 0x7faaa6f93920 sleep
2021.03.24 11:22:47.422323 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.422349 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.422481 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.422516 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=107
2021.03.24 11:22:47.422527 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.422537 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 107
2021.03.24 11:22:47.422545 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.422554 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 105, new matched idx: 106
2021.03.24 11:22:47.422569 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 107 106 102 
2021.03.24 11:22:47.422576 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 106
2021.03.24 11:22:47.422584 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.422592 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.422601 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 106, starting_idx: 91, cur_nxt_idx: 108
2021.03.24 11:22:47.422612 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=106, LastLogTerm=1, EntriesLength=1, CommitIndex=106, Term=1, peer_last_sent_idx 106
2021.03.24 11:22:47.422619 [ 1093 ] {} <Debug> RaftInstance: idx: 107
2021.03.24 11:22:47.422628 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.422687 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.422700 [ 1093 ] {} <Trace> RaftInstance: local log idx 107, target_commit_idx 106, quick_commit_index_ 106, state_->get_commit_idx() 105
2021.03.24 11:22:47.422708 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.422729 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.422756 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.422798 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.422820 [ 1096 ] {} <Debug> RaftInstance: commit upto 106, current idx 105
2021.03.24 11:22:47.422831 [ 1096 ] {} <Trace> RaftInstance: commit upto 106, current idx 106
2021.03.24 11:22:47.422904 [ 1096 ] {} <Debug> RaftInstance: notify cb 106 0x7faa0daae120
2021.03.24 11:22:47.422924 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 106, current idx 106
2021.03.24 11:22:47.422938 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.423016 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.423030 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 106 wake up (13658 us), return value (nil)
2021.03.24 11:22:47.423063 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.423079 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 108
2021.03.24 11:22:47.423093 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 108 0x7faa0daae020 sleep
2021.03.24 11:22:47.423115 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.423126 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.425316 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.425350 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 109
2021.03.24 11:22:47.425376 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 109 0x7faa1f09b820 sleep
2021.03.24 11:22:47.425416 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.425441 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.428625 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.428661 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=106
2021.03.24 11:22:47.428672 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.428683 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 106
2021.03.24 11:22:47.428691 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.428700 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 102, new matched idx: 105
2021.03.24 11:22:47.428711 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 109 106 105 
2021.03.24 11:22:47.428721 [ 1092 ] {} <Trace> RaftInstance: local log idx 109, target_commit_idx 106, quick_commit_index_ 106, state_->get_commit_idx() 106
2021.03.24 11:22:47.428729 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.428737 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.428747 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 105, starting_idx: 91, cur_nxt_idx: 110
2021.03.24 11:22:47.428759 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=105, LastLogTerm=1, EntriesLength=4, CommitIndex=106, Term=1, peer_last_sent_idx 103
2021.03.24 11:22:47.428767 [ 1092 ] {} <Debug> RaftInstance: idx range: 106-109
2021.03.24 11:22:47.428776 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.428851 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.429694 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.429712 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=108
2021.03.24 11:22:47.429721 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.429730 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 108
2021.03.24 11:22:47.429758 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.429767 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 106, new matched idx: 107
2021.03.24 11:22:47.429776 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 109 107 105 
2021.03.24 11:22:47.429784 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 107
2021.03.24 11:22:47.429792 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.429800 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.429808 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 107, starting_idx: 91, cur_nxt_idx: 110
2021.03.24 11:22:47.429818 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=107, LastLogTerm=1, EntriesLength=2, CommitIndex=107, Term=1, peer_last_sent_idx 107
2021.03.24 11:22:47.429825 [ 1092 ] {} <Debug> RaftInstance: idx range: 108-109
2021.03.24 11:22:47.429834 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.429874 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.429887 [ 1092 ] {} <Trace> RaftInstance: local log idx 109, target_commit_idx 107, quick_commit_index_ 107, state_->get_commit_idx() 106
2021.03.24 11:22:47.429895 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.429914 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.429923 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.429953 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.429974 [ 1096 ] {} <Debug> RaftInstance: commit upto 107, current idx 106
2021.03.24 11:22:47.430013 [ 1096 ] {} <Trace> RaftInstance: commit upto 107, current idx 107
2021.03.24 11:22:47.430043 [ 1096 ] {} <Debug> RaftInstance: notify cb 107 0x7faaa6f93920
2021.03.24 11:22:47.430063 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 107, current idx 107
2021.03.24 11:22:47.430072 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.430108 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.430128 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 107 wake up (7828 us), return value 0x7faaa9a69c70
2021.03.24 11:22:47.436738 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.436774 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=110
2021.03.24 11:22:47.436784 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.436795 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 110
2021.03.24 11:22:47.436803 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.436811 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 107, new matched idx: 109
2021.03.24 11:22:47.436823 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 109 109 105 
2021.03.24 11:22:47.436835 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 109
2021.03.24 11:22:47.436845 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.436853 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.436861 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 109, starting_idx: 91, cur_nxt_idx: 110
2021.03.24 11:22:47.436871 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=109, LastLogTerm=1, EntriesLength=0, CommitIndex=109, Term=1, peer_last_sent_idx 108
2021.03.24 11:22:47.436878 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.436886 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.436937 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.436966 [ 1093 ] {} <Trace> RaftInstance: local log idx 109, target_commit_idx 109, quick_commit_index_ 109, state_->get_commit_idx() 107
2021.03.24 11:22:47.436974 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.437034 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.437071 [ 1096 ] {} <Debug> RaftInstance: commit upto 109, current idx 107
2021.03.24 11:22:47.437115 [ 1096 ] {} <Trace> RaftInstance: commit upto 109, current idx 108
2021.03.24 11:22:47.437219 [ 1096 ] {} <Debug> RaftInstance: notify cb 108 0x7faa0daae020
2021.03.24 11:22:47.437250 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.437262 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 108 wake up (14156 us), return value (nil)
2021.03.24 11:22:47.437296 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.437320 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 110
2021.03.24 11:22:47.437336 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 110 0x7faa0daae120 sleep
2021.03.24 11:22:47.437351 [ 1096 ] {} <Trace> RaftInstance: commit upto 109, current idx 109
2021.03.24 11:22:47.437357 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.437374 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.437378 [ 1096 ] {} <Debug> RaftInstance: notify cb 109 0x7faa1f09b820
2021.03.24 11:22:47.437398 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 109, current idx 109
2021.03.24 11:22:47.437407 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.437421 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.437442 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 109 wake up (12052 us), return value (nil)
2021.03.24 11:22:47.443617 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.443666 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=110
2021.03.24 11:22:47.443676 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.443687 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 110
2021.03.24 11:22:47.443695 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.443704 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 109, new matched idx: 109
2021.03.24 11:22:47.443716 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 110 109 105 
2021.03.24 11:22:47.443725 [ 1092 ] {} <Trace> RaftInstance: local log idx 110, target_commit_idx 109, quick_commit_index_ 109, state_->get_commit_idx() 109
2021.03.24 11:22:47.443733 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.443741 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.443750 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 109, starting_idx: 91, cur_nxt_idx: 111
2021.03.24 11:22:47.443762 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=109, LastLogTerm=1, EntriesLength=1, CommitIndex=109, Term=1, peer_last_sent_idx 110
2021.03.24 11:22:47.443769 [ 1092 ] {} <Debug> RaftInstance: idx: 110
2021.03.24 11:22:47.443778 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.443851 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.449544 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.449588 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=110
2021.03.24 11:22:47.449615 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.449626 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 110
2021.03.24 11:22:47.449635 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.449643 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 105, new matched idx: 109
2021.03.24 11:22:47.449655 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 110 109 109 
2021.03.24 11:22:47.449663 [ 1093 ] {} <Trace> RaftInstance: local log idx 110, target_commit_idx 109, quick_commit_index_ 109, state_->get_commit_idx() 109
2021.03.24 11:22:47.449671 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.449680 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.449688 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 109, starting_idx: 91, cur_nxt_idx: 111
2021.03.24 11:22:47.449700 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=109, LastLogTerm=1, EntriesLength=1, CommitIndex=109, Term=1, peer_last_sent_idx 106
2021.03.24 11:22:47.449711 [ 1093 ] {} <Debug> RaftInstance: idx: 110
2021.03.24 11:22:47.449720 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.449785 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.449925 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.449975 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 111
2021.03.24 11:22:47.450008 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.450019 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.450031 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 111 0x7faaa6f93920 sleep
2021.03.24 11:22:47.450496 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.450515 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=111
2021.03.24 11:22:47.450524 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.450534 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 111
2021.03.24 11:22:47.450542 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.450577 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 109, new matched idx: 110
2021.03.24 11:22:47.450590 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 111 110 109 
2021.03.24 11:22:47.450598 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 110
2021.03.24 11:22:47.450606 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.450614 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.450622 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 110, starting_idx: 91, cur_nxt_idx: 112
2021.03.24 11:22:47.450633 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=110, LastLogTerm=1, EntriesLength=1, CommitIndex=110, Term=1, peer_last_sent_idx 110
2021.03.24 11:22:47.450640 [ 1092 ] {} <Debug> RaftInstance: idx: 111
2021.03.24 11:22:47.450649 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.450711 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.450724 [ 1092 ] {} <Trace> RaftInstance: local log idx 111, target_commit_idx 110, quick_commit_index_ 110, state_->get_commit_idx() 109
2021.03.24 11:22:47.450732 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.450747 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.450756 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.450770 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.450809 [ 1096 ] {} <Debug> RaftInstance: commit upto 110, current idx 109
2021.03.24 11:22:47.450840 [ 1096 ] {} <Trace> RaftInstance: commit upto 110, current idx 110
2021.03.24 11:22:47.450920 [ 1096 ] {} <Debug> RaftInstance: notify cb 110 0x7faa0daae120
2021.03.24 11:22:47.450950 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.450962 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 110 wake up (13616 us), return value (nil)
2021.03.24 11:22:47.450986 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.451001 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 112
2021.03.24 11:22:47.451020 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 112 0x7faa0daae020 sleep
2021.03.24 11:22:47.451036 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 110, current idx 110
2021.03.24 11:22:47.451048 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.451049 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.451092 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.452261 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.452287 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 113
2021.03.24 11:22:47.452324 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.452360 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.452329 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 113 0x7faa1f09b820 sleep
2021.03.24 11:22:47.457523 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.457554 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=112
2021.03.24 11:22:47.457564 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.457573 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 112
2021.03.24 11:22:47.457582 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.457590 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 110, new matched idx: 111
2021.03.24 11:22:47.457604 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 113 111 109 
2021.03.24 11:22:47.457612 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 111
2021.03.24 11:22:47.457620 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.457628 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.457636 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 111, starting_idx: 91, cur_nxt_idx: 114
2021.03.24 11:22:47.457648 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=111, LastLogTerm=1, EntriesLength=2, CommitIndex=111, Term=1, peer_last_sent_idx 111
2021.03.24 11:22:47.457655 [ 1093 ] {} <Debug> RaftInstance: idx range: 112-113
2021.03.24 11:22:47.457664 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.457714 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.457727 [ 1093 ] {} <Trace> RaftInstance: local log idx 113, target_commit_idx 111, quick_commit_index_ 111, state_->get_commit_idx() 110
2021.03.24 11:22:47.457738 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.457755 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.457764 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.457789 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.457832 [ 1096 ] {} <Debug> RaftInstance: commit upto 111, current idx 110
2021.03.24 11:22:47.457844 [ 1096 ] {} <Trace> RaftInstance: commit upto 111, current idx 111
2021.03.24 11:22:47.457895 [ 1096 ] {} <Debug> RaftInstance: notify cb 111 0x7faaa6f93920
2021.03.24 11:22:47.457917 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 111, current idx 111
2021.03.24 11:22:47.457927 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.457942 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.457963 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 111 wake up (7945 us), return value 0x7faaa9a69c80
2021.03.24 11:22:47.464374 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.464416 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=114
2021.03.24 11:22:47.464427 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.464438 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 114
2021.03.24 11:22:47.464446 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.464455 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 111, new matched idx: 113
2021.03.24 11:22:47.464466 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 113 113 109 
2021.03.24 11:22:47.464474 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 113
2021.03.24 11:22:47.464483 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.464491 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.464499 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 113, starting_idx: 91, cur_nxt_idx: 114
2021.03.24 11:22:47.464510 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=113, LastLogTerm=1, EntriesLength=0, CommitIndex=113, Term=1, peer_last_sent_idx 112
2021.03.24 11:22:47.464517 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.464526 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.464587 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.464600 [ 1092 ] {} <Trace> RaftInstance: local log idx 113, target_commit_idx 113, quick_commit_index_ 113, state_->get_commit_idx() 111
2021.03.24 11:22:47.464607 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.464650 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.464670 [ 1096 ] {} <Debug> RaftInstance: commit upto 113, current idx 111
2021.03.24 11:22:47.464680 [ 1096 ] {} <Trace> RaftInstance: commit upto 113, current idx 112
2021.03.24 11:22:47.464758 [ 1096 ] {} <Debug> RaftInstance: notify cb 112 0x7faa0daae020
2021.03.24 11:22:47.464789 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.464801 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 112 wake up (13775 us), return value (nil)
2021.03.24 11:22:47.464821 [ 1096 ] {} <Trace> RaftInstance: commit upto 113, current idx 113
2021.03.24 11:22:47.464872 [ 1096 ] {} <Debug> RaftInstance: notify cb 113 0x7faa1f09b820
2021.03.24 11:22:47.464890 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 113, current idx 113
2021.03.24 11:22:47.464899 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.464941 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.464963 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 113 wake up (12610 us), return value (nil)
2021.03.24 11:22:47.470340 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.470370 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=111
2021.03.24 11:22:47.470395 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.470405 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 111
2021.03.24 11:22:47.470413 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.470421 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 109, new matched idx: 110
2021.03.24 11:22:47.470431 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 113 113 110 
2021.03.24 11:22:47.470440 [ 1093 ] {} <Trace> RaftInstance: local log idx 113, target_commit_idx 113, quick_commit_index_ 113, state_->get_commit_idx() 113
2021.03.24 11:22:47.470448 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.470456 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.470464 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 110, starting_idx: 91, cur_nxt_idx: 114
2021.03.24 11:22:47.470475 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=110, LastLogTerm=1, EntriesLength=3, CommitIndex=113, Term=1, peer_last_sent_idx 110
2021.03.24 11:22:47.470482 [ 1093 ] {} <Debug> RaftInstance: idx range: 111-113
2021.03.24 11:22:47.470491 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.470544 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.471242 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.471285 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=114
2021.03.24 11:22:47.471296 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.471307 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 114
2021.03.24 11:22:47.471315 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.471323 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 113, new matched idx: 113
2021.03.24 11:22:47.471335 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 113 113 110 
2021.03.24 11:22:47.471344 [ 1094 ] {} <Trace> RaftInstance: local log idx 113, target_commit_idx 113, quick_commit_index_ 113, state_->get_commit_idx() 113
2021.03.24 11:22:47.473324 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.473376 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 114
2021.03.24 11:22:47.473407 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.473417 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.473427 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 113, starting_idx: 91, cur_nxt_idx: 115
2021.03.24 11:22:47.473439 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=113, LastLogTerm=1, EntriesLength=1, CommitIndex=113, Term=1, peer_last_sent_idx 114
2021.03.24 11:22:47.473447 [ 1097 ] {} <Debug> RaftInstance: idx: 114
2021.03.24 11:22:47.473457 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.473523 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.473541 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 114 0x7faa1f09b720 sleep
2021.03.24 11:22:47.477779 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.477813 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 115
2021.03.24 11:22:47.477845 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 115 0x7faa1f09b620 sleep
2021.03.24 11:22:47.477863 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.477900 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.480131 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.480178 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=115
2021.03.24 11:22:47.480190 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.480200 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 115
2021.03.24 11:22:47.480208 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.480215 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 113, new matched idx: 114
2021.03.24 11:22:47.480227 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 115 114 110 
2021.03.24 11:22:47.480235 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 114
2021.03.24 11:22:47.480244 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.480251 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.480259 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 114, starting_idx: 91, cur_nxt_idx: 116
2021.03.24 11:22:47.480270 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=114, LastLogTerm=1, EntriesLength=1, CommitIndex=114, Term=1, peer_last_sent_idx 114
2021.03.24 11:22:47.480277 [ 1092 ] {} <Debug> RaftInstance: idx: 115
2021.03.24 11:22:47.480286 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.480349 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.480362 [ 1092 ] {} <Trace> RaftInstance: local log idx 115, target_commit_idx 114, quick_commit_index_ 114, state_->get_commit_idx() 113
2021.03.24 11:22:47.480370 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.480387 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.480397 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.480419 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.480439 [ 1096 ] {} <Debug> RaftInstance: commit upto 114, current idx 113
2021.03.24 11:22:47.480450 [ 1096 ] {} <Trace> RaftInstance: commit upto 114, current idx 114
2021.03.24 11:22:47.480478 [ 1096 ] {} <Debug> RaftInstance: notify cb 114 0x7faa1f09b720
2021.03.24 11:22:47.480498 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 114, current idx 114
2021.03.24 11:22:47.480506 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.480517 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.480532 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 114 wake up (7123 us), return value 0x7faaa9a69c90
2021.03.24 11:22:47.483904 [ 1175 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.483927 [ 1175 ] {} <Debug> RaftInstance: append at log_idx 116
2021.03.24 11:22:47.483955 [ 1175 ] {} <Debug> RaftInstance: commit_ret_cv 116 0x7faa07c4f020 sleep
2021.03.24 11:22:47.483994 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.484023 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.487152 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.487186 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=116
2021.03.24 11:22:47.487197 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.487208 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 116
2021.03.24 11:22:47.487237 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.487246 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 114, new matched idx: 115
2021.03.24 11:22:47.487258 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 116 115 110 
2021.03.24 11:22:47.487266 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 115
2021.03.24 11:22:47.487275 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.487283 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.487291 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 115, starting_idx: 91, cur_nxt_idx: 117
2021.03.24 11:22:47.487304 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=115, LastLogTerm=1, EntriesLength=1, CommitIndex=115, Term=1, peer_last_sent_idx 115
2021.03.24 11:22:47.487311 [ 1095 ] {} <Debug> RaftInstance: idx: 116
2021.03.24 11:22:47.487320 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.487382 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.487395 [ 1095 ] {} <Trace> RaftInstance: local log idx 116, target_commit_idx 115, quick_commit_index_ 115, state_->get_commit_idx() 114
2021.03.24 11:22:47.487403 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.487419 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.487429 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.487438 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.487457 [ 1096 ] {} <Debug> RaftInstance: commit upto 115, current idx 114
2021.03.24 11:22:47.487468 [ 1096 ] {} <Trace> RaftInstance: commit upto 115, current idx 115
2021.03.24 11:22:47.487512 [ 1096 ] {} <Debug> RaftInstance: notify cb 115 0x7faa1f09b620
2021.03.24 11:22:47.487534 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 115, current idx 115
2021.03.24 11:22:47.487543 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.487562 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.487583 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 115 wake up (9729 us), return value (nil)
2021.03.24 11:22:47.491010 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.491039 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=114
2021.03.24 11:22:47.491053 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.491068 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 114
2021.03.24 11:22:47.491078 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.491095 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 110, new matched idx: 113
2021.03.24 11:22:47.491111 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 116 115 113 
2021.03.24 11:22:47.491123 [ 1094 ] {} <Trace> RaftInstance: local log idx 116, target_commit_idx 115, quick_commit_index_ 115, state_->get_commit_idx() 115
2021.03.24 11:22:47.491134 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.491145 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.491156 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 113, starting_idx: 91, cur_nxt_idx: 117
2021.03.24 11:22:47.491172 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=113, LastLogTerm=1, EntriesLength=3, CommitIndex=115, Term=1, peer_last_sent_idx 111
2021.03.24 11:22:47.491182 [ 1094 ] {} <Debug> RaftInstance: idx range: 114-116
2021.03.24 11:22:47.491194 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.491264 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.494180 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.494248 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=117
2021.03.24 11:22:47.494259 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.494270 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 117
2021.03.24 11:22:47.494278 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.494286 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 115, new matched idx: 116
2021.03.24 11:22:47.494298 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 116 116 113 
2021.03.24 11:22:47.494306 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 116
2021.03.24 11:22:47.494315 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.494324 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.494332 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 116, starting_idx: 91, cur_nxt_idx: 117
2021.03.24 11:22:47.494343 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=116, LastLogTerm=1, EntriesLength=0, CommitIndex=116, Term=1, peer_last_sent_idx 116
2021.03.24 11:22:47.494350 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.494358 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.494421 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.494434 [ 1095 ] {} <Trace> RaftInstance: local log idx 116, target_commit_idx 116, quick_commit_index_ 116, state_->get_commit_idx() 115
2021.03.24 11:22:47.494442 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.494476 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.494496 [ 1096 ] {} <Debug> RaftInstance: commit upto 116, current idx 115
2021.03.24 11:22:47.494506 [ 1096 ] {} <Trace> RaftInstance: commit upto 116, current idx 116
2021.03.24 11:22:47.494532 [ 1096 ] {} <Debug> RaftInstance: notify cb 116 0x7faa07c4f020
2021.03.24 11:22:47.494557 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 116, current idx 116
2021.03.24 11:22:47.494566 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.494579 [ 1175 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.494607 [ 1175 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 116 wake up (10623 us), return value 0x7faaa9a69ca0
2021.03.24 11:22:47.495274 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.495306 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 117
2021.03.24 11:22:47.495329 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 117 0x7faa0daae120 sleep
2021.03.24 11:22:47.495375 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.495398 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.496711 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.496756 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 118
2021.03.24 11:22:47.496782 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.496792 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.496804 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 118 0x7faa1f09bc20 sleep
2021.03.24 11:22:47.501300 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.501333 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=117
2021.03.24 11:22:47.501343 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.501366 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 117
2021.03.24 11:22:47.501375 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.501384 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 116, new matched idx: 116
2021.03.24 11:22:47.501396 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 118 116 113 
2021.03.24 11:22:47.501405 [ 1093 ] {} <Trace> RaftInstance: local log idx 118, target_commit_idx 116, quick_commit_index_ 116, state_->get_commit_idx() 116
2021.03.24 11:22:47.501413 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.501421 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.501430 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 116, starting_idx: 91, cur_nxt_idx: 119
2021.03.24 11:22:47.501437 [ 1093 ] {} <Debug> RaftInstance: last sent log (117) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.501449 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=116, LastLogTerm=1, EntriesLength=2, CommitIndex=116, Term=1, peer_last_sent_idx 117
2021.03.24 11:22:47.501457 [ 1093 ] {} <Debug> RaftInstance: idx range: 117-118
2021.03.24 11:22:47.501466 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.501528 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.507439 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.507508 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 119
2021.03.24 11:22:47.507534 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 119 0x7faa1f070b20 sleep
2021.03.24 11:22:47.507578 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.507609 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.508248 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.508271 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=119
2021.03.24 11:22:47.508282 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.508292 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 119
2021.03.24 11:22:47.508300 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.508308 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 116, new matched idx: 118
2021.03.24 11:22:47.508319 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 119 118 113 
2021.03.24 11:22:47.508327 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 118
2021.03.24 11:22:47.508335 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.508343 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.508352 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 118, starting_idx: 91, cur_nxt_idx: 120
2021.03.24 11:22:47.508364 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=118, LastLogTerm=1, EntriesLength=1, CommitIndex=118, Term=1, peer_last_sent_idx 117
2021.03.24 11:22:47.508371 [ 1093 ] {} <Debug> RaftInstance: idx: 119
2021.03.24 11:22:47.508380 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.508439 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.508453 [ 1093 ] {} <Trace> RaftInstance: local log idx 119, target_commit_idx 118, quick_commit_index_ 118, state_->get_commit_idx() 116
2021.03.24 11:22:47.508461 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.508477 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.508487 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.508516 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.508538 [ 1096 ] {} <Debug> RaftInstance: commit upto 118, current idx 116
2021.03.24 11:22:47.508548 [ 1096 ] {} <Trace> RaftInstance: commit upto 118, current idx 117
2021.03.24 11:22:47.508637 [ 1096 ] {} <Debug> RaftInstance: notify cb 117 0x7faa0daae120
2021.03.24 11:22:47.508669 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.508682 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 117 wake up (13347 us), return value (nil)
2021.03.24 11:22:47.508704 [ 1096 ] {} <Trace> RaftInstance: commit upto 118, current idx 118
2021.03.24 11:22:47.508754 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.508773 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 120
2021.03.24 11:22:47.508786 [ 1096 ] {} <Debug> RaftInstance: notify cb 118 0x7faa1f09bc20
2021.03.24 11:22:47.508824 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 120 0x7faa0daae020 sleep
2021.03.24 11:22:47.508838 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.508875 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.508882 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.508897 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 118 wake up (12111 us), return value (nil)
2021.03.24 11:22:47.508840 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 118, current idx 118
2021.03.24 11:22:47.508970 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.511869 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.511893 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=117
2021.03.24 11:22:47.511904 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.511913 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 117
2021.03.24 11:22:47.511921 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.511929 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 113, new matched idx: 116
2021.03.24 11:22:47.511939 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 120 118 116 
2021.03.24 11:22:47.511948 [ 1094 ] {} <Trace> RaftInstance: local log idx 120, target_commit_idx 118, quick_commit_index_ 118, state_->get_commit_idx() 118
2021.03.24 11:22:47.511956 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.511965 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.511973 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 116, starting_idx: 91, cur_nxt_idx: 121
2021.03.24 11:22:47.511984 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=116, LastLogTerm=1, EntriesLength=4, CommitIndex=118, Term=1, peer_last_sent_idx 114
2021.03.24 11:22:47.511992 [ 1094 ] {} <Debug> RaftInstance: idx range: 117-120
2021.03.24 11:22:47.512005 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.512053 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.515306 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.515353 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=120
2021.03.24 11:22:47.515363 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.515374 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 120
2021.03.24 11:22:47.515382 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.515390 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 118, new matched idx: 119
2021.03.24 11:22:47.515402 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 120 119 116 
2021.03.24 11:22:47.515433 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 119
2021.03.24 11:22:47.515442 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.515450 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.515459 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 119, starting_idx: 91, cur_nxt_idx: 121
2021.03.24 11:22:47.515470 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=119, LastLogTerm=1, EntriesLength=1, CommitIndex=119, Term=1, peer_last_sent_idx 119
2021.03.24 11:22:47.515478 [ 1092 ] {} <Debug> RaftInstance: idx: 120
2021.03.24 11:22:47.515487 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.515554 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.515567 [ 1092 ] {} <Trace> RaftInstance: local log idx 120, target_commit_idx 119, quick_commit_index_ 119, state_->get_commit_idx() 118
2021.03.24 11:22:47.515575 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.515588 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.515597 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.515619 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.515637 [ 1096 ] {} <Debug> RaftInstance: commit upto 119, current idx 118
2021.03.24 11:22:47.515647 [ 1096 ] {} <Trace> RaftInstance: commit upto 119, current idx 119
2021.03.24 11:22:47.515707 [ 1096 ] {} <Debug> RaftInstance: notify cb 119 0x7faa1f070b20
2021.03.24 11:22:47.515727 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 119, current idx 119
2021.03.24 11:22:47.515736 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.515760 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.515782 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 119 wake up (8230 us), return value (nil)
2021.03.24 11:22:47.522430 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.522479 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=121
2021.03.24 11:22:47.522490 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.522502 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 121
2021.03.24 11:22:47.522510 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.522518 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 119, new matched idx: 120
2021.03.24 11:22:47.522530 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 120 120 116 
2021.03.24 11:22:47.522538 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 120
2021.03.24 11:22:47.522547 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.522555 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.522563 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 120, starting_idx: 91, cur_nxt_idx: 121
2021.03.24 11:22:47.522574 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=120, LastLogTerm=1, EntriesLength=0, CommitIndex=120, Term=1, peer_last_sent_idx 120
2021.03.24 11:22:47.522582 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.522590 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.522655 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.522669 [ 1094 ] {} <Trace> RaftInstance: local log idx 120, target_commit_idx 120, quick_commit_index_ 120, state_->get_commit_idx() 119
2021.03.24 11:22:47.522677 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.522713 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.522734 [ 1096 ] {} <Debug> RaftInstance: commit upto 120, current idx 119
2021.03.24 11:22:47.522765 [ 1096 ] {} <Trace> RaftInstance: commit upto 120, current idx 120
2021.03.24 11:22:47.522849 [ 1096 ] {} <Debug> RaftInstance: notify cb 120 0x7faa0daae020
2021.03.24 11:22:47.522882 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.522893 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 120 wake up (14094 us), return value (nil)
2021.03.24 11:22:47.522917 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.522939 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 121
2021.03.24 11:22:47.522958 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 121 0x7faa0daae120 sleep
2021.03.24 11:22:47.522973 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 120, current idx 120
2021.03.24 11:22:47.522981 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.523020 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.523036 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.523857 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.523880 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 122
2021.03.24 11:22:47.523893 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 122 0x7faa1f070820 sleep
2021.03.24 11:22:47.523917 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.523937 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.529374 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.529426 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=121
2021.03.24 11:22:47.529437 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.529449 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 121
2021.03.24 11:22:47.529457 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.529466 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 120, new matched idx: 120
2021.03.24 11:22:47.529479 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 122 120 116 
2021.03.24 11:22:47.529488 [ 1093 ] {} <Trace> RaftInstance: local log idx 122, target_commit_idx 120, quick_commit_index_ 120, state_->get_commit_idx() 120
2021.03.24 11:22:47.529496 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.529505 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.529514 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 120, starting_idx: 91, cur_nxt_idx: 123
2021.03.24 11:22:47.529522 [ 1093 ] {} <Debug> RaftInstance: last sent log (121) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.529534 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=120, LastLogTerm=1, EntriesLength=2, CommitIndex=120, Term=1, peer_last_sent_idx 121
2021.03.24 11:22:47.529542 [ 1093 ] {} <Debug> RaftInstance: idx range: 121-122
2021.03.24 11:22:47.529551 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.529619 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.532717 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.532765 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=121
2021.03.24 11:22:47.532776 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.532787 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 121
2021.03.24 11:22:47.532795 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.532827 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 116, new matched idx: 120
2021.03.24 11:22:47.532841 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 122 120 120 
2021.03.24 11:22:47.532850 [ 1095 ] {} <Trace> RaftInstance: local log idx 122, target_commit_idx 120, quick_commit_index_ 120, state_->get_commit_idx() 120
2021.03.24 11:22:47.532858 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.532866 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.532875 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 120, starting_idx: 91, cur_nxt_idx: 123
2021.03.24 11:22:47.532888 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=120, LastLogTerm=1, EntriesLength=2, CommitIndex=120, Term=1, peer_last_sent_idx 117
2021.03.24 11:22:47.532895 [ 1095 ] {} <Debug> RaftInstance: idx range: 121-122
2021.03.24 11:22:47.532904 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.532971 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.535550 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.535600 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 123
2021.03.24 11:22:47.535624 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 123 0x7faaa6f93c20 sleep
2021.03.24 11:22:47.535647 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.535667 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.536436 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.536454 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=123
2021.03.24 11:22:47.536463 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.536472 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 123
2021.03.24 11:22:47.536479 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.536487 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 120, new matched idx: 122
2021.03.24 11:22:47.536496 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 123 122 120 
2021.03.24 11:22:47.536504 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 122
2021.03.24 11:22:47.536512 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.536520 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.536527 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 122, starting_idx: 91, cur_nxt_idx: 124
2021.03.24 11:22:47.536538 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=122, LastLogTerm=1, EntriesLength=1, CommitIndex=122, Term=1, peer_last_sent_idx 121
2021.03.24 11:22:47.536545 [ 1095 ] {} <Debug> RaftInstance: idx: 123
2021.03.24 11:22:47.536554 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.536598 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.536610 [ 1095 ] {} <Trace> RaftInstance: local log idx 123, target_commit_idx 122, quick_commit_index_ 122, state_->get_commit_idx() 120
2021.03.24 11:22:47.536618 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.536632 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.536641 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.536643 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.536662 [ 1096 ] {} <Debug> RaftInstance: commit upto 122, current idx 120
2021.03.24 11:22:47.536672 [ 1096 ] {} <Trace> RaftInstance: commit upto 122, current idx 121
2021.03.24 11:22:47.536778 [ 1096 ] {} <Debug> RaftInstance: notify cb 121 0x7faa0daae120
2021.03.24 11:22:47.536809 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.536822 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 121 wake up (13855 us), return value (nil)
2021.03.24 11:22:47.536847 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.536866 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 124
2021.03.24 11:22:47.536889 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 124 0x7faa0daae020 sleep
2021.03.24 11:22:47.536906 [ 1096 ] {} <Trace> RaftInstance: commit upto 122, current idx 122
2021.03.24 11:22:47.536907 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.536943 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.536965 [ 1096 ] {} <Debug> RaftInstance: notify cb 122 0x7faa1f070820
2021.03.24 11:22:47.536978 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 122, current idx 122
2021.03.24 11:22:47.536985 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.537001 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.537011 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 122 wake up (13108 us), return value (nil)
2021.03.24 11:22:47.543390 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.543425 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=124
2021.03.24 11:22:47.543435 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.543445 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 124
2021.03.24 11:22:47.543453 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.543461 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 122, new matched idx: 123
2021.03.24 11:22:47.543472 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 124 123 120 
2021.03.24 11:22:47.543480 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 123
2021.03.24 11:22:47.543489 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.543497 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.543505 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 123, starting_idx: 91, cur_nxt_idx: 125
2021.03.24 11:22:47.543516 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=123, LastLogTerm=1, EntriesLength=1, CommitIndex=123, Term=1, peer_last_sent_idx 123
2021.03.24 11:22:47.543523 [ 1094 ] {} <Debug> RaftInstance: idx: 124
2021.03.24 11:22:47.543532 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.543581 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.543594 [ 1094 ] {} <Trace> RaftInstance: local log idx 124, target_commit_idx 123, quick_commit_index_ 123, state_->get_commit_idx() 122
2021.03.24 11:22:47.543602 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.543614 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.543623 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.543644 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.543654 [ 1096 ] {} <Debug> RaftInstance: commit upto 123, current idx 122
2021.03.24 11:22:47.543662 [ 1096 ] {} <Trace> RaftInstance: commit upto 123, current idx 123
2021.03.24 11:22:47.543706 [ 1096 ] {} <Debug> RaftInstance: notify cb 123 0x7faaa6f93c20
2021.03.24 11:22:47.543729 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 123, current idx 123
2021.03.24 11:22:47.543761 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.543759 [ 1095 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:47.543786 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.543796 [ 1095 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:47.543868 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.543839 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 123 wake up (8151 us), return value (nil)
2021.03.24 11:22:47.550281 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.550323 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=125
2021.03.24 11:22:47.550333 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.550345 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 125
2021.03.24 11:22:47.550352 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.550361 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 123, new matched idx: 124
2021.03.24 11:22:47.550372 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 124 124 120 
2021.03.24 11:22:47.550380 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 124
2021.03.24 11:22:47.550390 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.550398 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.550406 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 124, starting_idx: 91, cur_nxt_idx: 125
2021.03.24 11:22:47.550417 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=124, LastLogTerm=1, EntriesLength=0, CommitIndex=124, Term=1, peer_last_sent_idx 124
2021.03.24 11:22:47.550424 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.550432 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.550495 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.550508 [ 1094 ] {} <Trace> RaftInstance: local log idx 124, target_commit_idx 124, quick_commit_index_ 124, state_->get_commit_idx() 123
2021.03.24 11:22:47.550516 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.550542 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.550553 [ 1096 ] {} <Debug> RaftInstance: commit upto 124, current idx 123
2021.03.24 11:22:47.550562 [ 1096 ] {} <Trace> RaftInstance: commit upto 124, current idx 124
2021.03.24 11:22:47.550631 [ 1096 ] {} <Debug> RaftInstance: notify cb 124 0x7faa0daae020
2021.03.24 11:22:47.550661 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.550673 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 124 wake up (13782 us), return value (nil)
2021.03.24 11:22:47.550696 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.550718 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 125
2021.03.24 11:22:47.550735 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 125 0x7faa0daae120 sleep
2021.03.24 11:22:47.550749 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 124, current idx 124
2021.03.24 11:22:47.550758 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.550757 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.550812 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.553724 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.553758 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=123
2021.03.24 11:22:47.553769 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.553801 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 123
2021.03.24 11:22:47.553810 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.553818 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 120, new matched idx: 122
2021.03.24 11:22:47.553828 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 125 124 122 
2021.03.24 11:22:47.553836 [ 1095 ] {} <Trace> RaftInstance: local log idx 125, target_commit_idx 124, quick_commit_index_ 124, state_->get_commit_idx() 124
2021.03.24 11:22:47.553844 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.553852 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.553861 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 122, starting_idx: 91, cur_nxt_idx: 126
2021.03.24 11:22:47.553873 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=122, LastLogTerm=1, EntriesLength=3, CommitIndex=124, Term=1, peer_last_sent_idx 121
2021.03.24 11:22:47.553880 [ 1095 ] {} <Debug> RaftInstance: idx range: 123-125
2021.03.24 11:22:47.553889 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.553945 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.557322 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.557353 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=125
2021.03.24 11:22:47.557365 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.557375 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 125
2021.03.24 11:22:47.557383 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.557391 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 124, new matched idx: 124
2021.03.24 11:22:47.557401 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 125 124 122 
2021.03.24 11:22:47.557410 [ 1092 ] {} <Trace> RaftInstance: local log idx 125, target_commit_idx 124, quick_commit_index_ 124, state_->get_commit_idx() 124
2021.03.24 11:22:47.557418 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.557427 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.557435 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 124, starting_idx: 91, cur_nxt_idx: 126
2021.03.24 11:22:47.557446 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=124, LastLogTerm=1, EntriesLength=1, CommitIndex=124, Term=1, peer_last_sent_idx 125
2021.03.24 11:22:47.557453 [ 1092 ] {} <Debug> RaftInstance: idx: 125
2021.03.24 11:22:47.557462 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.557516 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.563842 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.563902 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 126
2021.03.24 11:22:47.563927 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 126 0x7faaa6f93c20 sleep
2021.03.24 11:22:47.563971 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.564008 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.564230 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.564251 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=126
2021.03.24 11:22:47.564262 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.564273 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 126
2021.03.24 11:22:47.564281 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.564312 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 124, new matched idx: 125
2021.03.24 11:22:47.564324 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 126 125 122 
2021.03.24 11:22:47.564332 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 125
2021.03.24 11:22:47.564340 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.564348 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.564356 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 125, starting_idx: 91, cur_nxt_idx: 127
2021.03.24 11:22:47.564368 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=125, LastLogTerm=1, EntriesLength=1, CommitIndex=125, Term=1, peer_last_sent_idx 125
2021.03.24 11:22:47.564376 [ 1094 ] {} <Debug> RaftInstance: idx: 126
2021.03.24 11:22:47.564386 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.564454 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.564468 [ 1094 ] {} <Trace> RaftInstance: local log idx 126, target_commit_idx 125, quick_commit_index_ 125, state_->get_commit_idx() 124
2021.03.24 11:22:47.564476 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.564488 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.564496 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.564515 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.564527 [ 1096 ] {} <Debug> RaftInstance: commit upto 125, current idx 124
2021.03.24 11:22:47.564536 [ 1096 ] {} <Trace> RaftInstance: commit upto 125, current idx 125
2021.03.24 11:22:47.564603 [ 1096 ] {} <Debug> RaftInstance: notify cb 125 0x7faa0daae120
2021.03.24 11:22:47.564633 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.564646 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 125 wake up (13902 us), return value (nil)
2021.03.24 11:22:47.564666 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 125, current idx 125
2021.03.24 11:22:47.564675 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.565218 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.565250 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 127
2021.03.24 11:22:47.565272 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 127 0x7faa0daae020 sleep
2021.03.24 11:22:47.565291 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.565310 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.571151 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.571155 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.571204 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=127
2021.03.24 11:22:47.571248 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 128
2021.03.24 11:22:47.571259 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.571272 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 127
2021.03.24 11:22:47.571282 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.571272 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 128 0x7faa1f070b20 sleep
2021.03.24 11:22:47.571291 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 125, new matched idx: 126
2021.03.24 11:22:47.571310 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 128 126 122 
2021.03.24 11:22:47.571318 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 126
2021.03.24 11:22:47.571327 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.571353 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.571362 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 126, starting_idx: 91, cur_nxt_idx: 129
2021.03.24 11:22:47.571374 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=126, LastLogTerm=1, EntriesLength=2, CommitIndex=126, Term=1, peer_last_sent_idx 126
2021.03.24 11:22:47.571381 [ 1092 ] {} <Debug> RaftInstance: idx range: 127-128
2021.03.24 11:22:47.571391 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.571454 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.571467 [ 1092 ] {} <Trace> RaftInstance: local log idx 128, target_commit_idx 126, quick_commit_index_ 126, state_->get_commit_idx() 125
2021.03.24 11:22:47.571475 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.571487 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.571495 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.571520 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.571527 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.571549 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.571533 [ 1096 ] {} <Debug> RaftInstance: commit upto 126, current idx 125
2021.03.24 11:22:47.571574 [ 1096 ] {} <Trace> RaftInstance: commit upto 126, current idx 126
2021.03.24 11:22:47.571630 [ 1096 ] {} <Debug> RaftInstance: notify cb 126 0x7faaa6f93c20
2021.03.24 11:22:47.571654 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 126, current idx 126
2021.03.24 11:22:47.571663 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.571697 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.571729 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 126 wake up (7767 us), return value (nil)
2021.03.24 11:22:47.574484 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.574511 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=126
2021.03.24 11:22:47.574522 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.574531 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 126
2021.03.24 11:22:47.574539 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.574547 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 122, new matched idx: 125
2021.03.24 11:22:47.574557 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 128 126 125 
2021.03.24 11:22:47.574566 [ 1094 ] {} <Trace> RaftInstance: local log idx 128, target_commit_idx 126, quick_commit_index_ 126, state_->get_commit_idx() 126
2021.03.24 11:22:47.574574 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.574582 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.574590 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 125, starting_idx: 91, cur_nxt_idx: 129
2021.03.24 11:22:47.574601 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=125, LastLogTerm=1, EntriesLength=3, CommitIndex=126, Term=1, peer_last_sent_idx 123
2021.03.24 11:22:47.574608 [ 1094 ] {} <Debug> RaftInstance: idx range: 126-128
2021.03.24 11:22:47.574617 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.574667 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.578372 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.578409 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=129
2021.03.24 11:22:47.578459 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.578473 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 129
2021.03.24 11:22:47.578506 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.578517 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 126, new matched idx: 128
2021.03.24 11:22:47.578530 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 128 128 125 
2021.03.24 11:22:47.578538 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 128
2021.03.24 11:22:47.578547 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.578556 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.578564 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 128, starting_idx: 91, cur_nxt_idx: 129
2021.03.24 11:22:47.578575 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=128, LastLogTerm=1, EntriesLength=0, CommitIndex=128, Term=1, peer_last_sent_idx 127
2021.03.24 11:22:47.578583 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.578591 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.578648 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.578661 [ 1093 ] {} <Trace> RaftInstance: local log idx 128, target_commit_idx 128, quick_commit_index_ 128, state_->get_commit_idx() 126
2021.03.24 11:22:47.578669 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.578737 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.578769 [ 1096 ] {} <Debug> RaftInstance: commit upto 128, current idx 126
2021.03.24 11:22:47.578799 [ 1096 ] {} <Trace> RaftInstance: commit upto 128, current idx 127
2021.03.24 11:22:47.578893 [ 1096 ] {} <Debug> RaftInstance: notify cb 127 0x7faa0daae020
2021.03.24 11:22:47.578926 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.578939 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 127 wake up (13660 us), return value (nil)
2021.03.24 11:22:47.578960 [ 1096 ] {} <Trace> RaftInstance: commit upto 128, current idx 128
2021.03.24 11:22:47.578990 [ 1096 ] {} <Debug> RaftInstance: notify cb 128 0x7faa1f070b20
2021.03.24 11:22:47.579009 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 128, current idx 128
2021.03.24 11:22:47.579018 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.579033 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.579052 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 128 wake up (7765 us), return value (nil)
2021.03.24 11:22:47.585247 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.585290 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=129
2021.03.24 11:22:47.585301 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.585311 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 129
2021.03.24 11:22:47.585319 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.585327 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 128, new matched idx: 128
2021.03.24 11:22:47.585339 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 128 128 125 
2021.03.24 11:22:47.585347 [ 1094 ] {} <Trace> RaftInstance: local log idx 128, target_commit_idx 128, quick_commit_index_ 128, state_->get_commit_idx() 128
2021.03.24 11:22:47.591577 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.591632 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 129
2021.03.24 11:22:47.591685 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 129 0x7faa1f057420 sleep
2021.03.24 11:22:47.591728 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.591764 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.591775 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 128, starting_idx: 91, cur_nxt_idx: 130
2021.03.24 11:22:47.591790 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=128, LastLogTerm=1, EntriesLength=1, CommitIndex=128, Term=1, peer_last_sent_idx 129
2021.03.24 11:22:47.591798 [ 1097 ] {} <Debug> RaftInstance: idx: 129
2021.03.24 11:22:47.591810 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.591889 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.594882 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.594923 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 130
2021.03.24 11:22:47.594975 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 130 0x7faa1f070b20 sleep
2021.03.24 11:22:47.595012 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.595038 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.595021 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.595068 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=129
2021.03.24 11:22:47.595080 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.595091 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 129
2021.03.24 11:22:47.595111 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.595120 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 125, new matched idx: 128
2021.03.24 11:22:47.595131 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 130 128 128 
2021.03.24 11:22:47.595140 [ 1092 ] {} <Trace> RaftInstance: local log idx 130, target_commit_idx 128, quick_commit_index_ 128, state_->get_commit_idx() 128
2021.03.24 11:22:47.595148 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.595156 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.595164 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 128, starting_idx: 91, cur_nxt_idx: 131
2021.03.24 11:22:47.595175 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=128, LastLogTerm=1, EntriesLength=2, CommitIndex=128, Term=1, peer_last_sent_idx 126
2021.03.24 11:22:47.595182 [ 1092 ] {} <Debug> RaftInstance: idx range: 129-130
2021.03.24 11:22:47.595191 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.595245 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.598591 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.598624 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=130
2021.03.24 11:22:47.598634 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.598644 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 130
2021.03.24 11:22:47.598652 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.598660 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 128, new matched idx: 129
2021.03.24 11:22:47.598670 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 130 129 128 
2021.03.24 11:22:47.598678 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 129
2021.03.24 11:22:47.598687 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.598695 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.598734 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 129, starting_idx: 91, cur_nxt_idx: 131
2021.03.24 11:22:47.598747 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=129, LastLogTerm=1, EntriesLength=1, CommitIndex=129, Term=1, peer_last_sent_idx 129
2021.03.24 11:22:47.598754 [ 1094 ] {} <Debug> RaftInstance: idx: 130
2021.03.24 11:22:47.598763 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.598832 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.598845 [ 1094 ] {} <Trace> RaftInstance: local log idx 130, target_commit_idx 129, quick_commit_index_ 129, state_->get_commit_idx() 128
2021.03.24 11:22:47.598853 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.598866 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.598875 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.598908 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.598920 [ 1096 ] {} <Debug> RaftInstance: commit upto 129, current idx 128
2021.03.24 11:22:47.598930 [ 1096 ] {} <Trace> RaftInstance: commit upto 129, current idx 129
2021.03.24 11:22:47.598977 [ 1096 ] {} <Debug> RaftInstance: notify cb 129 0x7faa1f057420
2021.03.24 11:22:47.598999 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 129, current idx 129
2021.03.24 11:22:47.599008 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.599028 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.599070 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 129 wake up (7344 us), return value (nil)
2021.03.24 11:22:47.604436 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.604489 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 131
2021.03.24 11:22:47.604516 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 131 0x7faa0daae120 sleep
2021.03.24 11:22:47.604554 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.604574 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.605441 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.605461 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=131
2021.03.24 11:22:47.605471 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.605481 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 131
2021.03.24 11:22:47.605489 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.605497 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 129, new matched idx: 130
2021.03.24 11:22:47.605507 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 131 130 128 
2021.03.24 11:22:47.605515 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 130
2021.03.24 11:22:47.605523 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.605531 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.605540 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 130, starting_idx: 91, cur_nxt_idx: 132
2021.03.24 11:22:47.605552 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=130, LastLogTerm=1, EntriesLength=1, CommitIndex=130, Term=1, peer_last_sent_idx 130
2021.03.24 11:22:47.605559 [ 1094 ] {} <Debug> RaftInstance: idx: 131
2021.03.24 11:22:47.605568 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.605644 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.605658 [ 1094 ] {} <Trace> RaftInstance: local log idx 131, target_commit_idx 130, quick_commit_index_ 130, state_->get_commit_idx() 129
2021.03.24 11:22:47.605666 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.605694 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.605703 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.605720 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.605739 [ 1096 ] {} <Debug> RaftInstance: commit upto 130, current idx 129
2021.03.24 11:22:47.605747 [ 1096 ] {} <Trace> RaftInstance: commit upto 130, current idx 130
2021.03.24 11:22:47.605816 [ 1096 ] {} <Debug> RaftInstance: notify cb 130 0x7faa1f070b20
2021.03.24 11:22:47.605838 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 130, current idx 130
2021.03.24 11:22:47.605846 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.605860 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.605889 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 130 wake up (10888 us), return value (nil)
2021.03.24 11:22:47.612484 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.612515 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=132
2021.03.24 11:22:47.612525 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.612536 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 132
2021.03.24 11:22:47.612544 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.612552 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 130, new matched idx: 131
2021.03.24 11:22:47.612565 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 131 131 128 
2021.03.24 11:22:47.612572 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 131
2021.03.24 11:22:47.612582 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.612590 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.612598 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 131, starting_idx: 91, cur_nxt_idx: 132
2021.03.24 11:22:47.612610 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=131, LastLogTerm=1, EntriesLength=0, CommitIndex=131, Term=1, peer_last_sent_idx 131
2021.03.24 11:22:47.612617 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.612625 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.612676 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.612689 [ 1093 ] {} <Trace> RaftInstance: local log idx 131, target_commit_idx 131, quick_commit_index_ 131, state_->get_commit_idx() 130
2021.03.24 11:22:47.612697 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.612743 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.612787 [ 1096 ] {} <Debug> RaftInstance: commit upto 131, current idx 130
2021.03.24 11:22:47.612798 [ 1096 ] {} <Trace> RaftInstance: commit upto 131, current idx 131
2021.03.24 11:22:47.612867 [ 1096 ] {} <Debug> RaftInstance: notify cb 131 0x7faa0daae120
2021.03.24 11:22:47.612900 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.612912 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 131 wake up (8394 us), return value (nil)
2021.03.24 11:22:47.612934 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 131, current idx 131
2021.03.24 11:22:47.612943 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.613549 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.613579 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 132
2021.03.24 11:22:47.613609 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 132 0x7faa0daae020 sleep
2021.03.24 11:22:47.613630 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.613672 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.615685 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.615708 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=131
2021.03.24 11:22:47.615718 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.615737 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 131
2021.03.24 11:22:47.615745 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.615753 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 128, new matched idx: 130
2021.03.24 11:22:47.615764 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 132 131 130 
2021.03.24 11:22:47.615772 [ 1094 ] {} <Trace> RaftInstance: local log idx 132, target_commit_idx 131, quick_commit_index_ 131, state_->get_commit_idx() 131
2021.03.24 11:22:47.615780 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.615788 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.615797 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 130, starting_idx: 91, cur_nxt_idx: 133
2021.03.24 11:22:47.615808 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=130, LastLogTerm=1, EntriesLength=2, CommitIndex=131, Term=1, peer_last_sent_idx 129
2021.03.24 11:22:47.615816 [ 1094 ] {} <Debug> RaftInstance: idx range: 131-132
2021.03.24 11:22:47.615825 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.615886 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.618892 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.618939 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 133
2021.03.24 11:22:47.618962 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 133 0x7faa1f070820 sleep
2021.03.24 11:22:47.618987 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.619005 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.619515 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.619539 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=132
2021.03.24 11:22:47.619549 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.619559 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 132
2021.03.24 11:22:47.619567 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.619574 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 131, new matched idx: 131
2021.03.24 11:22:47.619585 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 133 131 130 
2021.03.24 11:22:47.619593 [ 1095 ] {} <Trace> RaftInstance: local log idx 133, target_commit_idx 131, quick_commit_index_ 131, state_->get_commit_idx() 131
2021.03.24 11:22:47.619601 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.619609 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.619617 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 131, starting_idx: 91, cur_nxt_idx: 134
2021.03.24 11:22:47.619624 [ 1095 ] {} <Debug> RaftInstance: last sent log (132) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.619635 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=131, LastLogTerm=1, EntriesLength=2, CommitIndex=131, Term=1, peer_last_sent_idx 132
2021.03.24 11:22:47.619642 [ 1095 ] {} <Debug> RaftInstance: idx range: 132-133
2021.03.24 11:22:47.619668 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.619745 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.626511 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.626536 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=134
2021.03.24 11:22:47.626554 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.626563 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 134
2021.03.24 11:22:47.626571 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.626579 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 131, new matched idx: 133
2021.03.24 11:22:47.626589 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 133 133 130 
2021.03.24 11:22:47.626615 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 133
2021.03.24 11:22:47.626634 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.626649 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.626663 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 133, starting_idx: 91, cur_nxt_idx: 134
2021.03.24 11:22:47.626673 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=133, LastLogTerm=1, EntriesLength=0, CommitIndex=133, Term=1, peer_last_sent_idx 132
2021.03.24 11:22:47.626680 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.626688 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.626743 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.626757 [ 1093 ] {} <Trace> RaftInstance: local log idx 133, target_commit_idx 133, quick_commit_index_ 133, state_->get_commit_idx() 131
2021.03.24 11:22:47.626765 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.626829 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.626870 [ 1096 ] {} <Debug> RaftInstance: commit upto 133, current idx 131
2021.03.24 11:22:47.626881 [ 1096 ] {} <Trace> RaftInstance: commit upto 133, current idx 132
2021.03.24 11:22:47.627017 [ 1096 ] {} <Debug> RaftInstance: notify cb 132 0x7faa0daae020
2021.03.24 11:22:47.627061 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.627078 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 132 wake up (13456 us), return value (nil)
2021.03.24 11:22:47.627108 [ 1096 ] {} <Trace> RaftInstance: commit upto 133, current idx 133
2021.03.24 11:22:47.627163 [ 1096 ] {} <Debug> RaftInstance: notify cb 133 0x7faa1f070820
2021.03.24 11:22:47.627182 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 133, current idx 133
2021.03.24 11:22:47.627192 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.627215 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.627230 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 133 wake up (8260 us), return value (nil)
2021.03.24 11:22:47.631019 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.631086 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 134
2021.03.24 11:22:47.631109 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 134 0x7faa1f070d20 sleep
2021.03.24 11:22:47.631151 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.631187 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.633701 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.633727 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=134
2021.03.24 11:22:47.633738 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.633749 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 134
2021.03.24 11:22:47.633757 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.633766 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 133, new matched idx: 133
2021.03.24 11:22:47.633778 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 134 133 130 
2021.03.24 11:22:47.633788 [ 1094 ] {} <Trace> RaftInstance: local log idx 134, target_commit_idx 133, quick_commit_index_ 133, state_->get_commit_idx() 133
2021.03.24 11:22:47.633815 [ 1176 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.633817 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.633855 [ 1176 ] {} <Debug> RaftInstance: append at log_idx 135
2021.03.24 11:22:47.633859 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.633883 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 133, starting_idx: 91, cur_nxt_idx: 136
2021.03.24 11:22:47.633892 [ 1094 ] {} <Debug> RaftInstance: last sent log (134) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.633892 [ 1176 ] {} <Debug> RaftInstance: commit_ret_cv 135 0x7faa1f021120 sleep
2021.03.24 11:22:47.633930 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=133, LastLogTerm=1, EntriesLength=2, CommitIndex=133, Term=1, peer_last_sent_idx 134
2021.03.24 11:22:47.633956 [ 1094 ] {} <Debug> RaftInstance: idx range: 134-135
2021.03.24 11:22:47.633967 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.634020 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.634072 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.634091 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.636506 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.636532 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=133
2021.03.24 11:22:47.636543 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.636553 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 133
2021.03.24 11:22:47.636561 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.636569 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 130, new matched idx: 132
2021.03.24 11:22:47.636579 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 135 133 132 
2021.03.24 11:22:47.636587 [ 1092 ] {} <Trace> RaftInstance: local log idx 135, target_commit_idx 133, quick_commit_index_ 133, state_->get_commit_idx() 133
2021.03.24 11:22:47.636594 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.636602 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.636610 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 132, starting_idx: 91, cur_nxt_idx: 136
2021.03.24 11:22:47.636621 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=132, LastLogTerm=1, EntriesLength=3, CommitIndex=133, Term=1, peer_last_sent_idx 131
2021.03.24 11:22:47.636628 [ 1092 ] {} <Debug> RaftInstance: idx range: 133-135
2021.03.24 11:22:47.636637 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.636688 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.640735 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.640769 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=136
2021.03.24 11:22:47.640779 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.640789 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 136
2021.03.24 11:22:47.640797 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.640814 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 133, new matched idx: 135
2021.03.24 11:22:47.640825 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 135 135 132 
2021.03.24 11:22:47.640832 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 135
2021.03.24 11:22:47.640841 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.640849 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.640857 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 135, starting_idx: 91, cur_nxt_idx: 136
2021.03.24 11:22:47.640868 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=135, LastLogTerm=1, EntriesLength=0, CommitIndex=135, Term=1, peer_last_sent_idx 134
2021.03.24 11:22:47.640875 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.640883 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.640945 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.640958 [ 1093 ] {} <Trace> RaftInstance: local log idx 135, target_commit_idx 135, quick_commit_index_ 135, state_->get_commit_idx() 133
2021.03.24 11:22:47.640965 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.641015 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.641045 [ 1096 ] {} <Debug> RaftInstance: commit upto 135, current idx 133
2021.03.24 11:22:47.641057 [ 1096 ] {} <Trace> RaftInstance: commit upto 135, current idx 134
2021.03.24 11:22:47.641126 [ 1096 ] {} <Debug> RaftInstance: notify cb 134 0x7faa1f070d20
2021.03.24 11:22:47.641152 [ 1096 ] {} <Trace> RaftInstance: commit upto 135, current idx 135
2021.03.24 11:22:47.641179 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.641223 [ 1096 ] {} <Debug> RaftInstance: notify cb 135 0x7faa1f021120
2021.03.24 11:22:47.641244 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 135, current idx 135
2021.03.24 11:22:47.641245 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 134 wake up (10073 us), return value (nil)
2021.03.24 11:22:47.641255 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.641271 [ 1176 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.641302 [ 1176 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 135 wake up (7390 us), return value 0x7faaa9a69cb0
2021.03.24 11:22:47.641830 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.641857 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 136
2021.03.24 11:22:47.641879 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 136 0x7faa0daae120 sleep
2021.03.24 11:22:47.641896 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.641923 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.647323 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.647374 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 137
2021.03.24 11:22:47.647400 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 137 0x7faa1f09bf20 sleep
2021.03.24 11:22:47.647441 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.647489 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.647579 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.647596 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=136
2021.03.24 11:22:47.647604 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.647613 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 136
2021.03.24 11:22:47.647621 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.647629 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 135, new matched idx: 135
2021.03.24 11:22:47.647638 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 137 135 132 
2021.03.24 11:22:47.647647 [ 1093 ] {} <Trace> RaftInstance: local log idx 137, target_commit_idx 135, quick_commit_index_ 135, state_->get_commit_idx() 135
2021.03.24 11:22:47.647655 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.647668 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.647675 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 135, starting_idx: 91, cur_nxt_idx: 138
2021.03.24 11:22:47.647683 [ 1093 ] {} <Debug> RaftInstance: last sent log (136) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.647694 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=135, LastLogTerm=1, EntriesLength=2, CommitIndex=135, Term=1, peer_last_sent_idx 136
2021.03.24 11:22:47.647702 [ 1093 ] {} <Debug> RaftInstance: idx range: 136-137
2021.03.24 11:22:47.647711 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.647776 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.649645 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.649677 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 138
2021.03.24 11:22:47.649698 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 138 0x7faa1f070d20 sleep
2021.03.24 11:22:47.649718 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.649737 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.654470 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.654496 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=138
2021.03.24 11:22:47.654506 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.654516 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 138
2021.03.24 11:22:47.654523 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.654531 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 135, new matched idx: 137
2021.03.24 11:22:47.654542 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 138 137 132 
2021.03.24 11:22:47.654550 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 137
2021.03.24 11:22:47.654558 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.654566 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.654574 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 137, starting_idx: 91, cur_nxt_idx: 139
2021.03.24 11:22:47.654596 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=137, LastLogTerm=1, EntriesLength=1, CommitIndex=137, Term=1, peer_last_sent_idx 136
2021.03.24 11:22:47.654605 [ 1093 ] {} <Debug> RaftInstance: idx: 138
2021.03.24 11:22:47.654614 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.654685 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.654698 [ 1093 ] {} <Trace> RaftInstance: local log idx 138, target_commit_idx 137, quick_commit_index_ 137, state_->get_commit_idx() 135
2021.03.24 11:22:47.654708 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.654718 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.654727 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.654737 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.654801 [ 1096 ] {} <Debug> RaftInstance: commit upto 137, current idx 135
2021.03.24 11:22:47.654815 [ 1096 ] {} <Trace> RaftInstance: commit upto 137, current idx 136
2021.03.24 11:22:47.654901 [ 1096 ] {} <Debug> RaftInstance: notify cb 136 0x7faa0daae120
2021.03.24 11:22:47.654936 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.654948 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 136 wake up (13062 us), return value (nil)
2021.03.24 11:22:47.654971 [ 1096 ] {} <Trace> RaftInstance: commit upto 137, current idx 137
2021.03.24 11:22:47.654996 [ 1096 ] {} <Debug> RaftInstance: notify cb 137 0x7faa1f09bf20
2021.03.24 11:22:47.655014 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 137, current idx 137
2021.03.24 11:22:47.655023 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.655038 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.655060 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 137 wake up (7645 us), return value 0x7faaa9a69cc0
2021.03.24 11:22:47.655732 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.655760 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 139
2021.03.24 11:22:47.655781 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 139 0x7faa0daae020 sleep
2021.03.24 11:22:47.655801 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.655860 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.657185 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.657206 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=136
2021.03.24 11:22:47.657215 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.657225 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 136
2021.03.24 11:22:47.657233 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.657241 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 132, new matched idx: 135
2021.03.24 11:22:47.657252 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 139 137 135 
2021.03.24 11:22:47.657261 [ 1094 ] {} <Trace> RaftInstance: local log idx 139, target_commit_idx 137, quick_commit_index_ 137, state_->get_commit_idx() 137
2021.03.24 11:22:47.657269 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.657278 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.657287 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 135, starting_idx: 91, cur_nxt_idx: 140
2021.03.24 11:22:47.657299 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=135, LastLogTerm=1, EntriesLength=4, CommitIndex=137, Term=1, peer_last_sent_idx 133
2021.03.24 11:22:47.657306 [ 1094 ] {} <Debug> RaftInstance: idx range: 136-139
2021.03.24 11:22:47.657315 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.657381 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.661573 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.661617 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=139
2021.03.24 11:22:47.661628 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.661639 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 139
2021.03.24 11:22:47.661647 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.661655 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 137, new matched idx: 138
2021.03.24 11:22:47.661690 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 139 138 135 
2021.03.24 11:22:47.661702 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 138
2021.03.24 11:22:47.661711 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.661719 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.661728 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 138, starting_idx: 91, cur_nxt_idx: 140
2021.03.24 11:22:47.661740 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=138, LastLogTerm=1, EntriesLength=1, CommitIndex=138, Term=1, peer_last_sent_idx 138
2021.03.24 11:22:47.661752 [ 1092 ] {} <Debug> RaftInstance: idx: 139
2021.03.24 11:22:47.661761 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.661814 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.661827 [ 1092 ] {} <Trace> RaftInstance: local log idx 139, target_commit_idx 138, quick_commit_index_ 138, state_->get_commit_idx() 137
2021.03.24 11:22:47.661835 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.661849 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.661858 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.661877 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.661912 [ 1096 ] {} <Debug> RaftInstance: commit upto 138, current idx 137
2021.03.24 11:22:47.661923 [ 1096 ] {} <Trace> RaftInstance: commit upto 138, current idx 138
2021.03.24 11:22:47.661967 [ 1096 ] {} <Debug> RaftInstance: notify cb 138 0x7faa1f070d20
2021.03.24 11:22:47.661990 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 138, current idx 138
2021.03.24 11:22:47.662000 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.662017 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.662039 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 138 wake up (12320 us), return value (nil)
2021.03.24 11:22:47.668668 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.668693 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=140
2021.03.24 11:22:47.668703 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.668712 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 140
2021.03.24 11:22:47.668720 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.668728 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 138, new matched idx: 139
2021.03.24 11:22:47.668738 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 139 139 135 
2021.03.24 11:22:47.668746 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 139
2021.03.24 11:22:47.668754 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.668762 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.668770 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 139, starting_idx: 91, cur_nxt_idx: 140
2021.03.24 11:22:47.668779 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=139, LastLogTerm=1, EntriesLength=0, CommitIndex=139, Term=1, peer_last_sent_idx 139
2021.03.24 11:22:47.668786 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.668804 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.668848 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.668861 [ 1093 ] {} <Trace> RaftInstance: local log idx 139, target_commit_idx 139, quick_commit_index_ 139, state_->get_commit_idx() 138
2021.03.24 11:22:47.668868 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.668918 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.668932 [ 1096 ] {} <Debug> RaftInstance: commit upto 139, current idx 138
2021.03.24 11:22:47.668940 [ 1096 ] {} <Trace> RaftInstance: commit upto 139, current idx 139
2021.03.24 11:22:47.669009 [ 1096 ] {} <Debug> RaftInstance: notify cb 139 0x7faa0daae020
2021.03.24 11:22:47.669024 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 139, current idx 139
2021.03.24 11:22:47.669032 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.669061 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.669089 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 139 wake up (13287 us), return value (nil)
2021.03.24 11:22:47.670356 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.670387 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 140
2021.03.24 11:22:47.670415 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.670425 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.670437 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 140 0x7faa1f070b20 sleep
2021.03.24 11:22:47.675074 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.694921 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 141
2021.03.24 11:22:47.694960 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 141 0x7faa1f070d20 sleep
2021.03.24 11:22:47.695011 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.695033 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.675303 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.695081 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=140
2021.03.24 11:22:47.677941 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.695092 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.695139 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 140
2021.03.24 11:22:47.695158 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.695167 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 139, new matched idx: 139
2021.03.24 11:22:47.695179 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 141 139 135 
2021.03.24 11:22:47.695187 [ 1093 ] {} <Trace> RaftInstance: local log idx 141, target_commit_idx 139, quick_commit_index_ 139, state_->get_commit_idx() 139
2021.03.24 11:22:47.695195 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.695203 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.695212 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 139, starting_idx: 91, cur_nxt_idx: 142
2021.03.24 11:22:47.695220 [ 1093 ] {} <Debug> RaftInstance: last sent log (140) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.695232 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=139, LastLogTerm=1, EntriesLength=2, CommitIndex=139, Term=1, peer_last_sent_idx 140
2021.03.24 11:22:47.695240 [ 1093 ] {} <Debug> RaftInstance: idx range: 140-141
2021.03.24 11:22:47.695250 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.695308 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.695353 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=140
2021.03.24 11:22:47.695372 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.695385 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 140
2021.03.24 11:22:47.695393 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.695402 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 135, new matched idx: 139
2021.03.24 11:22:47.695414 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 141 139 139 
2021.03.24 11:22:47.695422 [ 1094 ] {} <Trace> RaftInstance: local log idx 141, target_commit_idx 139, quick_commit_index_ 139, state_->get_commit_idx() 139
2021.03.24 11:22:47.695430 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.695440 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.695449 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 139, starting_idx: 91, cur_nxt_idx: 142
2021.03.24 11:22:47.695460 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=139, LastLogTerm=1, EntriesLength=2, CommitIndex=139, Term=1, peer_last_sent_idx 136
2021.03.24 11:22:47.695468 [ 1094 ] {} <Debug> RaftInstance: idx range: 140-141
2021.03.24 11:22:47.695479 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.695562 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.702003 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.702026 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=142
2021.03.24 11:22:47.702035 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.702044 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 142
2021.03.24 11:22:47.702052 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.702060 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 139, new matched idx: 141
2021.03.24 11:22:47.702070 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 141 141 139 
2021.03.24 11:22:47.702078 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 141
2021.03.24 11:22:47.702086 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.702094 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.702102 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 141, starting_idx: 91, cur_nxt_idx: 142
2021.03.24 11:22:47.702112 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=141, LastLogTerm=1, EntriesLength=0, CommitIndex=141, Term=1, peer_last_sent_idx 140
2021.03.24 11:22:47.702128 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.702137 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.702181 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.702193 [ 1093 ] {} <Trace> RaftInstance: local log idx 141, target_commit_idx 141, quick_commit_index_ 141, state_->get_commit_idx() 139
2021.03.24 11:22:47.702202 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.702240 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.702260 [ 1096 ] {} <Debug> RaftInstance: commit upto 141, current idx 139
2021.03.24 11:22:47.702271 [ 1096 ] {} <Trace> RaftInstance: commit upto 141, current idx 140
2021.03.24 11:22:47.702315 [ 1096 ] {} <Debug> RaftInstance: notify cb 140 0x7faa1f070b20
2021.03.24 11:22:47.702334 [ 1096 ] {} <Trace> RaftInstance: commit upto 141, current idx 141
2021.03.24 11:22:47.702357 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.702381 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 140 wake up (31953 us), return value (nil)
2021.03.24 11:22:47.702396 [ 1096 ] {} <Debug> RaftInstance: notify cb 141 0x7faa1f070d20
2021.03.24 11:22:47.702420 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 141, current idx 141
2021.03.24 11:22:47.702429 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.702457 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.702498 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 141 wake up (7488 us), return value 0x7faaa9a69cd0
2021.03.24 11:22:47.709282 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.709304 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=142
2021.03.24 11:22:47.709314 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.709324 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 142
2021.03.24 11:22:47.709332 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.709346 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 141, new matched idx: 141
2021.03.24 11:22:47.709357 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 141 141 139 
2021.03.24 11:22:47.709365 [ 1094 ] {} <Trace> RaftInstance: local log idx 141, target_commit_idx 141, quick_commit_index_ 141, state_->get_commit_idx() 141
2021.03.24 11:22:47.710643 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.710698 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 142
2021.03.24 11:22:47.710725 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 142 0x7faa1f070d20 sleep
2021.03.24 11:22:47.710745 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.710766 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.710794 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 141, starting_idx: 91, cur_nxt_idx: 143
2021.03.24 11:22:47.710810 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=141, LastLogTerm=1, EntriesLength=1, CommitIndex=141, Term=1, peer_last_sent_idx 142
2021.03.24 11:22:47.710842 [ 1097 ] {} <Debug> RaftInstance: idx: 142
2021.03.24 11:22:47.710853 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.710906 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.716063 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.740656 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=142
2021.03.24 11:22:47.740689 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.740703 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 142
2021.03.24 11:22:47.740712 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.740720 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 139, new matched idx: 141
2021.03.24 11:22:47.740734 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 142 141 141 
2021.03.24 11:22:47.740743 [ 1093 ] {} <Trace> RaftInstance: local log idx 142, target_commit_idx 141, quick_commit_index_ 141, state_->get_commit_idx() 141
2021.03.24 11:22:47.740751 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.740762 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.740771 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 141, starting_idx: 91, cur_nxt_idx: 143
2021.03.24 11:22:47.740783 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=141, LastLogTerm=1, EntriesLength=1, CommitIndex=141, Term=1, peer_last_sent_idx 140
2021.03.24 11:22:47.740791 [ 1093 ] {} <Debug> RaftInstance: idx: 142
2021.03.24 11:22:47.740802 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.740870 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.722880 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.740937 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 143
2021.03.24 11:22:47.717518 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.740977 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 143 0x7faa1f070b20 sleep
2021.03.24 11:22:47.740969 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.741004 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.741013 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 141, starting_idx: 91, cur_nxt_idx: 144
2021.03.24 11:22:47.741022 [ 1097 ] {} <Debug> RaftInstance: last sent log (142) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.741031 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=141, LastLogTerm=1, EntriesLength=2, CommitIndex=141, Term=1, peer_last_sent_idx 142
2021.03.24 11:22:47.741040 [ 1097 ] {} <Debug> RaftInstance: idx range: 142-143
2021.03.24 11:22:47.741049 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.741158 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.741218 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=143
2021.03.24 11:22:47.741236 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.741247 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 143
2021.03.24 11:22:47.741255 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.741263 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 141, new matched idx: 142
2021.03.24 11:22:47.741273 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 143 142 141 
2021.03.24 11:22:47.741281 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 142
2021.03.24 11:22:47.741310 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.741318 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.741326 [ 1094 ] {} <Trace> RaftInstance: local log idx 143, target_commit_idx 142, quick_commit_index_ 142, state_->get_commit_idx() 141
2021.03.24 11:22:47.741334 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.741354 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.741363 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.741383 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.741404 [ 1096 ] {} <Debug> RaftInstance: commit upto 142, current idx 141
2021.03.24 11:22:47.741415 [ 1096 ] {} <Trace> RaftInstance: commit upto 142, current idx 142
2021.03.24 11:22:47.741463 [ 1096 ] {} <Debug> RaftInstance: notify cb 142 0x7faa1f070d20
2021.03.24 11:22:47.741492 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 142, current idx 142
2021.03.24 11:22:47.741501 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.741515 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.741532 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 142 wake up (30797 us), return value (nil)
2021.03.24 11:22:47.742658 [ 1093 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:47.742675 [ 1093 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:47.742684 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.748025 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.748048 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=144
2021.03.24 11:22:47.748059 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.748070 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 144
2021.03.24 11:22:47.748078 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.748086 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 142, new matched idx: 143
2021.03.24 11:22:47.748099 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 143 143 141 
2021.03.24 11:22:47.748106 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 143
2021.03.24 11:22:47.748116 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.748124 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.748132 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 143, starting_idx: 91, cur_nxt_idx: 144
2021.03.24 11:22:47.748143 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=143, LastLogTerm=1, EntriesLength=0, CommitIndex=143, Term=1, peer_last_sent_idx 142
2021.03.24 11:22:47.748150 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.748158 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.748222 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.748245 [ 1094 ] {} <Trace> RaftInstance: local log idx 143, target_commit_idx 143, quick_commit_index_ 143, state_->get_commit_idx() 142
2021.03.24 11:22:47.748254 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.748292 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.748320 [ 1096 ] {} <Debug> RaftInstance: commit upto 143, current idx 142
2021.03.24 11:22:47.748331 [ 1096 ] {} <Trace> RaftInstance: commit upto 143, current idx 143
2021.03.24 11:22:47.748356 [ 1096 ] {} <Debug> RaftInstance: notify cb 143 0x7faa1f070b20
2021.03.24 11:22:47.748377 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 143, current idx 143
2021.03.24 11:22:47.748386 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.748394 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.748413 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 143 wake up (7437 us), return value 0x7faaa9a69ce0
2021.03.24 11:22:47.749969 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.750025 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 144
2021.03.24 11:22:47.750048 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 144 0x7faa1f070d20 sleep
2021.03.24 11:22:47.750129 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.750152 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.755102 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.755130 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=144
2021.03.24 11:22:47.755151 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.755162 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 144
2021.03.24 11:22:47.755171 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.755179 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 143, new matched idx: 143
2021.03.24 11:22:47.755190 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 144 143 141 
2021.03.24 11:22:47.755199 [ 1094 ] {} <Trace> RaftInstance: local log idx 144, target_commit_idx 143, quick_commit_index_ 143, state_->get_commit_idx() 143
2021.03.24 11:22:47.755207 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.755215 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.755223 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 143, starting_idx: 91, cur_nxt_idx: 145
2021.03.24 11:22:47.755235 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=143, LastLogTerm=1, EntriesLength=1, CommitIndex=143, Term=1, peer_last_sent_idx 144
2021.03.24 11:22:47.755242 [ 1094 ] {} <Debug> RaftInstance: idx: 144
2021.03.24 11:22:47.755251 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.755308 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.761475 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.761867 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.769229 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.787096 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 145
2021.03.24 11:22:47.787126 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 145 0x7faa0daae120 sleep
2021.03.24 11:22:47.786960 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=143
2021.03.24 11:22:47.787321 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.787341 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 143
2021.03.24 11:22:47.787350 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.768390 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.787358 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 141, new matched idx: 142
2021.03.24 11:22:47.787420 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 146
2021.03.24 11:22:47.787424 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 145 143 142 
2021.03.24 11:22:47.787443 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 146 0x7faa1f070b20 sleep
2021.03.24 11:22:47.787452 [ 1093 ] {} <Trace> RaftInstance: local log idx 146, target_commit_idx 143, quick_commit_index_ 143, state_->get_commit_idx() 143
2021.03.24 11:22:47.787462 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.787471 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.787480 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 142, starting_idx: 91, cur_nxt_idx: 147
2021.03.24 11:22:47.787492 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=142, LastLogTerm=1, EntriesLength=4, CommitIndex=143, Term=1, peer_last_sent_idx 142
2021.03.24 11:22:47.787499 [ 1093 ] {} <Debug> RaftInstance: idx range: 143-146
2021.03.24 11:22:47.787508 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.787589 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.787668 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=145
2021.03.24 11:22:47.787701 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.787713 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 145
2021.03.24 11:22:47.787721 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.787729 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 143, new matched idx: 144
2021.03.24 11:22:47.787741 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 146 144 142 
2021.03.24 11:22:47.787753 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 144
2021.03.24 11:22:47.787763 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.787771 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.787779 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 144, starting_idx: 91, cur_nxt_idx: 147
2021.03.24 11:22:47.787791 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=144, LastLogTerm=1, EntriesLength=2, CommitIndex=144, Term=1, peer_last_sent_idx 144
2021.03.24 11:22:47.787798 [ 1094 ] {} <Debug> RaftInstance: idx range: 145-146
2021.03.24 11:22:47.787810 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.787885 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.787898 [ 1094 ] {} <Trace> RaftInstance: local log idx 146, target_commit_idx 144, quick_commit_index_ 144, state_->get_commit_idx() 143
2021.03.24 11:22:47.787906 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.787925 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.787945 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.787943 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.787975 [ 1096 ] {} <Debug> RaftInstance: commit upto 144, current idx 143
2021.03.24 11:22:47.787985 [ 1096 ] {} <Trace> RaftInstance: commit upto 144, current idx 144
2021.03.24 11:22:47.787986 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.788009 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.788031 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.788040 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.788034 [ 1096 ] {} <Debug> RaftInstance: notify cb 144 0x7faa1f070d20
2021.03.24 11:22:47.788071 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 144, current idx 144
2021.03.24 11:22:47.788092 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.788104 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.788125 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 144 wake up (38059 us), return value (nil)
2021.03.24 11:22:47.794486 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.794505 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=147
2021.03.24 11:22:47.794515 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.794525 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 147
2021.03.24 11:22:47.794536 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.794543 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 144, new matched idx: 146
2021.03.24 11:22:47.794555 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 146 146 142 
2021.03.24 11:22:47.794562 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 146
2021.03.24 11:22:47.794571 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.794579 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.794587 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 146, starting_idx: 91, cur_nxt_idx: 147
2021.03.24 11:22:47.794598 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=146, LastLogTerm=1, EntriesLength=0, CommitIndex=146, Term=1, peer_last_sent_idx 145
2021.03.24 11:22:47.794605 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.794624 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.794686 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.794700 [ 1093 ] {} <Trace> RaftInstance: local log idx 146, target_commit_idx 146, quick_commit_index_ 146, state_->get_commit_idx() 144
2021.03.24 11:22:47.794709 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.794742 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.794755 [ 1096 ] {} <Debug> RaftInstance: commit upto 146, current idx 144
2021.03.24 11:22:47.794764 [ 1096 ] {} <Trace> RaftInstance: commit upto 146, current idx 145
2021.03.24 11:22:47.794815 [ 1096 ] {} <Debug> RaftInstance: notify cb 145 0x7faa0daae120
2021.03.24 11:22:47.794860 [ 1096 ] {} <Trace> RaftInstance: commit upto 146, current idx 146
2021.03.24 11:22:47.794889 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.794916 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 145 wake up (7766 us), return value (nil)
2021.03.24 11:22:47.794957 [ 1096 ] {} <Debug> RaftInstance: notify cb 146 0x7faa1f070b20
2021.03.24 11:22:47.794981 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 146, current idx 146
2021.03.24 11:22:47.794990 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.795001 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.795021 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 146 wake up (7553 us), return value (nil)
2021.03.24 11:22:47.795533 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.795558 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 147
2021.03.24 11:22:47.795580 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 147 0x7faa0daae020 sleep
2021.03.24 11:22:47.795598 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.795614 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.796405 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.796430 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 148
2021.03.24 11:22:47.796450 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 148 0x7faa1f070d20 sleep
2021.03.24 11:22:47.796463 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.796476 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.801337 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.801361 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=147
2021.03.24 11:22:47.801372 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.801406 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 147
2021.03.24 11:22:47.801417 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.801425 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 146, new matched idx: 146
2021.03.24 11:22:47.801437 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 148 146 142 
2021.03.24 11:22:47.801446 [ 1093 ] {} <Trace> RaftInstance: local log idx 148, target_commit_idx 146, quick_commit_index_ 146, state_->get_commit_idx() 146
2021.03.24 11:22:47.801455 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.801463 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.801472 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 146, starting_idx: 91, cur_nxt_idx: 149
2021.03.24 11:22:47.801480 [ 1093 ] {} <Debug> RaftInstance: last sent log (147) to peer 1 is not applied, cnt 1
2021.03.24 11:22:47.801491 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=146, LastLogTerm=1, EntriesLength=2, CommitIndex=146, Term=1, peer_last_sent_idx 147
2021.03.24 11:22:47.801506 [ 1093 ] {} <Debug> RaftInstance: idx range: 147-148
2021.03.24 11:22:47.801516 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.801574 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.808450 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.808509 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=149
2021.03.24 11:22:47.808550 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.808567 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 149
2021.03.24 11:22:47.808605 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.808567 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.808618 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 146, new matched idx: 148
2021.03.24 11:22:47.808651 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 148 148 142 
2021.03.24 11:22:47.808660 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 148
2021.03.24 11:22:47.808670 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.808678 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 142, starting_idx: 91, cur_nxt_idx: 149
2021.03.24 11:22:47.808687 [ 1094 ] {} <Debug> RaftInstance: last sent log (143) to peer 3 is not applied, cnt 1
2021.03.24 11:22:47.808700 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=142, LastLogTerm=1, EntriesLength=6, CommitIndex=148, Term=1, peer_last_sent_idx 143
2021.03.24 11:22:47.808708 [ 1094 ] {} <Debug> RaftInstance: idx range: 143-148
2021.03.24 11:22:47.808717 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.808787 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.808799 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.808807 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 148, starting_idx: 91, cur_nxt_idx: 149
2021.03.24 11:22:47.808817 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=148, LastLogTerm=1, EntriesLength=0, CommitIndex=148, Term=1, peer_last_sent_idx 147
2021.03.24 11:22:47.808824 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.808832 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.808866 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.808879 [ 1094 ] {} <Trace> RaftInstance: local log idx 148, target_commit_idx 148, quick_commit_index_ 148, state_->get_commit_idx() 146
2021.03.24 11:22:47.808887 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.808917 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=147
2021.03.24 11:22:47.808944 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.808967 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 147
2021.03.24 11:22:47.808928 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.808990 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.809013 [ 1096 ] {} <Debug> RaftInstance: commit upto 148, current idx 146
2021.03.24 11:22:47.809021 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 142, new matched idx: 146
2021.03.24 11:22:47.809025 [ 1096 ] {} <Trace> RaftInstance: commit upto 148, current idx 147
2021.03.24 11:22:47.809033 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 148 148 146 
2021.03.24 11:22:47.809042 [ 1093 ] {} <Trace> RaftInstance: local log idx 148, target_commit_idx 148, quick_commit_index_ 148, state_->get_commit_idx() 146
2021.03.24 11:22:47.809050 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.809060 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.809070 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.809159 [ 1096 ] {} <Debug> RaftInstance: notify cb 147 0x7faa0daae020
2021.03.24 11:22:47.809179 [ 1096 ] {} <Trace> RaftInstance: commit upto 148, current idx 148
2021.03.24 11:22:47.809201 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.809258 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 147 wake up (13625 us), return value (nil)
2021.03.24 11:22:47.809269 [ 1096 ] {} <Debug> RaftInstance: notify cb 148 0x7faa1f070d20
2021.03.24 11:22:47.809294 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 148, current idx 148
2021.03.24 11:22:47.809302 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.809373 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.809421 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 148 wake up (12903 us), return value 0x7faaa9a69cf0
2021.03.24 11:22:47.814983 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.815040 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 149
2021.03.24 11:22:47.815068 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.815078 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.815090 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 149 0x7faa1f070b20 sleep
2021.03.24 11:22:47.815663 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.815680 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=149
2021.03.24 11:22:47.815689 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.815698 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 149
2021.03.24 11:22:47.815706 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.815713 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 148, new matched idx: 148
2021.03.24 11:22:47.815723 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 149 148 146 
2021.03.24 11:22:47.815731 [ 1093 ] {} <Trace> RaftInstance: local log idx 149, target_commit_idx 148, quick_commit_index_ 148, state_->get_commit_idx() 148
2021.03.24 11:22:47.815739 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.815747 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.815755 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 148, starting_idx: 91, cur_nxt_idx: 150
2021.03.24 11:22:47.815767 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=148, LastLogTerm=1, EntriesLength=1, CommitIndex=148, Term=1, peer_last_sent_idx 149
2021.03.24 11:22:47.815787 [ 1093 ] {} <Debug> RaftInstance: idx: 149
2021.03.24 11:22:47.815796 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.815839 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.817884 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.817963 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 150
2021.03.24 11:22:47.817986 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 150 0x7faa1f070d20 sleep
2021.03.24 11:22:47.818001 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.818030 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.822642 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.822663 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=150
2021.03.24 11:22:47.822680 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.822690 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 150
2021.03.24 11:22:47.822710 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.822719 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 148, new matched idx: 149
2021.03.24 11:22:47.822731 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 150 149 146 
2021.03.24 11:22:47.822739 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 149
2021.03.24 11:22:47.822747 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.822767 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.822818 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 149, starting_idx: 91, cur_nxt_idx: 151
2021.03.24 11:22:47.822834 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=149, LastLogTerm=1, EntriesLength=1, CommitIndex=149, Term=1, peer_last_sent_idx 149
2021.03.24 11:22:47.822842 [ 1093 ] {} <Debug> RaftInstance: idx: 150
2021.03.24 11:22:47.822851 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.822918 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.822944 [ 1093 ] {} <Trace> RaftInstance: local log idx 150, target_commit_idx 149, quick_commit_index_ 149, state_->get_commit_idx() 148
2021.03.24 11:22:47.822952 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.822968 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.822978 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.822979 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.823017 [ 1096 ] {} <Debug> RaftInstance: commit upto 149, current idx 148
2021.03.24 11:22:47.823029 [ 1096 ] {} <Trace> RaftInstance: commit upto 149, current idx 149
2021.03.24 11:22:47.823066 [ 1096 ] {} <Debug> RaftInstance: notify cb 149 0x7faa1f070b20
2021.03.24 11:22:47.823086 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 149, current idx 149
2021.03.24 11:22:47.823094 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.823102 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.823116 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 149 wake up (8047 us), return value (nil)
2021.03.24 11:22:47.829389 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.829415 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=149
2021.03.24 11:22:47.829424 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.829434 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 149
2021.03.24 11:22:47.829443 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.829450 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 146, new matched idx: 148
2021.03.24 11:22:47.829461 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 150 149 148 
2021.03.24 11:22:47.829469 [ 1094 ] {} <Trace> RaftInstance: local log idx 150, target_commit_idx 149, quick_commit_index_ 149, state_->get_commit_idx() 149
2021.03.24 11:22:47.829477 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.829485 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.829493 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 148, starting_idx: 91, cur_nxt_idx: 151
2021.03.24 11:22:47.829506 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=148, LastLogTerm=1, EntriesLength=2, CommitIndex=149, Term=1, peer_last_sent_idx 143
2021.03.24 11:22:47.829514 [ 1094 ] {} <Debug> RaftInstance: idx range: 149-150
2021.03.24 11:22:47.829523 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.829571 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.829637 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.829669 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=151
2021.03.24 11:22:47.829679 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.829689 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 151
2021.03.24 11:22:47.829697 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.829704 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 149, new matched idx: 150
2021.03.24 11:22:47.829714 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 150 150 148 
2021.03.24 11:22:47.829721 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 150
2021.03.24 11:22:47.829729 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.829739 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.829747 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 150, starting_idx: 91, cur_nxt_idx: 151
2021.03.24 11:22:47.829757 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=150, LastLogTerm=1, EntriesLength=0, CommitIndex=150, Term=1, peer_last_sent_idx 150
2021.03.24 11:22:47.829764 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.829772 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.829807 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.829820 [ 1092 ] {} <Trace> RaftInstance: local log idx 150, target_commit_idx 150, quick_commit_index_ 150, state_->get_commit_idx() 149
2021.03.24 11:22:47.829828 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.829855 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.829877 [ 1096 ] {} <Debug> RaftInstance: commit upto 150, current idx 149
2021.03.24 11:22:47.829886 [ 1096 ] {} <Trace> RaftInstance: commit upto 150, current idx 150
2021.03.24 11:22:47.829909 [ 1096 ] {} <Debug> RaftInstance: notify cb 150 0x7faa1f070d20
2021.03.24 11:22:47.829928 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 150, current idx 150
2021.03.24 11:22:47.829942 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.830056 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.830115 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 150 wake up (12045 us), return value 0x7faaa9a69d00
2021.03.24 11:22:47.833553 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.833592 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 151
2021.03.24 11:22:47.833624 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 151 0x7faa0daae120 sleep
2021.03.24 11:22:47.833643 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.833671 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.836463 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.836484 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=151
2021.03.24 11:22:47.836494 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.836503 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 151
2021.03.24 11:22:47.836511 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.836526 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 150, new matched idx: 150
2021.03.24 11:22:47.836537 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 151 150 148 
2021.03.24 11:22:47.836545 [ 1094 ] {} <Trace> RaftInstance: local log idx 151, target_commit_idx 150, quick_commit_index_ 150, state_->get_commit_idx() 150
2021.03.24 11:22:47.836567 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.836575 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.836584 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 150, starting_idx: 91, cur_nxt_idx: 152
2021.03.24 11:22:47.836603 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=150, LastLogTerm=1, EntriesLength=1, CommitIndex=150, Term=1, peer_last_sent_idx 151
2021.03.24 11:22:47.836611 [ 1094 ] {} <Debug> RaftInstance: idx: 151
2021.03.24 11:22:47.836621 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.836676 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.838854 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.838884 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 152
2021.03.24 11:22:47.838910 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.838920 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.838940 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 152 0x7faa1f15de20 sleep
2021.03.24 11:22:47.843186 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.843291 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.843328 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=152
2021.03.24 11:22:47.843339 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.843349 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 152
2021.03.24 11:22:47.843357 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.843365 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 150, new matched idx: 151
2021.03.24 11:22:47.843376 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 152 151 148 
2021.03.24 11:22:47.843302 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 153
2021.03.24 11:22:47.843385 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 151
2021.03.24 11:22:47.843413 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.843416 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 153 0x7faa1f070d20 sleep
2021.03.24 11:22:47.843427 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.843453 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 151, starting_idx: 91, cur_nxt_idx: 154
2021.03.24 11:22:47.843470 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=151, LastLogTerm=1, EntriesLength=2, CommitIndex=151, Term=1, peer_last_sent_idx 151
2021.03.24 11:22:47.843477 [ 1094 ] {} <Debug> RaftInstance: idx range: 152-153
2021.03.24 11:22:47.843486 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.843547 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.843559 [ 1094 ] {} <Trace> RaftInstance: local log idx 153, target_commit_idx 151, quick_commit_index_ 151, state_->get_commit_idx() 150
2021.03.24 11:22:47.843567 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.843579 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.843587 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.843614 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.843626 [ 1096 ] {} <Debug> RaftInstance: commit upto 151, current idx 150
2021.03.24 11:22:47.843643 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.843648 [ 1096 ] {} <Trace> RaftInstance: commit upto 151, current idx 151
2021.03.24 11:22:47.843669 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.843740 [ 1096 ] {} <Debug> RaftInstance: notify cb 151 0x7faa0daae120
2021.03.24 11:22:47.843863 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 151, current idx 151
2021.03.24 11:22:47.843875 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.843928 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.843950 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 151 wake up (10309 us), return value (nil)
2021.03.24 11:22:47.844515 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.844544 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 154
2021.03.24 11:22:47.844567 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 154 0x7faa0daae020 sleep
2021.03.24 11:22:47.844588 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.844604 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.850405 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.850438 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=154
2021.03.24 11:22:47.850447 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.850457 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 154
2021.03.24 11:22:47.850466 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.850473 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 151, new matched idx: 153
2021.03.24 11:22:47.850483 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 154 153 148 
2021.03.24 11:22:47.850491 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 153
2021.03.24 11:22:47.850499 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.850507 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.850515 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 153, starting_idx: 91, cur_nxt_idx: 155
2021.03.24 11:22:47.850525 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=153, LastLogTerm=1, EntriesLength=1, CommitIndex=153, Term=1, peer_last_sent_idx 152
2021.03.24 11:22:47.850533 [ 1094 ] {} <Debug> RaftInstance: idx: 154
2021.03.24 11:22:47.850542 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.850597 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.850610 [ 1094 ] {} <Trace> RaftInstance: local log idx 154, target_commit_idx 153, quick_commit_index_ 153, state_->get_commit_idx() 151
2021.03.24 11:22:47.850618 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.850630 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.850639 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.850684 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.850734 [ 1096 ] {} <Debug> RaftInstance: commit upto 153, current idx 151
2021.03.24 11:22:47.850747 [ 1096 ] {} <Trace> RaftInstance: commit upto 153, current idx 152
2021.03.24 11:22:47.850839 [ 1096 ] {} <Debug> RaftInstance: notify cb 152 0x7faa1f15de20
2021.03.24 11:22:47.850863 [ 1096 ] {} <Trace> RaftInstance: commit upto 153, current idx 153
2021.03.24 11:22:47.850895 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.850927 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 152 wake up (11992 us), return value (nil)
2021.03.24 11:22:47.850941 [ 1096 ] {} <Debug> RaftInstance: notify cb 153 0x7faa1f070d20
2021.03.24 11:22:47.850963 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 153, current idx 153
2021.03.24 11:22:47.850971 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.851005 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.851022 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 153 wake up (7595 us), return value (nil)
2021.03.24 11:22:47.857254 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.857307 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=155
2021.03.24 11:22:47.857319 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.857330 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 155
2021.03.24 11:22:47.857338 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.857345 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 153, new matched idx: 154
2021.03.24 11:22:47.857357 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 154 154 148 
2021.03.24 11:22:47.857364 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 154
2021.03.24 11:22:47.857373 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.857381 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.857389 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 154, starting_idx: 91, cur_nxt_idx: 155
2021.03.24 11:22:47.857399 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=154, LastLogTerm=1, EntriesLength=0, CommitIndex=154, Term=1, peer_last_sent_idx 154
2021.03.24 11:22:47.857406 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.857414 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.857481 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.857495 [ 1092 ] {} <Trace> RaftInstance: local log idx 154, target_commit_idx 154, quick_commit_index_ 154, state_->get_commit_idx() 153
2021.03.24 11:22:47.857502 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.857530 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.857540 [ 1096 ] {} <Debug> RaftInstance: commit upto 154, current idx 153
2021.03.24 11:22:47.857549 [ 1096 ] {} <Trace> RaftInstance: commit upto 154, current idx 154
2021.03.24 11:22:47.857609 [ 1096 ] {} <Debug> RaftInstance: notify cb 154 0x7faa0daae020
2021.03.24 11:22:47.857626 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 154, current idx 154
2021.03.24 11:22:47.857633 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.857656 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.857685 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 154 wake up (13090 us), return value (nil)
2021.03.24 11:22:47.859200 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.859237 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 155
2021.03.24 11:22:47.859268 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.859278 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.859290 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 155 0x7faa1f15de20 sleep
2021.03.24 11:22:47.864153 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.864178 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=155
2021.03.24 11:22:47.864189 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.864198 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 155
2021.03.24 11:22:47.864207 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.864233 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 154, new matched idx: 154
2021.03.24 11:22:47.864244 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 155 154 148 
2021.03.24 11:22:47.864253 [ 1094 ] {} <Trace> RaftInstance: local log idx 155, target_commit_idx 154, quick_commit_index_ 154, state_->get_commit_idx() 154
2021.03.24 11:22:47.864261 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.864269 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.864277 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 154, starting_idx: 91, cur_nxt_idx: 156
2021.03.24 11:22:47.864289 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=154, LastLogTerm=1, EntriesLength=1, CommitIndex=154, Term=1, peer_last_sent_idx 155
2021.03.24 11:22:47.864296 [ 1094 ] {} <Debug> RaftInstance: idx: 155
2021.03.24 11:22:47.864305 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.864357 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.870980 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.871001 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=156
2021.03.24 11:22:47.871010 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.871009 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.871020 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 156
2021.03.24 11:22:47.871086 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 156
2021.03.24 11:22:47.871096 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.871108 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 154, new matched idx: 155
2021.03.24 11:22:47.871113 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 156 0x7faa1f15df20 sleep
2021.03.24 11:22:47.871120 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 156 155 148 
2021.03.24 11:22:47.871130 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 155
2021.03.24 11:22:47.871139 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.871147 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.871155 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 155, starting_idx: 91, cur_nxt_idx: 157
2021.03.24 11:22:47.871165 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=155, LastLogTerm=1, EntriesLength=1, CommitIndex=155, Term=1, peer_last_sent_idx 155
2021.03.24 11:22:47.871173 [ 1094 ] {} <Debug> RaftInstance: idx: 156
2021.03.24 11:22:47.871181 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.871225 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.871238 [ 1094 ] {} <Trace> RaftInstance: local log idx 156, target_commit_idx 155, quick_commit_index_ 155, state_->get_commit_idx() 154
2021.03.24 11:22:47.871246 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.871262 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.871271 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.871324 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.871336 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.871364 [ 1096 ] {} <Debug> RaftInstance: commit upto 155, current idx 154
2021.03.24 11:22:47.871386 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.871404 [ 1096 ] {} <Trace> RaftInstance: commit upto 155, current idx 155
2021.03.24 11:22:47.871487 [ 1096 ] {} <Debug> RaftInstance: notify cb 155 0x7faa1f15de20
2021.03.24 11:22:47.871507 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 155, current idx 155
2021.03.24 11:22:47.871516 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.871525 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.871554 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 155 wake up (12272 us), return value 0x7faaa9a69d10
2021.03.24 11:22:47.877983 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.878004 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=157
2021.03.24 11:22:47.878013 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.878022 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 157
2021.03.24 11:22:47.878037 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.878045 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 155, new matched idx: 156
2021.03.24 11:22:47.878055 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 156 156 148 
2021.03.24 11:22:47.878062 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 156
2021.03.24 11:22:47.878071 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.878078 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.878086 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 156, starting_idx: 91, cur_nxt_idx: 157
2021.03.24 11:22:47.878096 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=156, LastLogTerm=1, EntriesLength=0, CommitIndex=156, Term=1, peer_last_sent_idx 156
2021.03.24 11:22:47.878103 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.878111 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.878150 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.878163 [ 1095 ] {} <Trace> RaftInstance: local log idx 156, target_commit_idx 156, quick_commit_index_ 156, state_->get_commit_idx() 155
2021.03.24 11:22:47.878170 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.878255 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.878302 [ 1096 ] {} <Debug> RaftInstance: commit upto 156, current idx 155
2021.03.24 11:22:47.878313 [ 1096 ] {} <Trace> RaftInstance: commit upto 156, current idx 156
2021.03.24 11:22:47.878365 [ 1096 ] {} <Debug> RaftInstance: notify cb 156 0x7faa1f15df20
2021.03.24 11:22:47.878383 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 156, current idx 156
2021.03.24 11:22:47.878390 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.878407 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.878424 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 156 wake up (7299 us), return value (nil)
2021.03.24 11:22:47.879924 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.879955 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 157
2021.03.24 11:22:47.879982 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.879991 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.880003 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 157 0x7faa1f15de20 sleep
2021.03.24 11:22:47.884735 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.884754 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=157
2021.03.24 11:22:47.884763 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.884787 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 157
2021.03.24 11:22:47.884795 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.884803 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 156, new matched idx: 156
2021.03.24 11:22:47.884812 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 157 156 148 
2021.03.24 11:22:47.884820 [ 1094 ] {} <Trace> RaftInstance: local log idx 157, target_commit_idx 156, quick_commit_index_ 156, state_->get_commit_idx() 156
2021.03.24 11:22:47.884828 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.884836 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.884844 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 156, starting_idx: 91, cur_nxt_idx: 158
2021.03.24 11:22:47.884854 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=156, LastLogTerm=1, EntriesLength=1, CommitIndex=156, Term=1, peer_last_sent_idx 157
2021.03.24 11:22:47.884861 [ 1094 ] {} <Debug> RaftInstance: idx: 157
2021.03.24 11:22:47.884870 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.884918 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.891558 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.891596 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=158
2021.03.24 11:22:47.891605 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.891615 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 158
2021.03.24 11:22:47.891623 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.891631 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 156, new matched idx: 157
2021.03.24 11:22:47.891641 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 157 157 148 
2021.03.24 11:22:47.891648 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 157
2021.03.24 11:22:47.891657 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.891665 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.891672 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 157, starting_idx: 91, cur_nxt_idx: 158
2021.03.24 11:22:47.891683 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=157, LastLogTerm=1, EntriesLength=0, CommitIndex=157, Term=1, peer_last_sent_idx 157
2021.03.24 11:22:47.891690 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.891697 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.891764 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.891777 [ 1095 ] {} <Trace> RaftInstance: local log idx 157, target_commit_idx 157, quick_commit_index_ 157, state_->get_commit_idx() 156
2021.03.24 11:22:47.891786 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.891880 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.891941 [ 1096 ] {} <Debug> RaftInstance: commit upto 157, current idx 156
2021.03.24 11:22:47.891973 [ 1096 ] {} <Trace> RaftInstance: commit upto 157, current idx 157
2021.03.24 11:22:47.892024 [ 1096 ] {} <Debug> RaftInstance: notify cb 157 0x7faa1f15de20
2021.03.24 11:22:47.892056 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 157, current idx 157
2021.03.24 11:22:47.892065 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.892086 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.892108 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 157 wake up (12111 us), return value (nil)
2021.03.24 11:22:47.898290 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.898316 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=158
2021.03.24 11:22:47.898341 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.898351 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 158
2021.03.24 11:22:47.898359 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.898367 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 157, new matched idx: 157
2021.03.24 11:22:47.898377 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 157 157 148 
2021.03.24 11:22:47.898386 [ 1092 ] {} <Trace> RaftInstance: local log idx 157, target_commit_idx 157, quick_commit_index_ 157, state_->get_commit_idx() 157
2021.03.24 11:22:47.898428 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.898467 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 158
2021.03.24 11:22:47.898485 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 158 0x7faa1f057020 sleep
2021.03.24 11:22:47.898517 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.898537 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.898559 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 157, starting_idx: 91, cur_nxt_idx: 159
2021.03.24 11:22:47.898572 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=157, LastLogTerm=1, EntriesLength=1, CommitIndex=157, Term=1, peer_last_sent_idx 158
2021.03.24 11:22:47.898580 [ 1097 ] {} <Debug> RaftInstance: idx: 158
2021.03.24 11:22:47.898591 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.898653 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.900388 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.900415 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 159
2021.03.24 11:22:47.900440 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.900450 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.900461 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 159 0x7faa1f15de20 sleep
2021.03.24 11:22:47.903147 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.903165 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=151
2021.03.24 11:22:47.903174 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.903183 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 151
2021.03.24 11:22:47.903191 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.903198 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 148, new matched idx: 150
2021.03.24 11:22:47.903207 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 159 157 150 
2021.03.24 11:22:47.903215 [ 1094 ] {} <Trace> RaftInstance: local log idx 159, target_commit_idx 157, quick_commit_index_ 157, state_->get_commit_idx() 157
2021.03.24 11:22:47.903222 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.903242 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.903251 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 150, starting_idx: 91, cur_nxt_idx: 160
2021.03.24 11:22:47.903262 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=150, LastLogTerm=1, EntriesLength=9, CommitIndex=157, Term=1, peer_last_sent_idx 149
2021.03.24 11:22:47.903270 [ 1094 ] {} <Debug> RaftInstance: idx range: 151-159
2021.03.24 11:22:47.903278 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.903324 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.905369 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.905398 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=159
2021.03.24 11:22:47.905407 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.905416 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 159
2021.03.24 11:22:47.905424 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.905431 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 157, new matched idx: 158
2021.03.24 11:22:47.905440 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 159 158 150 
2021.03.24 11:22:47.905447 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 158
2021.03.24 11:22:47.905455 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.905463 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.905471 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 158, starting_idx: 91, cur_nxt_idx: 160
2021.03.24 11:22:47.905480 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=158, LastLogTerm=1, EntriesLength=1, CommitIndex=158, Term=1, peer_last_sent_idx 158
2021.03.24 11:22:47.905487 [ 1094 ] {} <Debug> RaftInstance: idx: 159
2021.03.24 11:22:47.905496 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.905529 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.905541 [ 1094 ] {} <Trace> RaftInstance: local log idx 159, target_commit_idx 158, quick_commit_index_ 158, state_->get_commit_idx() 157
2021.03.24 11:22:47.905548 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.905565 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.905575 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.905641 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.905702 [ 1096 ] {} <Debug> RaftInstance: commit upto 158, current idx 157
2021.03.24 11:22:47.905715 [ 1096 ] {} <Trace> RaftInstance: commit upto 158, current idx 158
2021.03.24 11:22:47.905768 [ 1096 ] {} <Debug> RaftInstance: notify cb 158 0x7faa1f057020
2021.03.24 11:22:47.905786 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 158, current idx 158
2021.03.24 11:22:47.905793 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.905801 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.905818 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 158 wake up (7319 us), return value (nil)
2021.03.24 11:22:47.912295 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.912313 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=160
2021.03.24 11:22:47.912321 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.912330 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 160
2021.03.24 11:22:47.912337 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.912345 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 158, new matched idx: 159
2021.03.24 11:22:47.912354 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 159 159 150 
2021.03.24 11:22:47.912361 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 159
2021.03.24 11:22:47.912369 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.912376 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.912384 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 159, starting_idx: 91, cur_nxt_idx: 160
2021.03.24 11:22:47.912393 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=159, LastLogTerm=1, EntriesLength=0, CommitIndex=159, Term=1, peer_last_sent_idx 159
2021.03.24 11:22:47.912409 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.912417 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.912466 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.912479 [ 1094 ] {} <Trace> RaftInstance: local log idx 159, target_commit_idx 159, quick_commit_index_ 159, state_->get_commit_idx() 158
2021.03.24 11:22:47.912486 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.912563 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.912617 [ 1096 ] {} <Debug> RaftInstance: commit upto 159, current idx 158
2021.03.24 11:22:47.912629 [ 1096 ] {} <Trace> RaftInstance: commit upto 159, current idx 159
2021.03.24 11:22:47.912665 [ 1096 ] {} <Debug> RaftInstance: notify cb 159 0x7faa1f15de20
2021.03.24 11:22:47.912687 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 159, current idx 159
2021.03.24 11:22:47.912696 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.912724 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.912746 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 159 wake up (12290 us), return value (nil)
2021.03.24 11:22:47.919252 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.919274 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=160
2021.03.24 11:22:47.919283 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.919293 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 160
2021.03.24 11:22:47.919301 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.919308 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 159, new matched idx: 159
2021.03.24 11:22:47.919318 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 159 159 150 
2021.03.24 11:22:47.919326 [ 1094 ] {} <Trace> RaftInstance: local log idx 159, target_commit_idx 159, quick_commit_index_ 159, state_->get_commit_idx() 159
2021.03.24 11:22:47.921183 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.921214 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 160
2021.03.24 11:22:47.921237 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 160 0x7faa1f09bb20 sleep
2021.03.24 11:22:47.921267 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.921286 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.921296 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 159, starting_idx: 91, cur_nxt_idx: 161
2021.03.24 11:22:47.921308 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=159, LastLogTerm=1, EntriesLength=1, CommitIndex=159, Term=1, peer_last_sent_idx 160
2021.03.24 11:22:47.921316 [ 1097 ] {} <Debug> RaftInstance: idx: 160
2021.03.24 11:22:47.921326 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.921378 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.923891 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.923910 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=160
2021.03.24 11:22:47.923919 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.923928 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 160
2021.03.24 11:22:47.923940 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.923948 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 150, new matched idx: 159
2021.03.24 11:22:47.923958 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 160 159 159 
2021.03.24 11:22:47.923981 [ 1093 ] {} <Trace> RaftInstance: local log idx 160, target_commit_idx 159, quick_commit_index_ 159, state_->get_commit_idx() 159
2021.03.24 11:22:47.923989 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.923997 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.924005 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 159, starting_idx: 91, cur_nxt_idx: 161
2021.03.24 11:22:47.924015 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=159, LastLogTerm=1, EntriesLength=1, CommitIndex=159, Term=1, peer_last_sent_idx 151
2021.03.24 11:22:47.924023 [ 1093 ] {} <Debug> RaftInstance: idx: 160
2021.03.24 11:22:47.924031 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.924073 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.925581 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.925604 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 161
2021.03.24 11:22:47.925626 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.925648 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.925660 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 161 0x7faa1f070d20 sleep
2021.03.24 11:22:47.928203 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.928223 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=161
2021.03.24 11:22:47.928233 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.928242 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 161
2021.03.24 11:22:47.928249 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.928260 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 159, new matched idx: 160
2021.03.24 11:22:47.928270 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 161 160 159 
2021.03.24 11:22:47.928278 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 160
2021.03.24 11:22:47.928286 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.928294 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.928302 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 160, starting_idx: 91, cur_nxt_idx: 162
2021.03.24 11:22:47.928312 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=160, LastLogTerm=1, EntriesLength=1, CommitIndex=160, Term=1, peer_last_sent_idx 160
2021.03.24 11:22:47.928319 [ 1092 ] {} <Debug> RaftInstance: idx: 161
2021.03.24 11:22:47.928328 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.928366 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.928382 [ 1092 ] {} <Trace> RaftInstance: local log idx 161, target_commit_idx 160, quick_commit_index_ 160, state_->get_commit_idx() 159
2021.03.24 11:22:47.928392 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.928409 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.928418 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.928473 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.928535 [ 1096 ] {} <Debug> RaftInstance: commit upto 160, current idx 159
2021.03.24 11:22:47.928547 [ 1096 ] {} <Trace> RaftInstance: commit upto 160, current idx 160
2021.03.24 11:22:47.928587 [ 1096 ] {} <Debug> RaftInstance: notify cb 160 0x7faa1f09bb20
2021.03.24 11:22:47.928605 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 160, current idx 160
2021.03.24 11:22:47.928614 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.928624 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.928652 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 160 wake up (7392 us), return value (nil)
2021.03.24 11:22:47.935630 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.935658 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=162
2021.03.24 11:22:47.935668 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.935678 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 162
2021.03.24 11:22:47.935686 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.935694 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 160, new matched idx: 161
2021.03.24 11:22:47.935705 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 161 161 159 
2021.03.24 11:22:47.935713 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 161
2021.03.24 11:22:47.935721 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.935729 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.935737 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 161, starting_idx: 91, cur_nxt_idx: 162
2021.03.24 11:22:47.935747 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=161, LastLogTerm=1, EntriesLength=0, CommitIndex=161, Term=1, peer_last_sent_idx 161
2021.03.24 11:22:47.935754 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.935762 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.935809 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.935822 [ 1092 ] {} <Trace> RaftInstance: local log idx 161, target_commit_idx 161, quick_commit_index_ 161, state_->get_commit_idx() 160
2021.03.24 11:22:47.935829 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.935906 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.935960 [ 1096 ] {} <Debug> RaftInstance: commit upto 161, current idx 160
2021.03.24 11:22:47.935972 [ 1096 ] {} <Trace> RaftInstance: commit upto 161, current idx 161
2021.03.24 11:22:47.936009 [ 1096 ] {} <Debug> RaftInstance: notify cb 161 0x7faa1f070d20
2021.03.24 11:22:47.936030 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 161, current idx 161
2021.03.24 11:22:47.936039 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.936062 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.936083 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 161 wake up (10441 us), return value (nil)
2021.03.24 11:22:47.936896 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.936924 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 162
2021.03.24 11:22:47.936944 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 162 0x7faa1f057020 sleep
2021.03.24 11:22:47.936964 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.936981 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.942580 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.942599 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=162
2021.03.24 11:22:47.942608 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.942616 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 162
2021.03.24 11:22:47.942624 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.942632 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 161, new matched idx: 161
2021.03.24 11:22:47.942641 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 162 161 159 
2021.03.24 11:22:47.942650 [ 1094 ] {} <Trace> RaftInstance: local log idx 162, target_commit_idx 161, quick_commit_index_ 161, state_->get_commit_idx() 161
2021.03.24 11:22:47.942672 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.942681 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.942689 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 161, starting_idx: 91, cur_nxt_idx: 163
2021.03.24 11:22:47.942699 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=161, LastLogTerm=1, EntriesLength=1, CommitIndex=161, Term=1, peer_last_sent_idx 162
2021.03.24 11:22:47.942707 [ 1094 ] {} <Debug> RaftInstance: idx: 162
2021.03.24 11:22:47.942716 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.942754 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.944666 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.944687 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=161
2021.03.24 11:22:47.944697 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.944707 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 161
2021.03.24 11:22:47.944719 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.944727 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 159, new matched idx: 160
2021.03.24 11:22:47.944737 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 162 161 160 
2021.03.24 11:22:47.944746 [ 1093 ] {} <Trace> RaftInstance: local log idx 162, target_commit_idx 161, quick_commit_index_ 161, state_->get_commit_idx() 161
2021.03.24 11:22:47.944754 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.944762 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.944770 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 160, starting_idx: 91, cur_nxt_idx: 163
2021.03.24 11:22:47.944780 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=160, LastLogTerm=1, EntriesLength=2, CommitIndex=161, Term=1, peer_last_sent_idx 160
2021.03.24 11:22:47.944788 [ 1093 ] {} <Debug> RaftInstance: idx range: 161-162
2021.03.24 11:22:47.944796 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.944842 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.944845 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.944884 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 163
2021.03.24 11:22:47.944908 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 163 0x7faa0daae120 sleep
2021.03.24 11:22:47.944927 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.944950 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.949534 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.949553 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=163
2021.03.24 11:22:47.949562 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.949570 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 163
2021.03.24 11:22:47.949578 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.949586 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 161, new matched idx: 162
2021.03.24 11:22:47.949595 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 163 162 160 
2021.03.24 11:22:47.949602 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 162
2021.03.24 11:22:47.949610 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.949618 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.949634 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 162, starting_idx: 91, cur_nxt_idx: 164
2021.03.24 11:22:47.949645 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=162, LastLogTerm=1, EntriesLength=1, CommitIndex=162, Term=1, peer_last_sent_idx 162
2021.03.24 11:22:47.949652 [ 1094 ] {} <Debug> RaftInstance: idx: 163
2021.03.24 11:22:47.949660 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.949700 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.949713 [ 1094 ] {} <Trace> RaftInstance: local log idx 163, target_commit_idx 162, quick_commit_index_ 162, state_->get_commit_idx() 161
2021.03.24 11:22:47.949721 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.949735 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.949744 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.949793 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.949858 [ 1096 ] {} <Debug> RaftInstance: commit upto 162, current idx 161
2021.03.24 11:22:47.949879 [ 1096 ] {} <Trace> RaftInstance: commit upto 162, current idx 162
2021.03.24 11:22:47.949976 [ 1096 ] {} <Debug> RaftInstance: notify cb 162 0x7faa1f057020
2021.03.24 11:22:47.949995 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 162, current idx 162
2021.03.24 11:22:47.950003 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.950008 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.950025 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 162 wake up (13069 us), return value (nil)
2021.03.24 11:22:47.956079 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.956131 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 164
2021.03.24 11:22:47.956159 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.956169 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.956180 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 164 0x7faa1f15de20 sleep
2021.03.24 11:22:47.956462 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.956498 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=164
2021.03.24 11:22:47.956509 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.956518 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 164
2021.03.24 11:22:47.956526 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.956534 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 162, new matched idx: 163
2021.03.24 11:22:47.956544 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 164 163 160 
2021.03.24 11:22:47.956552 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 163
2021.03.24 11:22:47.956564 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.956572 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.956580 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 163, starting_idx: 91, cur_nxt_idx: 165
2021.03.24 11:22:47.956590 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=163, LastLogTerm=1, EntriesLength=1, CommitIndex=163, Term=1, peer_last_sent_idx 163
2021.03.24 11:22:47.956598 [ 1095 ] {} <Debug> RaftInstance: idx: 164
2021.03.24 11:22:47.956607 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.956647 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.956670 [ 1095 ] {} <Trace> RaftInstance: local log idx 164, target_commit_idx 163, quick_commit_index_ 163, state_->get_commit_idx() 162
2021.03.24 11:22:47.956678 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.956694 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.956703 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.956741 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.956804 [ 1096 ] {} <Debug> RaftInstance: commit upto 163, current idx 162
2021.03.24 11:22:47.956817 [ 1096 ] {} <Trace> RaftInstance: commit upto 163, current idx 163
2021.03.24 11:22:47.956860 [ 1096 ] {} <Debug> RaftInstance: notify cb 163 0x7faa0daae120
2021.03.24 11:22:47.956936 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.956956 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 163 wake up (12034 us), return value (nil)
2021.03.24 11:22:47.956908 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 163, current idx 163
2021.03.24 11:22:47.957068 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.958223 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.958264 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 165
2021.03.24 11:22:47.958287 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 165 0x7faa0daae020 sleep
2021.03.24 11:22:47.958313 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.958318 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.958351 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 166
2021.03.24 11:22:47.958379 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.958403 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 166 0x7faa0dbd7320 sleep
2021.03.24 11:22:47.958409 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.958426 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.963502 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.963527 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=165
2021.03.24 11:22:47.963537 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.963547 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 165
2021.03.24 11:22:47.963558 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.963565 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 163, new matched idx: 164
2021.03.24 11:22:47.963576 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 166 164 160 
2021.03.24 11:22:47.963583 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 164
2021.03.24 11:22:47.963591 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.963599 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.963607 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 164, starting_idx: 91, cur_nxt_idx: 167
2021.03.24 11:22:47.963617 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=164, LastLogTerm=1, EntriesLength=2, CommitIndex=164, Term=1, peer_last_sent_idx 164
2021.03.24 11:22:47.963624 [ 1092 ] {} <Debug> RaftInstance: idx range: 165-166
2021.03.24 11:22:47.963633 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.963679 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.963692 [ 1092 ] {} <Trace> RaftInstance: local log idx 166, target_commit_idx 164, quick_commit_index_ 164, state_->get_commit_idx() 163
2021.03.24 11:22:47.963699 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.963767 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.963783 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.963863 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.963990 [ 1096 ] {} <Debug> RaftInstance: commit upto 164, current idx 163
2021.03.24 11:22:47.964005 [ 1096 ] {} <Trace> RaftInstance: commit upto 164, current idx 164
2021.03.24 11:22:47.964034 [ 1096 ] {} <Debug> RaftInstance: notify cb 164 0x7faa1f15de20
2021.03.24 11:22:47.964057 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 164, current idx 164
2021.03.24 11:22:47.964066 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.964086 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.964107 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 164 wake up (7936 us), return value 0x7faaa9a69d20
2021.03.24 11:22:47.967955 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.967975 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=163
2021.03.24 11:22:47.967985 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.967993 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 163
2021.03.24 11:22:47.968001 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.968009 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 160, new matched idx: 162
2021.03.24 11:22:47.968019 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 166 164 162 
2021.03.24 11:22:47.968027 [ 1092 ] {} <Trace> RaftInstance: local log idx 166, target_commit_idx 164, quick_commit_index_ 164, state_->get_commit_idx() 164
2021.03.24 11:22:47.968035 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.968042 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.968050 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 162, starting_idx: 91, cur_nxt_idx: 167
2021.03.24 11:22:47.968060 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=162, LastLogTerm=1, EntriesLength=4, CommitIndex=164, Term=1, peer_last_sent_idx 161
2021.03.24 11:22:47.968068 [ 1092 ] {} <Debug> RaftInstance: idx range: 163-166
2021.03.24 11:22:47.968076 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.968118 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.970540 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.970562 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=167
2021.03.24 11:22:47.970571 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.970581 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 167
2021.03.24 11:22:47.970589 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.970596 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 164, new matched idx: 166
2021.03.24 11:22:47.970606 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 166 166 162 
2021.03.24 11:22:47.970613 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 166
2021.03.24 11:22:47.970621 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.970629 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.970646 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 166, starting_idx: 91, cur_nxt_idx: 167
2021.03.24 11:22:47.970656 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=166, LastLogTerm=1, EntriesLength=0, CommitIndex=166, Term=1, peer_last_sent_idx 165
2021.03.24 11:22:47.970663 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.970672 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.970713 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.970727 [ 1093 ] {} <Trace> RaftInstance: local log idx 166, target_commit_idx 166, quick_commit_index_ 166, state_->get_commit_idx() 164
2021.03.24 11:22:47.970735 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.970819 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.970882 [ 1096 ] {} <Debug> RaftInstance: commit upto 166, current idx 164
2021.03.24 11:22:47.970902 [ 1096 ] {} <Trace> RaftInstance: commit upto 166, current idx 165
2021.03.24 11:22:47.970980 [ 1096 ] {} <Debug> RaftInstance: notify cb 165 0x7faa0daae020
2021.03.24 11:22:47.971001 [ 1096 ] {} <Trace> RaftInstance: commit upto 166, current idx 166
2021.03.24 11:22:47.971021 [ 1096 ] {} <Debug> RaftInstance: notify cb 166 0x7faa0dbd7320
2021.03.24 11:22:47.971059 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.971090 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 165 wake up (12778 us), return value (nil)
2021.03.24 11:22:47.971106 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.971128 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 166 wake up (12700 us), return value (nil)
2021.03.24 11:22:47.971145 [ 1177 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.971161 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 166, current idx 166
2021.03.24 11:22:47.971184 [ 1177 ] {} <Debug> RaftInstance: append at log_idx 167
2021.03.24 11:22:47.971187 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.971220 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.971232 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.971245 [ 1177 ] {} <Debug> RaftInstance: commit_ret_cv 167 0x7faa1f070e20 sleep
2021.03.24 11:22:47.977371 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.977420 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=167
2021.03.24 11:22:47.977432 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.977441 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 167
2021.03.24 11:22:47.977449 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.977456 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 166, new matched idx: 166
2021.03.24 11:22:47.977466 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 167 166 162 
2021.03.24 11:22:47.977474 [ 1092 ] {} <Trace> RaftInstance: local log idx 167, target_commit_idx 166, quick_commit_index_ 166, state_->get_commit_idx() 166
2021.03.24 11:22:47.977482 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.977490 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.977498 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 166, starting_idx: 91, cur_nxt_idx: 168
2021.03.24 11:22:47.977508 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=166, LastLogTerm=1, EntriesLength=1, CommitIndex=166, Term=1, peer_last_sent_idx 167
2021.03.24 11:22:47.977524 [ 1092 ] {} <Debug> RaftInstance: idx: 167
2021.03.24 11:22:47.977533 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.977578 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.979492 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.979518 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 168
2021.03.24 11:22:47.979533 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 168 0x7faa1f057020 sleep
2021.03.24 11:22:47.979550 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.979559 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.984093 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.984118 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 169
2021.03.24 11:22:47.984139 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 169 0x7faa1f09bb20 sleep
2021.03.24 11:22:47.984157 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.984174 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.984256 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.984279 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=168
2021.03.24 11:22:47.984289 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.984302 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 168
2021.03.24 11:22:47.984310 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.984318 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 166, new matched idx: 167
2021.03.24 11:22:47.984328 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 169 167 162 
2021.03.24 11:22:47.984335 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 167
2021.03.24 11:22:47.984344 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.984351 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.984359 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 167, starting_idx: 91, cur_nxt_idx: 170
2021.03.24 11:22:47.984370 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=167, LastLogTerm=1, EntriesLength=2, CommitIndex=167, Term=1, peer_last_sent_idx 167
2021.03.24 11:22:47.984377 [ 1095 ] {} <Debug> RaftInstance: idx range: 168-169
2021.03.24 11:22:47.984386 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.984440 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.984453 [ 1095 ] {} <Trace> RaftInstance: local log idx 169, target_commit_idx 167, quick_commit_index_ 167, state_->get_commit_idx() 166
2021.03.24 11:22:47.984461 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.984478 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.984487 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.984568 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.984634 [ 1096 ] {} <Debug> RaftInstance: commit upto 167, current idx 166
2021.03.24 11:22:47.984648 [ 1096 ] {} <Trace> RaftInstance: commit upto 167, current idx 167
2021.03.24 11:22:47.984698 [ 1096 ] {} <Debug> RaftInstance: notify cb 167 0x7faa1f070e20
2021.03.24 11:22:47.984722 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 167, current idx 167
2021.03.24 11:22:47.984744 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.984748 [ 1177 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.984808 [ 1177 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 167 wake up (13540 us), return value 0x7faa1bb79a00
2021.03.24 11:22:47.984860 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.984884 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 170
2021.03.24 11:22:47.984907 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 170 0x7faa0daae120 sleep
2021.03.24 11:22:47.984918 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.984931 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.988942 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:47.988963 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=167
2021.03.24 11:22:47.988971 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.988981 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 167
2021.03.24 11:22:47.988991 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:47.988999 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 162, new matched idx: 166
2021.03.24 11:22:47.989008 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 170 167 166 
2021.03.24 11:22:47.989017 [ 1095 ] {} <Trace> RaftInstance: local log idx 170, target_commit_idx 167, quick_commit_index_ 167, state_->get_commit_idx() 167
2021.03.24 11:22:47.989024 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:47.989032 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:47.989040 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 166, starting_idx: 91, cur_nxt_idx: 171
2021.03.24 11:22:47.989052 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=166, LastLogTerm=1, EntriesLength=4, CommitIndex=167, Term=1, peer_last_sent_idx 163
2021.03.24 11:22:47.989059 [ 1095 ] {} <Debug> RaftInstance: idx range: 167-170
2021.03.24 11:22:47.989068 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:47.989142 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.991258 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.991276 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=170
2021.03.24 11:22:47.991285 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.991294 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 170
2021.03.24 11:22:47.991302 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.991309 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 167, new matched idx: 169
2021.03.24 11:22:47.991319 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 170 169 166 
2021.03.24 11:22:47.991326 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 169
2021.03.24 11:22:47.991337 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.991345 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.991353 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 169, starting_idx: 91, cur_nxt_idx: 171
2021.03.24 11:22:47.991363 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=169, LastLogTerm=1, EntriesLength=1, CommitIndex=169, Term=1, peer_last_sent_idx 168
2021.03.24 11:22:47.991370 [ 1095 ] {} <Debug> RaftInstance: idx: 170
2021.03.24 11:22:47.991378 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.991426 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.991438 [ 1095 ] {} <Trace> RaftInstance: local log idx 170, target_commit_idx 169, quick_commit_index_ 169, state_->get_commit_idx() 167
2021.03.24 11:22:47.991446 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.991462 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:47.991471 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:47.991554 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.991610 [ 1096 ] {} <Debug> RaftInstance: commit upto 169, current idx 167
2021.03.24 11:22:47.991623 [ 1096 ] {} <Trace> RaftInstance: commit upto 169, current idx 168
2021.03.24 11:22:47.991706 [ 1096 ] {} <Debug> RaftInstance: notify cb 168 0x7faa1f057020
2021.03.24 11:22:47.991728 [ 1096 ] {} <Trace> RaftInstance: commit upto 169, current idx 169
2021.03.24 11:22:47.991751 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:47.991782 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 168 wake up (12216 us), return value (nil)
2021.03.24 11:22:47.991807 [ 1096 ] {} <Debug> RaftInstance: notify cb 169 0x7faa1f09bb20
2021.03.24 11:22:47.991835 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 169, current idx 169
2021.03.24 11:22:47.991844 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.991888 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:47.991916 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 169 wake up (7729 us), return value (nil)
2021.03.24 11:22:47.998204 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:47.998237 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=171
2021.03.24 11:22:47.998246 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:47.998255 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 171
2021.03.24 11:22:47.998263 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:47.998271 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 169, new matched idx: 170
2021.03.24 11:22:47.998281 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 170 170 166 
2021.03.24 11:22:47.998288 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 170
2021.03.24 11:22:47.998297 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.998304 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:47.998313 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 170, starting_idx: 91, cur_nxt_idx: 171
2021.03.24 11:22:47.998322 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=170, LastLogTerm=1, EntriesLength=0, CommitIndex=170, Term=1, peer_last_sent_idx 170
2021.03.24 11:22:47.998330 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:47.998338 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:47.998399 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:47.998413 [ 1095 ] {} <Trace> RaftInstance: local log idx 170, target_commit_idx 170, quick_commit_index_ 170, state_->get_commit_idx() 169
2021.03.24 11:22:47.998421 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:47.998535 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:47.998593 [ 1096 ] {} <Debug> RaftInstance: commit upto 170, current idx 169
2021.03.24 11:22:47.998606 [ 1096 ] {} <Trace> RaftInstance: commit upto 170, current idx 170
2021.03.24 11:22:47.998643 [ 1096 ] {} <Debug> RaftInstance: notify cb 170 0x7faa0daae120
2021.03.24 11:22:47.998664 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 170, current idx 170
2021.03.24 11:22:47.998672 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:47.998688 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:47.998730 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 170 wake up (13781 us), return value (nil)
2021.03.24 11:22:47.998753 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:47.998790 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 171
2021.03.24 11:22:47.998815 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 171 0x7faa0daae020 sleep
2021.03.24 11:22:47.998843 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:47.998861 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.000059 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.000082 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 172
2021.03.24 11:22:48.000101 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 172 0x7faa1f15de20 sleep
2021.03.24 11:22:48.000117 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.000134 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.005162 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.005185 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=171
2021.03.24 11:22:48.005225 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.005235 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 171
2021.03.24 11:22:48.005244 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.005252 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 170, new matched idx: 170
2021.03.24 11:22:48.005262 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 172 170 166 
2021.03.24 11:22:48.005271 [ 1094 ] {} <Trace> RaftInstance: local log idx 172, target_commit_idx 170, quick_commit_index_ 170, state_->get_commit_idx() 170
2021.03.24 11:22:48.005279 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.005286 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.005295 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 170, starting_idx: 91, cur_nxt_idx: 173
2021.03.24 11:22:48.005302 [ 1094 ] {} <Debug> RaftInstance: last sent log (171) to peer 1 is not applied, cnt 1
2021.03.24 11:22:48.005313 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=170, LastLogTerm=1, EntriesLength=2, CommitIndex=170, Term=1, peer_last_sent_idx 171
2021.03.24 11:22:48.005320 [ 1094 ] {} <Debug> RaftInstance: idx range: 171-172
2021.03.24 11:22:48.005329 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.005378 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.009594 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.009621 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=171
2021.03.24 11:22:48.009631 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.009641 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 171
2021.03.24 11:22:48.009649 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.009657 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 166, new matched idx: 170
2021.03.24 11:22:48.009667 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 172 170 170 
2021.03.24 11:22:48.009676 [ 1093 ] {} <Trace> RaftInstance: local log idx 172, target_commit_idx 170, quick_commit_index_ 170, state_->get_commit_idx() 170
2021.03.24 11:22:48.009701 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.009709 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.009718 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 170, starting_idx: 91, cur_nxt_idx: 173
2021.03.24 11:22:48.009729 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=170, LastLogTerm=1, EntriesLength=2, CommitIndex=170, Term=1, peer_last_sent_idx 167
2021.03.24 11:22:48.009736 [ 1093 ] {} <Debug> RaftInstance: idx range: 171-172
2021.03.24 11:22:48.009745 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.009797 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.011712 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.011744 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 173
2021.03.24 11:22:48.011767 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 173 0x7faa0dbd7320 sleep
2021.03.24 11:22:48.011787 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.011805 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.012210 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.012227 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=173
2021.03.24 11:22:48.012236 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.012245 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 173
2021.03.24 11:22:48.012253 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.012260 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 170, new matched idx: 172
2021.03.24 11:22:48.012270 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 173 172 170 
2021.03.24 11:22:48.012278 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 172
2021.03.24 11:22:48.012286 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.012294 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.012302 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 172, starting_idx: 91, cur_nxt_idx: 174
2021.03.24 11:22:48.012312 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=172, LastLogTerm=1, EntriesLength=1, CommitIndex=172, Term=1, peer_last_sent_idx 171
2021.03.24 11:22:48.012320 [ 1093 ] {} <Debug> RaftInstance: idx: 173
2021.03.24 11:22:48.012328 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.012365 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.012378 [ 1093 ] {} <Trace> RaftInstance: local log idx 173, target_commit_idx 172, quick_commit_index_ 172, state_->get_commit_idx() 170
2021.03.24 11:22:48.012386 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.012403 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.012413 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.012436 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.012457 [ 1096 ] {} <Debug> RaftInstance: commit upto 172, current idx 170
2021.03.24 11:22:48.012466 [ 1096 ] {} <Trace> RaftInstance: commit upto 172, current idx 171
2021.03.24 11:22:48.012504 [ 1096 ] {} <Debug> RaftInstance: notify cb 171 0x7faa0daae020
2021.03.24 11:22:48.012524 [ 1096 ] {} <Trace> RaftInstance: commit upto 172, current idx 172
2021.03.24 11:22:48.012560 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:48.012587 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 171 wake up (13744 us), return value (nil)
2021.03.24 11:22:48.012613 [ 1096 ] {} <Debug> RaftInstance: notify cb 172 0x7faa1f15de20
2021.03.24 11:22:48.012622 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.012642 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 174
2021.03.24 11:22:48.012652 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 172, current idx 172
2021.03.24 11:22:48.012661 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 174 0x7faa0daae120 sleep
2021.03.24 11:22:48.012662 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.012679 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.012695 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.012732 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:48.012797 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 172 wake up (12606 us), return value 0x7faa1bb79a20
2021.03.24 11:22:48.019132 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.019162 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=174
2021.03.24 11:22:48.019171 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.019182 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 174
2021.03.24 11:22:48.019190 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.019198 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 172, new matched idx: 173
2021.03.24 11:22:48.019209 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 174 173 170 
2021.03.24 11:22:48.019217 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 173
2021.03.24 11:22:48.019247 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.019288 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.019299 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 173, starting_idx: 91, cur_nxt_idx: 175
2021.03.24 11:22:48.019311 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=173, LastLogTerm=1, EntriesLength=1, CommitIndex=173, Term=1, peer_last_sent_idx 173
2021.03.24 11:22:48.019318 [ 1093 ] {} <Debug> RaftInstance: idx: 174
2021.03.24 11:22:48.019327 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.019380 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.019394 [ 1093 ] {} <Trace> RaftInstance: local log idx 174, target_commit_idx 173, quick_commit_index_ 173, state_->get_commit_idx() 172
2021.03.24 11:22:48.019402 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.019418 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.019427 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.019452 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.019479 [ 1096 ] {} <Debug> RaftInstance: commit upto 173, current idx 172
2021.03.24 11:22:48.019489 [ 1096 ] {} <Trace> RaftInstance: commit upto 173, current idx 173
2021.03.24 11:22:48.019522 [ 1096 ] {} <Debug> RaftInstance: notify cb 173 0x7faa0dbd7320
2021.03.24 11:22:48.019542 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 173, current idx 173
2021.03.24 11:22:48.019551 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.019576 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.019602 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 173 wake up (7813 us), return value (nil)
2021.03.24 11:22:48.025953 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.025982 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=175
2021.03.24 11:22:48.026011 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.026021 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 175
2021.03.24 11:22:48.026029 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.026037 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 173, new matched idx: 174
2021.03.24 11:22:48.026048 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 174 174 170 
2021.03.24 11:22:48.026056 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 174
2021.03.24 11:22:48.026064 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.026073 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.026081 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 174, starting_idx: 91, cur_nxt_idx: 175
2021.03.24 11:22:48.026091 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=174, LastLogTerm=1, EntriesLength=0, CommitIndex=174, Term=1, peer_last_sent_idx 174
2021.03.24 11:22:48.026098 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.026106 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.026169 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.026183 [ 1093 ] {} <Trace> RaftInstance: local log idx 174, target_commit_idx 174, quick_commit_index_ 174, state_->get_commit_idx() 173
2021.03.24 11:22:48.026191 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.026230 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.026250 [ 1096 ] {} <Debug> RaftInstance: commit upto 174, current idx 173
2021.03.24 11:22:48.026259 [ 1096 ] {} <Trace> RaftInstance: commit upto 174, current idx 174
2021.03.24 11:22:48.026321 [ 1096 ] {} <Debug> RaftInstance: notify cb 174 0x7faa0daae120
2021.03.24 11:22:48.026339 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 174, current idx 174
2021.03.24 11:22:48.026348 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.026364 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.026383 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 174 wake up (13703 us), return value (nil)
2021.03.24 11:22:48.026407 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.026429 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 175
2021.03.24 11:22:48.026451 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 175 0x7faa0daae020 sleep
2021.03.24 11:22:48.026481 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.026499 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.028813 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.028837 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 176
2021.03.24 11:22:48.028858 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 176 0x7faa0dbd7320 sleep
2021.03.24 11:22:48.028881 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.028898 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.030256 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.030274 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=173
2021.03.24 11:22:48.030283 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.030292 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 173
2021.03.24 11:22:48.030308 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.030317 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 170, new matched idx: 172
2021.03.24 11:22:48.030326 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 176 174 172 
2021.03.24 11:22:48.030335 [ 1093 ] {} <Trace> RaftInstance: local log idx 176, target_commit_idx 174, quick_commit_index_ 174, state_->get_commit_idx() 174
2021.03.24 11:22:48.030343 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.030351 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.030359 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 172, starting_idx: 91, cur_nxt_idx: 177
2021.03.24 11:22:48.030370 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=172, LastLogTerm=1, EntriesLength=4, CommitIndex=174, Term=1, peer_last_sent_idx 171
2021.03.24 11:22:48.030377 [ 1093 ] {} <Debug> RaftInstance: idx range: 173-176
2021.03.24 11:22:48.030386 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.030432 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.032710 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.032733 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=175
2021.03.24 11:22:48.032742 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.032752 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 175
2021.03.24 11:22:48.032760 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.032768 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 174, new matched idx: 174
2021.03.24 11:22:48.032778 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 176 174 172 
2021.03.24 11:22:48.032787 [ 1093 ] {} <Trace> RaftInstance: local log idx 176, target_commit_idx 174, quick_commit_index_ 174, state_->get_commit_idx() 174
2021.03.24 11:22:48.032795 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.032803 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.032811 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 174, starting_idx: 91, cur_nxt_idx: 177
2021.03.24 11:22:48.032819 [ 1093 ] {} <Debug> RaftInstance: last sent log (175) to peer 1 is not applied, cnt 1
2021.03.24 11:22:48.032829 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=174, LastLogTerm=1, EntriesLength=2, CommitIndex=174, Term=1, peer_last_sent_idx 175
2021.03.24 11:22:48.032836 [ 1093 ] {} <Debug> RaftInstance: idx range: 175-176
2021.03.24 11:22:48.032845 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.032888 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.039587 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.039595 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.039672 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=177
2021.03.24 11:22:48.039737 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 177
2021.03.24 11:22:48.039742 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.039767 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 177
2021.03.24 11:22:48.039774 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 177 0x7faa1f15de20 sleep
2021.03.24 11:22:48.039778 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.039804 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 174, new matched idx: 176
2021.03.24 11:22:48.039817 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 177 176 172 
2021.03.24 11:22:48.039844 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 176
2021.03.24 11:22:48.039853 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.039861 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.039870 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 176, starting_idx: 91, cur_nxt_idx: 178
2021.03.24 11:22:48.039891 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=176, LastLogTerm=1, EntriesLength=1, CommitIndex=176, Term=1, peer_last_sent_idx 175
2021.03.24 11:22:48.039900 [ 1093 ] {} <Debug> RaftInstance: idx: 177
2021.03.24 11:22:48.039910 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.039971 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.039984 [ 1093 ] {} <Trace> RaftInstance: local log idx 177, target_commit_idx 176, quick_commit_index_ 176, state_->get_commit_idx() 174
2021.03.24 11:22:48.039992 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.040007 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.040017 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.040043 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.040047 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.040079 [ 1096 ] {} <Debug> RaftInstance: commit upto 176, current idx 174
2021.03.24 11:22:48.040099 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.040118 [ 1096 ] {} <Trace> RaftInstance: commit upto 176, current idx 175
2021.03.24 11:22:48.040192 [ 1096 ] {} <Debug> RaftInstance: notify cb 175 0x7faa0daae020
2021.03.24 11:22:48.040213 [ 1096 ] {} <Trace> RaftInstance: commit upto 176, current idx 176
2021.03.24 11:22:48.040233 [ 1096 ] {} <Debug> RaftInstance: notify cb 176 0x7faa0dbd7320
2021.03.24 11:22:48.040253 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 176, current idx 176
2021.03.24 11:22:48.040261 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.040265 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:48.040287 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 175 wake up (13819 us), return value (nil)
2021.03.24 11:22:48.040312 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.040328 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 176 wake up (11459 us), return value (nil)
2021.03.24 11:22:48.044068 [ 1095 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2021.03.24 11:22:48.044119 [ 1095 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:48.044131 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.046671 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.046717 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=178
2021.03.24 11:22:48.046728 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.046740 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 178
2021.03.24 11:22:48.046749 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.046756 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 176, new matched idx: 177
2021.03.24 11:22:48.046769 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 177 177 172 
2021.03.24 11:22:48.046802 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 177
2021.03.24 11:22:48.046821 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.046829 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.046838 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 177, starting_idx: 91, cur_nxt_idx: 178
2021.03.24 11:22:48.046874 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=177, LastLogTerm=1, EntriesLength=0, CommitIndex=177, Term=1, peer_last_sent_idx 177
2021.03.24 11:22:48.046881 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.046890 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.046971 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.046985 [ 1093 ] {} <Trace> RaftInstance: local log idx 177, target_commit_idx 177, quick_commit_index_ 177, state_->get_commit_idx() 176
2021.03.24 11:22:48.046993 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.047052 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.047078 [ 1096 ] {} <Debug> RaftInstance: commit upto 177, current idx 176
2021.03.24 11:22:48.047090 [ 1096 ] {} <Trace> RaftInstance: commit upto 177, current idx 177
2021.03.24 11:22:48.047153 [ 1096 ] {} <Debug> RaftInstance: notify cb 177 0x7faa1f15de20
2021.03.24 11:22:48.047178 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 177, current idx 177
2021.03.24 11:22:48.047187 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.047198 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.047220 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 177 wake up (7437 us), return value (nil)
2021.03.24 11:22:48.050970 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.051010 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=177
2021.03.24 11:22:48.051020 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.051031 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 177
2021.03.24 11:22:48.051039 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.051047 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 172, new matched idx: 176
2021.03.24 11:22:48.051061 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 177 177 176 
2021.03.24 11:22:48.051069 [ 1093 ] {} <Trace> RaftInstance: local log idx 177, target_commit_idx 177, quick_commit_index_ 177, state_->get_commit_idx() 177
2021.03.24 11:22:48.051078 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.051087 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.051095 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 176, starting_idx: 91, cur_nxt_idx: 178
2021.03.24 11:22:48.051107 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=176, LastLogTerm=1, EntriesLength=1, CommitIndex=177, Term=1, peer_last_sent_idx 173
2021.03.24 11:22:48.051115 [ 1093 ] {} <Debug> RaftInstance: idx: 177
2021.03.24 11:22:48.051124 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.051199 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.054690 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.054728 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=178
2021.03.24 11:22:48.054739 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.054750 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 178
2021.03.24 11:22:48.054758 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.054766 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 177, new matched idx: 177
2021.03.24 11:22:48.054798 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 177 177 176 
2021.03.24 11:22:48.054809 [ 1093 ] {} <Trace> RaftInstance: local log idx 177, target_commit_idx 177, quick_commit_index_ 177, state_->get_commit_idx() 177
2021.03.24 11:22:48.055803 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.055867 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 178
2021.03.24 11:22:48.055895 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 178 0x7faa1f070b20 sleep
2021.03.24 11:22:48.055936 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.055958 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.055968 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 177, starting_idx: 91, cur_nxt_idx: 179
2021.03.24 11:22:48.055981 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=177, LastLogTerm=1, EntriesLength=1, CommitIndex=177, Term=1, peer_last_sent_idx 178
2021.03.24 11:22:48.055990 [ 1097 ] {} <Debug> RaftInstance: idx: 178
2021.03.24 11:22:48.056000 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.056088 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.064501 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.064541 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=179
2021.03.24 11:22:48.064551 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.064562 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 179
2021.03.24 11:22:48.064570 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.064578 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 177, new matched idx: 178
2021.03.24 11:22:48.064591 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 178 178 176 
2021.03.24 11:22:48.064598 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 178
2021.03.24 11:22:48.064607 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.064615 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.064623 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 178, starting_idx: 91, cur_nxt_idx: 179
2021.03.24 11:22:48.064633 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=178, LastLogTerm=1, EntriesLength=0, CommitIndex=178, Term=1, peer_last_sent_idx 178
2021.03.24 11:22:48.064640 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.064649 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.064706 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.064719 [ 1093 ] {} <Trace> RaftInstance: local log idx 178, target_commit_idx 178, quick_commit_index_ 178, state_->get_commit_idx() 177
2021.03.24 11:22:48.064726 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.064772 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.064799 [ 1096 ] {} <Debug> RaftInstance: commit upto 178, current idx 177
2021.03.24 11:22:48.064810 [ 1096 ] {} <Trace> RaftInstance: commit upto 178, current idx 178
2021.03.24 11:22:48.064913 [ 1096 ] {} <Debug> RaftInstance: notify cb 178 0x7faa1f070b20
2021.03.24 11:22:48.064943 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 178, current idx 178
2021.03.24 11:22:48.064951 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.064986 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.065044 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 178 wake up (9080 us), return value (nil)
2021.03.24 11:22:48.069452 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.069529 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 179
2021.03.24 11:22:48.069580 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 179 0x7faa1f15de20 sleep
2021.03.24 11:22:48.069607 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.069628 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.071515 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.071544 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=179
2021.03.24 11:22:48.071554 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.071565 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 179
2021.03.24 11:22:48.071574 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.071582 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 178, new matched idx: 178
2021.03.24 11:22:48.071594 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 179 178 176 
2021.03.24 11:22:48.071613 [ 1094 ] {} <Trace> RaftInstance: local log idx 179, target_commit_idx 178, quick_commit_index_ 178, state_->get_commit_idx() 178
2021.03.24 11:22:48.071622 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.071632 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.071665 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 178, starting_idx: 91, cur_nxt_idx: 180
2021.03.24 11:22:48.071631 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.071683 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=178, LastLogTerm=1, EntriesLength=1, CommitIndex=178, Term=1, peer_last_sent_idx 179
2021.03.24 11:22:48.071735 [ 1094 ] {} <Debug> RaftInstance: idx: 179
2021.03.24 11:22:48.071747 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.071828 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.071862 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=178
2021.03.24 11:22:48.071882 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.071893 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 178
2021.03.24 11:22:48.071901 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.071910 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 176, new matched idx: 177
2021.03.24 11:22:48.071921 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 179 178 177 
2021.03.24 11:22:48.071930 [ 1093 ] {} <Trace> RaftInstance: local log idx 179, target_commit_idx 178, quick_commit_index_ 178, state_->get_commit_idx() 178
2021.03.24 11:22:48.071938 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.071946 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.071955 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 177, starting_idx: 91, cur_nxt_idx: 180
2021.03.24 11:22:48.071966 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=177, LastLogTerm=1, EntriesLength=2, CommitIndex=178, Term=1, peer_last_sent_idx 177
2021.03.24 11:22:48.071974 [ 1093 ] {} <Debug> RaftInstance: idx range: 178-179
2021.03.24 11:22:48.071984 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.072038 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.073344 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.073374 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 180
2021.03.24 11:22:48.073414 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.073425 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.073455 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 180 0x7faa1f070d20 sleep
2021.03.24 11:22:48.078553 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.078613 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=180
2021.03.24 11:22:48.078624 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.078637 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 180
2021.03.24 11:22:48.078646 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.078654 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 178, new matched idx: 179
2021.03.24 11:22:48.078667 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 180 179 177 
2021.03.24 11:22:48.078675 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 179
2021.03.24 11:22:48.078685 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.078693 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.078702 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 179, starting_idx: 91, cur_nxt_idx: 181
2021.03.24 11:22:48.078715 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=179, LastLogTerm=1, EntriesLength=1, CommitIndex=179, Term=1, peer_last_sent_idx 179
2021.03.24 11:22:48.078722 [ 1094 ] {} <Debug> RaftInstance: idx: 180
2021.03.24 11:22:48.078731 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.078826 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.078840 [ 1094 ] {} <Trace> RaftInstance: local log idx 180, target_commit_idx 179, quick_commit_index_ 179, state_->get_commit_idx() 178
2021.03.24 11:22:48.078848 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.078869 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.078878 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.078909 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.078941 [ 1096 ] {} <Debug> RaftInstance: commit upto 179, current idx 178
2021.03.24 11:22:48.078953 [ 1096 ] {} <Trace> RaftInstance: commit upto 179, current idx 179
2021.03.24 11:22:48.078994 [ 1096 ] {} <Debug> RaftInstance: notify cb 179 0x7faa1f15de20
2021.03.24 11:22:48.079016 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 179, current idx 179
2021.03.24 11:22:48.079024 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.079038 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.079056 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 179 wake up (9464 us), return value (nil)
2021.03.24 11:22:48.085848 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.085890 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=181
2021.03.24 11:22:48.085900 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.085911 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 181
2021.03.24 11:22:48.085932 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.085940 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 179, new matched idx: 180
2021.03.24 11:22:48.085952 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 180 180 177 
2021.03.24 11:22:48.085960 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 180
2021.03.24 11:22:48.085969 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.085977 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.085986 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 180, starting_idx: 91, cur_nxt_idx: 181
2021.03.24 11:22:48.085996 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=180, LastLogTerm=1, EntriesLength=0, CommitIndex=180, Term=1, peer_last_sent_idx 180
2021.03.24 11:22:48.086028 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.086037 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.086110 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.086123 [ 1094 ] {} <Trace> RaftInstance: local log idx 180, target_commit_idx 180, quick_commit_index_ 180, state_->get_commit_idx() 179
2021.03.24 11:22:48.086131 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.086203 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.086245 [ 1096 ] {} <Debug> RaftInstance: commit upto 180, current idx 179
2021.03.24 11:22:48.086256 [ 1096 ] {} <Trace> RaftInstance: commit upto 180, current idx 180
2021.03.24 11:22:48.086295 [ 1096 ] {} <Debug> RaftInstance: notify cb 180 0x7faa1f070d20
2021.03.24 11:22:48.086317 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 180, current idx 180
2021.03.24 11:22:48.086325 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.086351 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.086376 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 180 wake up (12956 us), return value (nil)
2021.03.24 11:22:48.092814 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.092871 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=180
2021.03.24 11:22:48.092882 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.092893 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 180
2021.03.24 11:22:48.092901 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.092908 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 177, new matched idx: 179
2021.03.24 11:22:48.092920 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 180 180 179 
2021.03.24 11:22:48.092929 [ 1094 ] {} <Trace> RaftInstance: local log idx 180, target_commit_idx 180, quick_commit_index_ 180, state_->get_commit_idx() 180
2021.03.24 11:22:48.092936 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.092944 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.092953 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 179, starting_idx: 91, cur_nxt_idx: 181
2021.03.24 11:22:48.092965 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=179, LastLogTerm=1, EntriesLength=1, CommitIndex=180, Term=1, peer_last_sent_idx 178
2021.03.24 11:22:48.092972 [ 1094 ] {} <Debug> RaftInstance: idx: 180
2021.03.24 11:22:48.092981 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.093114 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.093183 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.093197 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=181
2021.03.24 11:22:48.093205 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.093213 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 181
2021.03.24 11:22:48.093221 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.093228 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 180, new matched idx: 180
2021.03.24 11:22:48.093236 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 180 180 179 
2021.03.24 11:22:48.093245 [ 1092 ] {} <Trace> RaftInstance: local log idx 180, target_commit_idx 180, quick_commit_index_ 180, state_->get_commit_idx() 180
2021.03.24 11:22:48.095326 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.095406 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 181
2021.03.24 11:22:48.095428 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 181 0x7faa0dbd7520 sleep
2021.03.24 11:22:48.095470 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.095512 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.095523 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 180, starting_idx: 91, cur_nxt_idx: 182
2021.03.24 11:22:48.095538 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=180, LastLogTerm=1, EntriesLength=1, CommitIndex=180, Term=1, peer_last_sent_idx 181
2021.03.24 11:22:48.095556 [ 1097 ] {} <Debug> RaftInstance: idx: 181
2021.03.24 11:22:48.095567 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.095645 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.098946 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.098989 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 182
2021.03.24 11:22:48.099014 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.099023 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.099047 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 182 0x7faa1f070d20 sleep
2021.03.24 11:22:48.102336 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.102362 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=182
2021.03.24 11:22:48.102372 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.102382 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 182
2021.03.24 11:22:48.102391 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.102399 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 180, new matched idx: 181
2021.03.24 11:22:48.102410 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 182 181 179 
2021.03.24 11:22:48.102418 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 181
2021.03.24 11:22:48.102426 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.102434 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.102442 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 181, starting_idx: 91, cur_nxt_idx: 183
2021.03.24 11:22:48.102452 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=181, LastLogTerm=1, EntriesLength=1, CommitIndex=181, Term=1, peer_last_sent_idx 181
2021.03.24 11:22:48.102460 [ 1095 ] {} <Debug> RaftInstance: idx: 182
2021.03.24 11:22:48.102468 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.102507 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.102520 [ 1095 ] {} <Trace> RaftInstance: local log idx 182, target_commit_idx 181, quick_commit_index_ 181, state_->get_commit_idx() 180
2021.03.24 11:22:48.102528 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.102545 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.102554 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.102583 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.102630 [ 1096 ] {} <Debug> RaftInstance: commit upto 181, current idx 180
2021.03.24 11:22:48.102642 [ 1096 ] {} <Trace> RaftInstance: commit upto 181, current idx 181
2021.03.24 11:22:48.102686 [ 1096 ] {} <Debug> RaftInstance: notify cb 181 0x7faa0dbd7520
2021.03.24 11:22:48.102706 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 181, current idx 181
2021.03.24 11:22:48.102714 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.102745 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.102768 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 181 wake up (7319 us), return value (nil)
2021.03.24 11:22:48.109318 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.109344 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=183
2021.03.24 11:22:48.109353 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.109362 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 183
2021.03.24 11:22:48.109370 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.109382 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 181, new matched idx: 182
2021.03.24 11:22:48.109392 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 182 182 179 
2021.03.24 11:22:48.109399 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 182
2021.03.24 11:22:48.109408 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.109415 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.109423 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 182, starting_idx: 91, cur_nxt_idx: 183
2021.03.24 11:22:48.109433 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=182, LastLogTerm=1, EntriesLength=0, CommitIndex=182, Term=1, peer_last_sent_idx 182
2021.03.24 11:22:48.109440 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.109448 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.109489 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.109502 [ 1095 ] {} <Trace> RaftInstance: local log idx 182, target_commit_idx 182, quick_commit_index_ 182, state_->get_commit_idx() 181
2021.03.24 11:22:48.109510 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.109547 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.109570 [ 1096 ] {} <Debug> RaftInstance: commit upto 182, current idx 181
2021.03.24 11:22:48.109581 [ 1096 ] {} <Trace> RaftInstance: commit upto 182, current idx 182
2021.03.24 11:22:48.109640 [ 1096 ] {} <Debug> RaftInstance: notify cb 182 0x7faa1f070d20
2021.03.24 11:22:48.109660 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 182, current idx 182
2021.03.24 11:22:48.109669 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.109682 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.109698 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 182 wake up (10674 us), return value (nil)
2021.03.24 11:22:48.111106 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.111173 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 183
2021.03.24 11:22:48.111197 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 183 0x7faa0dbd7520 sleep
2021.03.24 11:22:48.111227 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.111244 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.113635 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.113657 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=181
2021.03.24 11:22:48.113667 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.113676 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 181
2021.03.24 11:22:48.113684 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.113692 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 179, new matched idx: 180
2021.03.24 11:22:48.113719 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 183 182 180 
2021.03.24 11:22:48.113728 [ 1092 ] {} <Trace> RaftInstance: local log idx 183, target_commit_idx 182, quick_commit_index_ 182, state_->get_commit_idx() 182
2021.03.24 11:22:48.113736 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.113744 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.113752 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 180, starting_idx: 91, cur_nxt_idx: 184
2021.03.24 11:22:48.113763 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=180, LastLogTerm=1, EntriesLength=3, CommitIndex=182, Term=1, peer_last_sent_idx 180
2021.03.24 11:22:48.113771 [ 1092 ] {} <Debug> RaftInstance: idx range: 181-183
2021.03.24 11:22:48.113779 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.113847 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.116157 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.116177 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=183
2021.03.24 11:22:48.116186 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.116195 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 183
2021.03.24 11:22:48.116203 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.116221 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 182, new matched idx: 182
2021.03.24 11:22:48.116232 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 183 182 180 
2021.03.24 11:22:48.116241 [ 1095 ] {} <Trace> RaftInstance: local log idx 183, target_commit_idx 182, quick_commit_index_ 182, state_->get_commit_idx() 182
2021.03.24 11:22:48.116249 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.116262 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.116270 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 182, starting_idx: 91, cur_nxt_idx: 184
2021.03.24 11:22:48.116280 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=182, LastLogTerm=1, EntriesLength=1, CommitIndex=182, Term=1, peer_last_sent_idx 183
2021.03.24 11:22:48.116288 [ 1095 ] {} <Debug> RaftInstance: idx: 183
2021.03.24 11:22:48.116297 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.116337 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.117562 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.117602 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 184
2021.03.24 11:22:48.117625 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 184 0x7faa0daae120 sleep
2021.03.24 11:22:48.117647 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.117664 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.123010 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.123041 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=184
2021.03.24 11:22:48.123050 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.123060 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 184
2021.03.24 11:22:48.123067 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.123075 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 182, new matched idx: 183
2021.03.24 11:22:48.123084 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 184 183 180 
2021.03.24 11:22:48.123092 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 183
2021.03.24 11:22:48.123110 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.123118 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.123126 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 183, starting_idx: 91, cur_nxt_idx: 185
2021.03.24 11:22:48.123137 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=183, LastLogTerm=1, EntriesLength=1, CommitIndex=183, Term=1, peer_last_sent_idx 183
2021.03.24 11:22:48.123144 [ 1095 ] {} <Debug> RaftInstance: idx: 184
2021.03.24 11:22:48.123152 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.123193 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.123206 [ 1095 ] {} <Trace> RaftInstance: local log idx 184, target_commit_idx 183, quick_commit_index_ 183, state_->get_commit_idx() 182
2021.03.24 11:22:48.123213 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.123231 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.123240 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.123266 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.123286 [ 1096 ] {} <Debug> RaftInstance: commit upto 183, current idx 182
2021.03.24 11:22:48.123296 [ 1096 ] {} <Trace> RaftInstance: commit upto 183, current idx 183
2021.03.24 11:22:48.123354 [ 1096 ] {} <Debug> RaftInstance: notify cb 183 0x7faa0dbd7520
2021.03.24 11:22:48.123370 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 183, current idx 183
2021.03.24 11:22:48.123378 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.123408 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.123419 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 183 wake up (12220 us), return value (nil)
2021.03.24 11:22:48.129618 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.129663 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 185
2021.03.24 11:22:48.129691 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.129701 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.129713 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 185 0x7faa1f070d20 sleep
2021.03.24 11:22:48.129838 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.129859 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=185
2021.03.24 11:22:48.129869 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.129883 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 185
2021.03.24 11:22:48.129892 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.129900 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 183, new matched idx: 184
2021.03.24 11:22:48.129910 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 185 184 180 
2021.03.24 11:22:48.129918 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 184
2021.03.24 11:22:48.129926 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.129934 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.129943 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 184, starting_idx: 91, cur_nxt_idx: 186
2021.03.24 11:22:48.129954 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=184, LastLogTerm=1, EntriesLength=1, CommitIndex=184, Term=1, peer_last_sent_idx 184
2021.03.24 11:22:48.129961 [ 1094 ] {} <Debug> RaftInstance: idx: 185
2021.03.24 11:22:48.129986 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.130041 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.130055 [ 1094 ] {} <Trace> RaftInstance: local log idx 185, target_commit_idx 184, quick_commit_index_ 184, state_->get_commit_idx() 183
2021.03.24 11:22:48.130062 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.130074 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.130083 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.130103 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.130113 [ 1096 ] {} <Debug> RaftInstance: commit upto 184, current idx 183
2021.03.24 11:22:48.130121 [ 1096 ] {} <Trace> RaftInstance: commit upto 184, current idx 184
2021.03.24 11:22:48.130149 [ 1096 ] {} <Debug> RaftInstance: notify cb 184 0x7faa0daae120
2021.03.24 11:22:48.130167 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 184, current idx 184
2021.03.24 11:22:48.130175 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.130197 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.130216 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 184 wake up (12576 us), return value (nil)
2021.03.24 11:22:48.131026 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.131049 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 186
2021.03.24 11:22:48.131069 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 186 0x7faa0daae020 sleep
2021.03.24 11:22:48.131087 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.131102 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.131652 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.131674 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 187
2021.03.24 11:22:48.131696 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.131705 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.131716 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 187 0x7faa0dbd7220 sleep
2021.03.24 11:22:48.134436 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.134457 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=184
2021.03.24 11:22:48.134467 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.134475 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 184
2021.03.24 11:22:48.134483 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.134491 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 180, new matched idx: 183
2021.03.24 11:22:48.134501 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 187 184 183 
2021.03.24 11:22:48.134509 [ 1095 ] {} <Trace> RaftInstance: local log idx 187, target_commit_idx 184, quick_commit_index_ 184, state_->get_commit_idx() 184
2021.03.24 11:22:48.134516 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.134524 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.134532 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 183, starting_idx: 91, cur_nxt_idx: 188
2021.03.24 11:22:48.134543 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=183, LastLogTerm=1, EntriesLength=4, CommitIndex=184, Term=1, peer_last_sent_idx 181
2021.03.24 11:22:48.134550 [ 1095 ] {} <Debug> RaftInstance: idx range: 184-187
2021.03.24 11:22:48.134559 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.134633 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.137167 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.137190 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=186
2021.03.24 11:22:48.137199 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.137208 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 186
2021.03.24 11:22:48.137216 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.137224 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 184, new matched idx: 185
2021.03.24 11:22:48.137234 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 187 185 183 
2021.03.24 11:22:48.137241 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 185
2021.03.24 11:22:48.137249 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.137257 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.137265 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 185, starting_idx: 91, cur_nxt_idx: 188
2021.03.24 11:22:48.137276 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=185, LastLogTerm=1, EntriesLength=2, CommitIndex=185, Term=1, peer_last_sent_idx 185
2021.03.24 11:22:48.137284 [ 1095 ] {} <Debug> RaftInstance: idx range: 186-187
2021.03.24 11:22:48.137292 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.137329 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.137341 [ 1095 ] {} <Trace> RaftInstance: local log idx 187, target_commit_idx 185, quick_commit_index_ 185, state_->get_commit_idx() 184
2021.03.24 11:22:48.137349 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.137360 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.137368 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.137391 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.137401 [ 1096 ] {} <Debug> RaftInstance: commit upto 185, current idx 184
2021.03.24 11:22:48.137410 [ 1096 ] {} <Trace> RaftInstance: commit upto 185, current idx 185
2021.03.24 11:22:48.137436 [ 1096 ] {} <Debug> RaftInstance: notify cb 185 0x7faa1f070d20
2021.03.24 11:22:48.137454 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 185, current idx 185
2021.03.24 11:22:48.137462 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.137472 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:48.137490 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 185 wake up (7792 us), return value (nil)
2021.03.24 11:22:48.143951 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.143990 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=188
2021.03.24 11:22:48.144000 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.144011 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 188
2021.03.24 11:22:48.144019 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.144027 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 185, new matched idx: 187
2021.03.24 11:22:48.144039 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 187 187 183 
2021.03.24 11:22:48.144047 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 187
2021.03.24 11:22:48.144056 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.144064 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.144072 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 187, starting_idx: 91, cur_nxt_idx: 188
2021.03.24 11:22:48.144084 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=187, LastLogTerm=1, EntriesLength=0, CommitIndex=187, Term=1, peer_last_sent_idx 186
2021.03.24 11:22:48.144107 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.144116 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.144167 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.144180 [ 1095 ] {} <Trace> RaftInstance: local log idx 187, target_commit_idx 187, quick_commit_index_ 187, state_->get_commit_idx() 185
2021.03.24 11:22:48.144188 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.144217 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.144228 [ 1096 ] {} <Debug> RaftInstance: commit upto 187, current idx 185
2021.03.24 11:22:48.144236 [ 1096 ] {} <Trace> RaftInstance: commit upto 187, current idx 186
2021.03.24 11:22:48.144370 [ 1096 ] {} <Debug> RaftInstance: notify cb 186 0x7faa0daae020
2021.03.24 11:22:48.144393 [ 1096 ] {} <Trace> RaftInstance: commit upto 187, current idx 187
2021.03.24 11:22:48.144431 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.144458 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 186 wake up (13363 us), return value (nil)
2021.03.24 11:22:48.144465 [ 1096 ] {} <Debug> RaftInstance: notify cb 187 0x7faa0dbd7220
2021.03.24 11:22:48.144488 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 187, current idx 187
2021.03.24 11:22:48.144497 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.144505 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.144524 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 187 wake up (12813 us), return value (nil)
2021.03.24 11:22:48.146269 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.146301 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 188
2021.03.24 11:22:48.146325 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 188 0x7faa0daae120 sleep
2021.03.24 11:22:48.146348 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.146364 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.150789 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.150820 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=188
2021.03.24 11:22:48.150830 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.150841 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 188
2021.03.24 11:22:48.150849 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.150857 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 187, new matched idx: 187
2021.03.24 11:22:48.150868 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 188 187 183 
2021.03.24 11:22:48.150877 [ 1093 ] {} <Trace> RaftInstance: local log idx 188, target_commit_idx 187, quick_commit_index_ 187, state_->get_commit_idx() 187
2021.03.24 11:22:48.150885 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.150892 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.150901 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 187, starting_idx: 91, cur_nxt_idx: 189
2021.03.24 11:22:48.150912 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=187, LastLogTerm=1, EntriesLength=1, CommitIndex=187, Term=1, peer_last_sent_idx 188
2021.03.24 11:22:48.150919 [ 1093 ] {} <Debug> RaftInstance: idx: 188
2021.03.24 11:22:48.150941 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.150988 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.155140 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.155165 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=188
2021.03.24 11:22:48.155176 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.155185 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 188
2021.03.24 11:22:48.155192 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.155205 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 183, new matched idx: 187
2021.03.24 11:22:48.155215 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 188 187 187 
2021.03.24 11:22:48.155224 [ 1094 ] {} <Trace> RaftInstance: local log idx 188, target_commit_idx 187, quick_commit_index_ 187, state_->get_commit_idx() 187
2021.03.24 11:22:48.155232 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.155240 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.155248 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 187, starting_idx: 91, cur_nxt_idx: 189
2021.03.24 11:22:48.155259 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=187, LastLogTerm=1, EntriesLength=1, CommitIndex=187, Term=1, peer_last_sent_idx 184
2021.03.24 11:22:48.155266 [ 1094 ] {} <Debug> RaftInstance: idx: 188
2021.03.24 11:22:48.155275 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.155335 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.157306 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.157339 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 189
2021.03.24 11:22:48.157362 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 189 0x7faa0dbd7820 sleep
2021.03.24 11:22:48.157380 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.157397 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.157536 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.157553 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=189
2021.03.24 11:22:48.157561 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.157570 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 189
2021.03.24 11:22:48.157578 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.157586 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 187, new matched idx: 188
2021.03.24 11:22:48.157596 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 189 188 187 
2021.03.24 11:22:48.157603 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 188
2021.03.24 11:22:48.157624 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.157633 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.157641 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 188, starting_idx: 91, cur_nxt_idx: 190
2021.03.24 11:22:48.157651 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=188, LastLogTerm=1, EntriesLength=1, CommitIndex=188, Term=1, peer_last_sent_idx 188
2021.03.24 11:22:48.157659 [ 1093 ] {} <Debug> RaftInstance: idx: 189
2021.03.24 11:22:48.157667 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.157707 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.157720 [ 1093 ] {} <Trace> RaftInstance: local log idx 189, target_commit_idx 188, quick_commit_index_ 188, state_->get_commit_idx() 187
2021.03.24 11:22:48.157740 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.157757 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.157766 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.157816 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.157889 [ 1096 ] {} <Debug> RaftInstance: commit upto 188, current idx 187
2021.03.24 11:22:48.157911 [ 1096 ] {} <Trace> RaftInstance: commit upto 188, current idx 188
2021.03.24 11:22:48.158092 [ 1096 ] {} <Debug> RaftInstance: notify cb 188 0x7faa0daae120
2021.03.24 11:22:48.158117 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 188, current idx 188
2021.03.24 11:22:48.158127 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.158159 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.158181 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 188 wake up (11840 us), return value (nil)
2021.03.24 11:22:48.158724 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.158746 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 190
2021.03.24 11:22:48.158769 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 190 0x7faa0daae020 sleep
2021.03.24 11:22:48.158811 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.158828 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.164534 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.164568 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=190
2021.03.24 11:22:48.164579 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.164590 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 190
2021.03.24 11:22:48.164598 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.164607 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 188, new matched idx: 189
2021.03.24 11:22:48.164618 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 190 189 187 
2021.03.24 11:22:48.164626 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 189
2021.03.24 11:22:48.164635 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.164643 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.164655 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 189, starting_idx: 91, cur_nxt_idx: 191
2021.03.24 11:22:48.164667 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=189, LastLogTerm=1, EntriesLength=1, CommitIndex=189, Term=1, peer_last_sent_idx 189
2021.03.24 11:22:48.164675 [ 1093 ] {} <Debug> RaftInstance: idx: 190
2021.03.24 11:22:48.164684 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.164749 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.164762 [ 1093 ] {} <Trace> RaftInstance: local log idx 190, target_commit_idx 189, quick_commit_index_ 189, state_->get_commit_idx() 188
2021.03.24 11:22:48.164770 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.164789 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.164798 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.164842 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.164880 [ 1096 ] {} <Debug> RaftInstance: commit upto 189, current idx 188
2021.03.24 11:22:48.164896 [ 1096 ] {} <Trace> RaftInstance: commit upto 189, current idx 189
2021.03.24 11:22:48.164961 [ 1096 ] {} <Debug> RaftInstance: notify cb 189 0x7faa0dbd7820
2021.03.24 11:22:48.165002 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 189, current idx 189
2021.03.24 11:22:48.165010 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.165030 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.165042 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 189 wake up (7675 us), return value (nil)
2021.03.24 11:22:48.171450 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.171479 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=191
2021.03.24 11:22:48.171488 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.171498 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 191
2021.03.24 11:22:48.171506 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.171514 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 189, new matched idx: 190
2021.03.24 11:22:48.171525 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 190 190 187 
2021.03.24 11:22:48.171532 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 190
2021.03.24 11:22:48.171541 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.171549 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.171557 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 190, starting_idx: 91, cur_nxt_idx: 191
2021.03.24 11:22:48.171567 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=190, LastLogTerm=1, EntriesLength=0, CommitIndex=190, Term=1, peer_last_sent_idx 190
2021.03.24 11:22:48.171574 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.171583 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.171642 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.171656 [ 1093 ] {} <Trace> RaftInstance: local log idx 190, target_commit_idx 190, quick_commit_index_ 190, state_->get_commit_idx() 189
2021.03.24 11:22:48.171664 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.171729 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.171771 [ 1096 ] {} <Debug> RaftInstance: commit upto 190, current idx 189
2021.03.24 11:22:48.171782 [ 1096 ] {} <Trace> RaftInstance: commit upto 190, current idx 190
2021.03.24 11:22:48.171953 [ 1096 ] {} <Debug> RaftInstance: notify cb 190 0x7faa0daae020
2021.03.24 11:22:48.172004 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 190, current idx 190
2021.03.24 11:22:48.172015 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.172014 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.172058 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 190 wake up (13252 us), return value (nil)
2021.03.24 11:22:48.176011 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.176031 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=189
2021.03.24 11:22:48.176040 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.176049 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 189
2021.03.24 11:22:48.176056 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.176064 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 187, new matched idx: 188
2021.03.24 11:22:48.176074 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 190 190 188 
2021.03.24 11:22:48.176082 [ 1093 ] {} <Trace> RaftInstance: local log idx 190, target_commit_idx 190, quick_commit_index_ 190, state_->get_commit_idx() 190
2021.03.24 11:22:48.176104 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.176113 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.176134 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 188, starting_idx: 91, cur_nxt_idx: 191
2021.03.24 11:22:48.176145 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=188, LastLogTerm=1, EntriesLength=2, CommitIndex=190, Term=1, peer_last_sent_idx 188
2021.03.24 11:22:48.176152 [ 1093 ] {} <Debug> RaftInstance: idx range: 189-190
2021.03.24 11:22:48.176161 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.176203 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.178311 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.178330 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=191
2021.03.24 11:22:48.178339 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.178348 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 191
2021.03.24 11:22:48.178356 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.178364 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 190, new matched idx: 190
2021.03.24 11:22:48.178373 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 190 190 188 
2021.03.24 11:22:48.178382 [ 1093 ] {} <Trace> RaftInstance: local log idx 190, target_commit_idx 190, quick_commit_index_ 190, state_->get_commit_idx() 190
2021.03.24 11:22:48.185003 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.185060 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 191
2021.03.24 11:22:48.185142 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.185157 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.185166 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 190, starting_idx: 91, cur_nxt_idx: 192
2021.03.24 11:22:48.185178 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=190, LastLogTerm=1, EntriesLength=1, CommitIndex=190, Term=1, peer_last_sent_idx 191
2021.03.24 11:22:48.185186 [ 1097 ] {} <Debug> RaftInstance: idx: 191
2021.03.24 11:22:48.185196 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.185248 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.185265 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 191 0x7faa1f070b20 sleep
2021.03.24 11:22:48.190486 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.190513 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 192
2021.03.24 11:22:48.190537 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.190547 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.190559 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 192 0x7faa1f070f20 sleep
2021.03.24 11:22:48.191901 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.191958 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=192
2021.03.24 11:22:48.191969 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.191980 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 192
2021.03.24 11:22:48.191989 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.191997 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 190, new matched idx: 191
2021.03.24 11:22:48.192010 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 192 191 188 
2021.03.24 11:22:48.192048 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 191
2021.03.24 11:22:48.192059 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.192067 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.192075 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 191, starting_idx: 91, cur_nxt_idx: 193
2021.03.24 11:22:48.192088 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=191, LastLogTerm=1, EntriesLength=1, CommitIndex=191, Term=1, peer_last_sent_idx 191
2021.03.24 11:22:48.192096 [ 1095 ] {} <Debug> RaftInstance: idx: 192
2021.03.24 11:22:48.192105 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.192168 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.192181 [ 1095 ] {} <Trace> RaftInstance: local log idx 192, target_commit_idx 191, quick_commit_index_ 191, state_->get_commit_idx() 190
2021.03.24 11:22:48.192189 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.192214 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.192223 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.192254 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.192278 [ 1096 ] {} <Debug> RaftInstance: commit upto 191, current idx 190
2021.03.24 11:22:48.192288 [ 1096 ] {} <Trace> RaftInstance: commit upto 191, current idx 191
2021.03.24 11:22:48.192351 [ 1096 ] {} <Debug> RaftInstance: notify cb 191 0x7faa1f070b20
2021.03.24 11:22:48.192372 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 191, current idx 191
2021.03.24 11:22:48.192381 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.192393 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.192412 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 191 wake up (7315 us), return value (nil)
2021.03.24 11:22:48.196940 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.196975 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=191
2021.03.24 11:22:48.196984 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.197006 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 191
2021.03.24 11:22:48.197014 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.197022 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 188, new matched idx: 190
2021.03.24 11:22:48.197031 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 192 191 190 
2021.03.24 11:22:48.197040 [ 1095 ] {} <Trace> RaftInstance: local log idx 192, target_commit_idx 191, quick_commit_index_ 191, state_->get_commit_idx() 191
2021.03.24 11:22:48.197047 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.197056 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.197064 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 190, starting_idx: 91, cur_nxt_idx: 193
2021.03.24 11:22:48.197076 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=190, LastLogTerm=1, EntriesLength=2, CommitIndex=191, Term=1, peer_last_sent_idx 189
2021.03.24 11:22:48.197117 [ 1095 ] {} <Debug> RaftInstance: idx range: 191-192
2021.03.24 11:22:48.197132 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.197189 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.198880 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.198907 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=193
2021.03.24 11:22:48.198918 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.198945 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 193
2021.03.24 11:22:48.198954 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.198966 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 191, new matched idx: 192
2021.03.24 11:22:48.198977 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 192 192 190 
2021.03.24 11:22:48.198985 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 192
2021.03.24 11:22:48.198993 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.199001 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.199009 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 192, starting_idx: 91, cur_nxt_idx: 193
2021.03.24 11:22:48.199019 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=192, LastLogTerm=1, EntriesLength=0, CommitIndex=192, Term=1, peer_last_sent_idx 192
2021.03.24 11:22:48.199026 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.199034 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.199080 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.199094 [ 1094 ] {} <Trace> RaftInstance: local log idx 192, target_commit_idx 192, quick_commit_index_ 192, state_->get_commit_idx() 191
2021.03.24 11:22:48.199104 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.199134 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.199154 [ 1096 ] {} <Debug> RaftInstance: commit upto 192, current idx 191
2021.03.24 11:22:48.199163 [ 1096 ] {} <Trace> RaftInstance: commit upto 192, current idx 192
2021.03.24 11:22:48.199212 [ 1096 ] {} <Debug> RaftInstance: notify cb 192 0x7faa1f070f20
2021.03.24 11:22:48.199238 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 192, current idx 192
2021.03.24 11:22:48.199252 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.199300 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.199322 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 192 wake up (8768 us), return value (nil)
2021.03.24 11:22:48.205725 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.205771 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=193
2021.03.24 11:22:48.205780 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.205792 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 193
2021.03.24 11:22:48.205800 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.205808 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 192, new matched idx: 192
2021.03.24 11:22:48.205820 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 192 192 190 
2021.03.24 11:22:48.205829 [ 1095 ] {} <Trace> RaftInstance: local log idx 192, target_commit_idx 192, quick_commit_index_ 192, state_->get_commit_idx() 192
2021.03.24 11:22:48.212435 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.212488 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 193
2021.03.24 11:22:48.212520 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.212537 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.212546 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 192, starting_idx: 91, cur_nxt_idx: 194
2021.03.24 11:22:48.212559 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=192, LastLogTerm=1, EntriesLength=1, CommitIndex=192, Term=1, peer_last_sent_idx 193
2021.03.24 11:22:48.212567 [ 1097 ] {} <Debug> RaftInstance: idx: 193
2021.03.24 11:22:48.212578 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.212654 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.212673 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 193 0x7faa0dbd7820 sleep
2021.03.24 11:22:48.213630 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.213658 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 194
2021.03.24 11:22:48.213684 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.213695 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.213706 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 194 0x7faa1f070f20 sleep
2021.03.24 11:22:48.217733 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.217785 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=193
2021.03.24 11:22:48.217797 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.217809 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 193
2021.03.24 11:22:48.217817 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.217825 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 190, new matched idx: 192
2021.03.24 11:22:48.217837 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 194 192 192 
2021.03.24 11:22:48.217847 [ 1092 ] {} <Trace> RaftInstance: local log idx 194, target_commit_idx 192, quick_commit_index_ 192, state_->get_commit_idx() 192
2021.03.24 11:22:48.217855 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.217864 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.217872 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 192, starting_idx: 91, cur_nxt_idx: 195
2021.03.24 11:22:48.217885 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=192, LastLogTerm=1, EntriesLength=2, CommitIndex=192, Term=1, peer_last_sent_idx 191
2021.03.24 11:22:48.217892 [ 1092 ] {} <Debug> RaftInstance: idx range: 193-194
2021.03.24 11:22:48.217901 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.217966 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.219345 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.219363 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=194
2021.03.24 11:22:48.219371 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.219380 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 194
2021.03.24 11:22:48.219388 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.219396 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 192, new matched idx: 193
2021.03.24 11:22:48.219405 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 194 193 192 
2021.03.24 11:22:48.219413 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 193
2021.03.24 11:22:48.219421 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.219429 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.219437 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 193, starting_idx: 91, cur_nxt_idx: 195
2021.03.24 11:22:48.219446 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=193, LastLogTerm=1, EntriesLength=1, CommitIndex=193, Term=1, peer_last_sent_idx 193
2021.03.24 11:22:48.219454 [ 1092 ] {} <Debug> RaftInstance: idx: 194
2021.03.24 11:22:48.219462 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.219503 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.219517 [ 1092 ] {} <Trace> RaftInstance: local log idx 194, target_commit_idx 193, quick_commit_index_ 193, state_->get_commit_idx() 192
2021.03.24 11:22:48.219545 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.219558 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.219567 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.219586 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.219597 [ 1096 ] {} <Debug> RaftInstance: commit upto 193, current idx 192
2021.03.24 11:22:48.219606 [ 1096 ] {} <Trace> RaftInstance: commit upto 193, current idx 193
2021.03.24 11:22:48.219668 [ 1096 ] {} <Debug> RaftInstance: notify cb 193 0x7faa0dbd7820
2021.03.24 11:22:48.219690 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 193, current idx 193
2021.03.24 11:22:48.219699 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.219744 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.219769 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 193 wake up (7235 us), return value (nil)
2021.03.24 11:22:48.226220 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.226256 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=195
2021.03.24 11:22:48.226268 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.226280 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 195
2021.03.24 11:22:48.226289 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.226297 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 193, new matched idx: 194
2021.03.24 11:22:48.226309 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 194 194 192 
2021.03.24 11:22:48.226317 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 194
2021.03.24 11:22:48.226326 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.226334 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.226343 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 194, starting_idx: 91, cur_nxt_idx: 195
2021.03.24 11:22:48.226354 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=194, LastLogTerm=1, EntriesLength=0, CommitIndex=194, Term=1, peer_last_sent_idx 194
2021.03.24 11:22:48.226361 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.226370 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.226419 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.226432 [ 1095 ] {} <Trace> RaftInstance: local log idx 194, target_commit_idx 194, quick_commit_index_ 194, state_->get_commit_idx() 193
2021.03.24 11:22:48.226440 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.226496 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.226546 [ 1096 ] {} <Debug> RaftInstance: commit upto 194, current idx 193
2021.03.24 11:22:48.226558 [ 1096 ] {} <Trace> RaftInstance: commit upto 194, current idx 194
2021.03.24 11:22:48.226636 [ 1096 ] {} <Debug> RaftInstance: notify cb 194 0x7faa1f070f20
2021.03.24 11:22:48.226662 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 194, current idx 194
2021.03.24 11:22:48.226671 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.226692 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.226714 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 194 wake up (13015 us), return value (nil)
2021.03.24 11:22:48.233128 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.233198 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=195
2021.03.24 11:22:48.233210 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.233222 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 195
2021.03.24 11:22:48.233230 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.233239 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 194, new matched idx: 194
2021.03.24 11:22:48.233251 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 194 194 192 
2021.03.24 11:22:48.233260 [ 1092 ] {} <Trace> RaftInstance: local log idx 194, target_commit_idx 194, quick_commit_index_ 194, state_->get_commit_idx() 194
2021.03.24 11:22:48.235267 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.235312 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 195
2021.03.24 11:22:48.235355 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.235366 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.235376 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 194, starting_idx: 91, cur_nxt_idx: 196
2021.03.24 11:22:48.235407 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=194, LastLogTerm=1, EntriesLength=1, CommitIndex=194, Term=1, peer_last_sent_idx 195
2021.03.24 11:22:48.235415 [ 1097 ] {} <Debug> RaftInstance: idx: 195
2021.03.24 11:22:48.235426 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.235486 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.235511 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 195 0x7faa0dbd7820 sleep
2021.03.24 11:22:48.238474 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.238494 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=195
2021.03.24 11:22:48.238503 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.238512 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 195
2021.03.24 11:22:48.238520 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.238528 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 192, new matched idx: 194
2021.03.24 11:22:48.238538 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 195 194 194 
2021.03.24 11:22:48.238546 [ 1093 ] {} <Trace> RaftInstance: local log idx 195, target_commit_idx 194, quick_commit_index_ 194, state_->get_commit_idx() 194
2021.03.24 11:22:48.238554 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.238562 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.238569 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 194, starting_idx: 91, cur_nxt_idx: 196
2021.03.24 11:22:48.238579 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=194, LastLogTerm=1, EntriesLength=1, CommitIndex=194, Term=1, peer_last_sent_idx 193
2021.03.24 11:22:48.238586 [ 1093 ] {} <Debug> RaftInstance: idx: 195
2021.03.24 11:22:48.238594 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.238632 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.239564 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.239598 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 196
2021.03.24 11:22:48.239618 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 196 0x7faa0dbd7920 sleep
2021.03.24 11:22:48.239632 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.239647 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.242208 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.242225 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=196
2021.03.24 11:22:48.242233 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.242242 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 196
2021.03.24 11:22:48.242251 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.242258 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 194, new matched idx: 195
2021.03.24 11:22:48.242267 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 196 195 194 
2021.03.24 11:22:48.242274 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 195
2021.03.24 11:22:48.242282 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.242289 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.242297 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 195, starting_idx: 91, cur_nxt_idx: 197
2021.03.24 11:22:48.242306 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=195, LastLogTerm=1, EntriesLength=1, CommitIndex=195, Term=1, peer_last_sent_idx 195
2021.03.24 11:22:48.242313 [ 1093 ] {} <Debug> RaftInstance: idx: 196
2021.03.24 11:22:48.242321 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.242359 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.242371 [ 1093 ] {} <Trace> RaftInstance: local log idx 196, target_commit_idx 195, quick_commit_index_ 195, state_->get_commit_idx() 194
2021.03.24 11:22:48.242379 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.242393 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.242401 [ 1093 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.242425 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.242472 [ 1096 ] {} <Debug> RaftInstance: commit upto 195, current idx 194
2021.03.24 11:22:48.242486 [ 1096 ] {} <Trace> RaftInstance: commit upto 195, current idx 195
2021.03.24 11:22:48.242547 [ 1096 ] {} <Debug> RaftInstance: notify cb 195 0x7faa0dbd7820
2021.03.24 11:22:48.242571 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 195, current idx 195
2021.03.24 11:22:48.242581 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.242586 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.242607 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 195 wake up (7258 us), return value (nil)
2021.03.24 11:22:48.242770 [ 1092 ] {} <Debug> RaftInstance: heartbeat timeout for 1
2021.03.24 11:22:48.242814 [ 1092 ] {} <Trace> RaftInstance: (update) new target priority: 1
2021.03.24 11:22:48.242835 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.249214 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.249235 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=197
2021.03.24 11:22:48.249244 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.249253 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 197
2021.03.24 11:22:48.249261 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.249269 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 195, new matched idx: 196
2021.03.24 11:22:48.249279 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 196 196 194 
2021.03.24 11:22:48.249286 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 196
2021.03.24 11:22:48.249295 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.249302 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.249324 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 196, starting_idx: 91, cur_nxt_idx: 197
2021.03.24 11:22:48.249335 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=196, LastLogTerm=1, EntriesLength=0, CommitIndex=196, Term=1, peer_last_sent_idx 196
2021.03.24 11:22:48.249342 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.249350 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.249420 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.249435 [ 1094 ] {} <Trace> RaftInstance: local log idx 196, target_commit_idx 196, quick_commit_index_ 196, state_->get_commit_idx() 195
2021.03.24 11:22:48.249442 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.249505 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.249549 [ 1096 ] {} <Debug> RaftInstance: commit upto 196, current idx 195
2021.03.24 11:22:48.249560 [ 1096 ] {} <Trace> RaftInstance: commit upto 196, current idx 196
2021.03.24 11:22:48.249609 [ 1096 ] {} <Debug> RaftInstance: notify cb 196 0x7faa0dbd7920
2021.03.24 11:22:48.249632 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 196, current idx 196
2021.03.24 11:22:48.249641 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.249679 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.249699 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 196 wake up (10063 us), return value (nil)
2021.03.24 11:22:48.253348 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.253377 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 197
2021.03.24 11:22:48.253400 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 197 0x7faa0daae120 sleep
2021.03.24 11:22:48.253423 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.253439 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.256092 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.256109 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=197
2021.03.24 11:22:48.256118 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.256126 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 197
2021.03.24 11:22:48.256134 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.256144 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 196, new matched idx: 196
2021.03.24 11:22:48.256153 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 197 196 194 
2021.03.24 11:22:48.256161 [ 1093 ] {} <Trace> RaftInstance: local log idx 197, target_commit_idx 196, quick_commit_index_ 196, state_->get_commit_idx() 196
2021.03.24 11:22:48.256169 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.256176 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.256184 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 196, starting_idx: 91, cur_nxt_idx: 198
2021.03.24 11:22:48.256194 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=196, LastLogTerm=1, EntriesLength=1, CommitIndex=196, Term=1, peer_last_sent_idx 197
2021.03.24 11:22:48.256201 [ 1093 ] {} <Debug> RaftInstance: idx: 197
2021.03.24 11:22:48.256210 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.256257 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.257873 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.258005 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 198
2021.03.24 11:22:48.258031 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 198 0x7faa1f070120 sleep
2021.03.24 11:22:48.258058 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.258078 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.259203 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.259227 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=196
2021.03.24 11:22:48.259237 [ 1094 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.259246 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 196
2021.03.24 11:22:48.259254 [ 1094 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.259262 [ 1094 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 194, new matched idx: 195
2021.03.24 11:22:48.259271 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 198 196 195 
2021.03.24 11:22:48.259280 [ 1094 ] {} <Trace> RaftInstance: local log idx 198, target_commit_idx 196, quick_commit_index_ 196, state_->get_commit_idx() 196
2021.03.24 11:22:48.259288 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.259296 [ 1094 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.259304 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 195, starting_idx: 91, cur_nxt_idx: 199
2021.03.24 11:22:48.259315 [ 1094 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=195, LastLogTerm=1, EntriesLength=3, CommitIndex=196, Term=1, peer_last_sent_idx 195
2021.03.24 11:22:48.259323 [ 1094 ] {} <Debug> RaftInstance: idx range: 196-198
2021.03.24 11:22:48.259332 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.259387 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.263210 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.263234 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=198
2021.03.24 11:22:48.263252 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.263262 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 198
2021.03.24 11:22:48.263270 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.263278 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 196, new matched idx: 197
2021.03.24 11:22:48.263287 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 198 197 195 
2021.03.24 11:22:48.263295 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 197
2021.03.24 11:22:48.263303 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.263311 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.263319 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 197, starting_idx: 91, cur_nxt_idx: 199
2021.03.24 11:22:48.263330 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=197, LastLogTerm=1, EntriesLength=1, CommitIndex=197, Term=1, peer_last_sent_idx 197
2021.03.24 11:22:48.263337 [ 1095 ] {} <Debug> RaftInstance: idx: 198
2021.03.24 11:22:48.263346 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.263390 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.263412 [ 1095 ] {} <Trace> RaftInstance: local log idx 198, target_commit_idx 197, quick_commit_index_ 197, state_->get_commit_idx() 196
2021.03.24 11:22:48.263419 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.263435 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.263444 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.263497 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.263541 [ 1096 ] {} <Debug> RaftInstance: commit upto 197, current idx 196
2021.03.24 11:22:48.263570 [ 1096 ] {} <Trace> RaftInstance: commit upto 197, current idx 197
2021.03.24 11:22:48.263642 [ 1096 ] {} <Debug> RaftInstance: notify cb 197 0x7faa0daae120
2021.03.24 11:22:48.263665 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 197, current idx 197
2021.03.24 11:22:48.263674 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.263688 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.263716 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 197 wake up (10294 us), return value (nil)
2021.03.24 11:22:48.264768 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.264791 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 199
2021.03.24 11:22:48.264811 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 199 0x7faa0daae020 sleep
2021.03.24 11:22:48.264863 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.264888 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.269490 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.269517 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 200
2021.03.24 11:22:48.269543 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.269552 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.269564 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 200 0x7faa0dbd7820 sleep
2021.03.24 11:22:48.270249 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.270269 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=199
2021.03.24 11:22:48.270277 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.270287 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 199
2021.03.24 11:22:48.270295 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.270303 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 197, new matched idx: 198
2021.03.24 11:22:48.270312 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 200 198 195 
2021.03.24 11:22:48.270320 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 198
2021.03.24 11:22:48.270328 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.270335 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.270343 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 198, starting_idx: 91, cur_nxt_idx: 201
2021.03.24 11:22:48.270354 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=198, LastLogTerm=1, EntriesLength=2, CommitIndex=198, Term=1, peer_last_sent_idx 198
2021.03.24 11:22:48.270361 [ 1095 ] {} <Debug> RaftInstance: idx range: 199-200
2021.03.24 11:22:48.270369 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.270419 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.270432 [ 1095 ] {} <Trace> RaftInstance: local log idx 200, target_commit_idx 198, quick_commit_index_ 198, state_->get_commit_idx() 197
2021.03.24 11:22:48.270440 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.270459 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.270468 [ 1095 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.270500 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.270522 [ 1096 ] {} <Debug> RaftInstance: commit upto 198, current idx 197
2021.03.24 11:22:48.270535 [ 1096 ] {} <Trace> RaftInstance: commit upto 198, current idx 198
2021.03.24 11:22:48.270626 [ 1096 ] {} <Debug> RaftInstance: notify cb 198 0x7faa1f070120
2021.03.24 11:22:48.270641 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 198, current idx 198
2021.03.24 11:22:48.270648 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.270664 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:48.270674 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 198 wake up (12642 us), return value (nil)
2021.03.24 11:22:48.277165 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.277188 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=201
2021.03.24 11:22:48.277197 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.277206 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 201
2021.03.24 11:22:48.277214 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.277222 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 198, new matched idx: 200
2021.03.24 11:22:48.277232 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 200 200 195 
2021.03.24 11:22:48.277285 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 200
2021.03.24 11:22:48.277299 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.277307 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.277316 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 200, starting_idx: 91, cur_nxt_idx: 201
2021.03.24 11:22:48.277326 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=200, LastLogTerm=1, EntriesLength=0, CommitIndex=200, Term=1, peer_last_sent_idx 199
2021.03.24 11:22:48.277337 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.277345 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.277384 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.277397 [ 1095 ] {} <Trace> RaftInstance: local log idx 200, target_commit_idx 200, quick_commit_index_ 200, state_->get_commit_idx() 198
2021.03.24 11:22:48.277404 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.277441 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.277462 [ 1096 ] {} <Debug> RaftInstance: commit upto 200, current idx 198
2021.03.24 11:22:48.277472 [ 1096 ] {} <Trace> RaftInstance: commit upto 200, current idx 199
2021.03.24 11:22:48.277528 [ 1096 ] {} <Debug> RaftInstance: notify cb 199 0x7faa0daae020
2021.03.24 11:22:48.277551 [ 1096 ] {} <Trace> RaftInstance: commit upto 200, current idx 200
2021.03.24 11:22:48.277570 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.277605 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 199 wake up (12764 us), return value (nil)
2021.03.24 11:22:48.277621 [ 1096 ] {} <Debug> RaftInstance: notify cb 200 0x7faa0dbd7820
2021.03.24 11:22:48.277648 [ 1096 ] {} <Information> RaftInstance: creating a snapshot for index 200
2021.03.24 11:22:48.277659 [ 1096 ] {} <Information> RaftInstance: create snapshot idx 200 log_term 1
2021.03.24 11:22:48.277670 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.277690 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 200 wake up (8135 us), return value (nil)
2021.03.24 11:22:48.277730 [ 1096 ] {} <Information> RaftInstance: create snapshot idx 200 log_term 1 done: 61 us elapsed
2021.03.24 11:22:48.277749 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 200, current idx 200
2021.03.24 11:22:48.277757 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.279862 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.299433 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=199
2021.03.24 11:22:48.284110 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.295874 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.297531 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.299462 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.299562 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 201
2021.03.24 11:22:48.299622 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 199
2021.03.24 11:22:48.299656 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.299666 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 195, new matched idx: 198
2021.03.24 11:22:48.299672 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 201 0x7faa0dbd7820 sleep
2021.03.24 11:22:48.299678 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 201 200 198 
2021.03.24 11:22:48.299725 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 202
2021.03.24 11:22:48.299728 [ 1092 ] {} <Trace> RaftInstance: local log idx 202, target_commit_idx 200, quick_commit_index_ 200, state_->get_commit_idx() 200
2021.03.24 11:22:48.299749 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 202 0x7faa1f021020 sleep
2021.03.24 11:22:48.299756 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.299766 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.299775 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 198, starting_idx: 91, cur_nxt_idx: 203
2021.03.24 11:22:48.299787 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=198, LastLogTerm=1, EntriesLength=4, CommitIndex=200, Term=1, peer_last_sent_idx 196
2021.03.24 11:22:48.299795 [ 1092 ] {} <Debug> RaftInstance: idx range: 199-202
2021.03.24 11:22:48.299805 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.299858 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.299934 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=201
2021.03.24 11:22:48.299956 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.299967 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 201
2021.03.24 11:22:48.299975 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.299988 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 200, new matched idx: 200
2021.03.24 11:22:48.299998 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 202 200 198 
2021.03.24 11:22:48.300007 [ 1094 ] {} <Trace> RaftInstance: local log idx 202, target_commit_idx 200, quick_commit_index_ 200, state_->get_commit_idx() 200
2021.03.24 11:22:48.300014 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.300022 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.300031 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 200, starting_idx: 91, cur_nxt_idx: 203
2021.03.24 11:22:48.300039 [ 1094 ] {} <Debug> RaftInstance: last sent log (201) to peer 1 is not applied, cnt 1
2021.03.24 11:22:48.300049 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=200, LastLogTerm=1, EntriesLength=2, CommitIndex=200, Term=1, peer_last_sent_idx 201
2021.03.24 11:22:48.300072 [ 1094 ] {} <Debug> RaftInstance: idx range: 201-202
2021.03.24 11:22:48.300081 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.300134 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.300174 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.300188 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.300198 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.300206 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.304594 [ 1103 ] {} <Debug> RaftInstance: snapshot created, compact the log store
2021.03.24 11:22:48.304605 [ 1103 ] {} <Debug> RaftInstance: log_store_ compact upto 190
2021.03.24 11:22:48.307207 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.307232 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=203
2021.03.24 11:22:48.307241 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.307250 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 203
2021.03.24 11:22:48.307258 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.307265 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 200, new matched idx: 202
2021.03.24 11:22:48.307275 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 202 202 198 
2021.03.24 11:22:48.307282 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 202
2021.03.24 11:22:48.307290 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.307298 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.307306 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 202, starting_idx: 191, cur_nxt_idx: 203
2021.03.24 11:22:48.307315 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=202, LastLogTerm=1, EntriesLength=0, CommitIndex=202, Term=1, peer_last_sent_idx 201
2021.03.24 11:22:48.307322 [ 1092 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.307330 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.307372 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.307385 [ 1092 ] {} <Trace> RaftInstance: local log idx 202, target_commit_idx 202, quick_commit_index_ 202, state_->get_commit_idx() 200
2021.03.24 11:22:48.307393 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.307416 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.307428 [ 1096 ] {} <Debug> RaftInstance: commit upto 202, current idx 200
2021.03.24 11:22:48.307436 [ 1096 ] {} <Trace> RaftInstance: commit upto 202, current idx 201
2021.03.24 11:22:48.307477 [ 1096 ] {} <Debug> RaftInstance: notify cb 201 0x7faa0dbd7820
2021.03.24 11:22:48.307499 [ 1096 ] {} <Trace> RaftInstance: commit upto 202, current idx 202
2021.03.24 11:22:48.307520 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.307542 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 201 wake up (7868 us), return value (nil)
2021.03.24 11:22:48.307548 [ 1096 ] {} <Debug> RaftInstance: notify cb 202 0x7faa1f021020
2021.03.24 11:22:48.307609 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 202, current idx 202
2021.03.24 11:22:48.307619 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.307636 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.307662 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 202 wake up (7882 us), return value (nil)
2021.03.24 11:22:48.311866 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.311901 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 203
2021.03.24 11:22:48.311925 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 203 0x7faa0daae120 sleep
2021.03.24 11:22:48.311945 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.311963 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.314076 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.314098 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=203
2021.03.24 11:22:48.314108 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.314118 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 203
2021.03.24 11:22:48.314126 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.314134 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 202, new matched idx: 202
2021.03.24 11:22:48.314144 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 203 202 198 
2021.03.24 11:22:48.314152 [ 1094 ] {} <Trace> RaftInstance: local log idx 203, target_commit_idx 202, quick_commit_index_ 202, state_->get_commit_idx() 202
2021.03.24 11:22:48.314160 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.314168 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.314177 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 202, starting_idx: 191, cur_nxt_idx: 204
2021.03.24 11:22:48.314188 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=202, LastLogTerm=1, EntriesLength=1, CommitIndex=202, Term=1, peer_last_sent_idx 203
2021.03.24 11:22:48.314195 [ 1094 ] {} <Debug> RaftInstance: idx: 203
2021.03.24 11:22:48.314204 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.314260 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.315838 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.315863 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 204
2021.03.24 11:22:48.315881 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 204 0x7faa1f070f20 sleep
2021.03.24 11:22:48.315896 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.315913 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.320368 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.320389 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=203
2021.03.24 11:22:48.320399 [ 1093 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.320407 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 203
2021.03.24 11:22:48.320415 [ 1093 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.320423 [ 1093 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 198, new matched idx: 202
2021.03.24 11:22:48.320432 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 204 202 202 
2021.03.24 11:22:48.320441 [ 1093 ] {} <Trace> RaftInstance: local log idx 204, target_commit_idx 202, quick_commit_index_ 202, state_->get_commit_idx() 202
2021.03.24 11:22:48.320449 [ 1093 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.320456 [ 1093 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.320465 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 202, starting_idx: 191, cur_nxt_idx: 205
2021.03.24 11:22:48.320475 [ 1093 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=202, LastLogTerm=1, EntriesLength=2, CommitIndex=202, Term=1, peer_last_sent_idx 199
2021.03.24 11:22:48.320496 [ 1093 ] {} <Debug> RaftInstance: idx range: 203-204
2021.03.24 11:22:48.320505 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.320548 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.320931 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.320949 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=204
2021.03.24 11:22:48.320958 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.320967 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 204
2021.03.24 11:22:48.320975 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.320983 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 202, new matched idx: 203
2021.03.24 11:22:48.320992 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 204 203 202 
2021.03.24 11:22:48.321000 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 203
2021.03.24 11:22:48.321008 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.321016 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.321023 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 203, starting_idx: 191, cur_nxt_idx: 205
2021.03.24 11:22:48.321033 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=203, LastLogTerm=1, EntriesLength=1, CommitIndex=203, Term=1, peer_last_sent_idx 203
2021.03.24 11:22:48.321041 [ 1094 ] {} <Debug> RaftInstance: idx: 204
2021.03.24 11:22:48.321049 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.321099 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.321114 [ 1094 ] {} <Trace> RaftInstance: local log idx 204, target_commit_idx 203, quick_commit_index_ 203, state_->get_commit_idx() 202
2021.03.24 11:22:48.321122 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.321138 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.321148 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.321168 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.321186 [ 1096 ] {} <Debug> RaftInstance: commit upto 203, current idx 202
2021.03.24 11:22:48.321201 [ 1096 ] {} <Trace> RaftInstance: commit upto 203, current idx 203
2021.03.24 11:22:48.321255 [ 1096 ] {} <Debug> RaftInstance: notify cb 203 0x7faa0daae120
2021.03.24 11:22:48.321290 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 203, current idx 203
2021.03.24 11:22:48.321299 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.321330 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.321346 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 203 wake up (9412 us), return value (nil)
2021.03.24 11:22:48.321938 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.321961 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 205
2021.03.24 11:22:48.321980 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 205 0x7faa0daae020 sleep
2021.03.24 11:22:48.321999 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.322022 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.327436 [ 1093 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.327487 [ 1093 ] {} <Debug> RaftInstance: append at log_idx 206
2021.03.24 11:22:48.327514 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.327524 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.327542 [ 1093 ] {} <Debug> RaftInstance: commit_ret_cv 206 0x7faa1f070120 sleep
2021.03.24 11:22:48.327650 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.327670 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=205
2021.03.24 11:22:48.327679 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.327688 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 205
2021.03.24 11:22:48.327696 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.327704 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 203, new matched idx: 204
2021.03.24 11:22:48.327714 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 206 204 202 
2021.03.24 11:22:48.327722 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 204
2021.03.24 11:22:48.327730 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.327738 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.327746 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 204, starting_idx: 191, cur_nxt_idx: 207
2021.03.24 11:22:48.327757 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=204, LastLogTerm=1, EntriesLength=2, CommitIndex=204, Term=1, peer_last_sent_idx 204
2021.03.24 11:22:48.327765 [ 1094 ] {} <Debug> RaftInstance: idx range: 205-206
2021.03.24 11:22:48.327774 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.327853 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.327868 [ 1094 ] {} <Trace> RaftInstance: local log idx 206, target_commit_idx 204, quick_commit_index_ 204, state_->get_commit_idx() 203
2021.03.24 11:22:48.327877 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.327896 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.327906 [ 1094 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.327913 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.327940 [ 1096 ] {} <Debug> RaftInstance: commit upto 204, current idx 203
2021.03.24 11:22:48.327949 [ 1096 ] {} <Trace> RaftInstance: commit upto 204, current idx 204
2021.03.24 11:22:48.328001 [ 1096 ] {} <Debug> RaftInstance: notify cb 204 0x7faa1f070f20
2021.03.24 11:22:48.328016 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 204, current idx 204
2021.03.24 11:22:48.328024 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.328048 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:48.328057 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 204 wake up (12165 us), return value (nil)
2021.03.24 11:22:48.334623 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.334649 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=207
2021.03.24 11:22:48.334658 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.334668 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 207
2021.03.24 11:22:48.334676 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.334684 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 204, new matched idx: 206
2021.03.24 11:22:48.334694 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 206 206 202 
2021.03.24 11:22:48.334703 [ 1094 ] {} <Debug> RaftInstance: trigger commit upto 206
2021.03.24 11:22:48.334711 [ 1094 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.334719 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.334727 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 206, starting_idx: 191, cur_nxt_idx: 207
2021.03.24 11:22:48.334749 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=206, LastLogTerm=1, EntriesLength=0, CommitIndex=206, Term=1, peer_last_sent_idx 205
2021.03.24 11:22:48.334757 [ 1094 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.334765 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.334838 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.334852 [ 1094 ] {} <Trace> RaftInstance: local log idx 206, target_commit_idx 206, quick_commit_index_ 206, state_->get_commit_idx() 204
2021.03.24 11:22:48.334860 [ 1094 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.334932 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.334955 [ 1096 ] {} <Debug> RaftInstance: commit upto 206, current idx 204
2021.03.24 11:22:48.334965 [ 1096 ] {} <Trace> RaftInstance: commit upto 206, current idx 205
2021.03.24 11:22:48.335051 [ 1096 ] {} <Debug> RaftInstance: notify cb 205 0x7faa0daae020
2021.03.24 11:22:48.335071 [ 1096 ] {} <Trace> RaftInstance: commit upto 206, current idx 206
2021.03.24 11:22:48.335093 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.335116 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 205 wake up (13116 us), return value (nil)
2021.03.24 11:22:48.335126 [ 1096 ] {} <Debug> RaftInstance: notify cb 206 0x7faa1f070120
2021.03.24 11:22:48.335138 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.335147 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 206, current idx 206
2021.03.24 11:22:48.335156 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.335157 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 207
2021.03.24 11:22:48.335169 [ 1093 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.335193 [ 1093 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 206 wake up (7664 us), return value (nil)
2021.03.24 11:22:48.335205 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 207 0x7faa0daae120 sleep
2021.03.24 11:22:48.335215 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.335268 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.336270 [ 1092 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.336294 [ 1092 ] {} <Debug> RaftInstance: append at log_idx 208
2021.03.24 11:22:48.336313 [ 1092 ] {} <Debug> RaftInstance: commit_ret_cv 208 0x7faa1f070f20 sleep
2021.03.24 11:22:48.336327 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.336341 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.341542 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.341570 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=207
2021.03.24 11:22:48.341581 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.341590 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 207
2021.03.24 11:22:48.341598 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.341607 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 206, new matched idx: 206
2021.03.24 11:22:48.341617 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 208 206 202 
2021.03.24 11:22:48.341625 [ 1095 ] {} <Trace> RaftInstance: local log idx 208, target_commit_idx 206, quick_commit_index_ 206, state_->get_commit_idx() 206
2021.03.24 11:22:48.341650 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.341658 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.341667 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 206, starting_idx: 191, cur_nxt_idx: 209
2021.03.24 11:22:48.341675 [ 1095 ] {} <Debug> RaftInstance: last sent log (207) to peer 1 is not applied, cnt 1
2021.03.24 11:22:48.341686 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=206, LastLogTerm=1, EntriesLength=2, CommitIndex=206, Term=1, peer_last_sent_idx 207
2021.03.24 11:22:48.341694 [ 1095 ] {} <Debug> RaftInstance: idx range: 207-208
2021.03.24 11:22:48.341703 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.341752 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.342643 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.342660 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=205
2021.03.24 11:22:48.342668 [ 1095 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.342677 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 205
2021.03.24 11:22:48.342684 [ 1095 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.342697 [ 1095 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 202, new matched idx: 204
2021.03.24 11:22:48.342706 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 208 206 204 
2021.03.24 11:22:48.342714 [ 1095 ] {} <Trace> RaftInstance: local log idx 208, target_commit_idx 206, quick_commit_index_ 206, state_->get_commit_idx() 206
2021.03.24 11:22:48.342722 [ 1095 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.342729 [ 1095 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.342737 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 204, starting_idx: 191, cur_nxt_idx: 209
2021.03.24 11:22:48.342747 [ 1095 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=204, LastLogTerm=1, EntriesLength=4, CommitIndex=206, Term=1, peer_last_sent_idx 203
2021.03.24 11:22:48.342760 [ 1095 ] {} <Debug> RaftInstance: idx range: 205-208
2021.03.24 11:22:48.342768 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.342849 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.348580 [ 1093 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.348620 [ 1093 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=209
2021.03.24 11:22:48.348630 [ 1093 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.348641 [ 1093 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 209
2021.03.24 11:22:48.348650 [ 1093 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.348658 [ 1093 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 206, new matched idx: 208
2021.03.24 11:22:48.348669 [ 1093 ] {} <Trace> RaftInstance: quorum idx 1, 208 208 204 
2021.03.24 11:22:48.348677 [ 1093 ] {} <Debug> RaftInstance: trigger commit upto 208
2021.03.24 11:22:48.348687 [ 1093 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.348695 [ 1093 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.348704 [ 1093 ] {} <Trace> RaftInstance: last_log_idx: 208, starting_idx: 191, cur_nxt_idx: 209
2021.03.24 11:22:48.348715 [ 1093 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=208, LastLogTerm=1, EntriesLength=0, CommitIndex=208, Term=1, peer_last_sent_idx 207
2021.03.24 11:22:48.348722 [ 1093 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.348730 [ 1093 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.348800 [ 1093 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.348813 [ 1093 ] {} <Trace> RaftInstance: local log idx 208, target_commit_idx 208, quick_commit_index_ 208, state_->get_commit_idx() 206
2021.03.24 11:22:48.348821 [ 1093 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.348894 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.348947 [ 1096 ] {} <Debug> RaftInstance: commit upto 208, current idx 206
2021.03.24 11:22:48.348959 [ 1096 ] {} <Trace> RaftInstance: commit upto 208, current idx 207
2021.03.24 11:22:48.349041 [ 1096 ] {} <Debug> RaftInstance: notify cb 207 0x7faa0daae120
2021.03.24 11:22:48.349065 [ 1096 ] {} <Trace> RaftInstance: commit upto 208, current idx 208
2021.03.24 11:22:48.349102 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.349125 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 207 wake up (13917 us), return value (nil)
2021.03.24 11:22:48.349151 [ 1096 ] {} <Debug> RaftInstance: notify cb 208 0x7faa1f070f20
2021.03.24 11:22:48.349166 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 208, current idx 208
2021.03.24 11:22:48.349173 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.349192 [ 1092 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.349203 [ 1092 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 208 wake up (12883 us), return value (nil)
2021.03.24 11:22:48.349705 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.349736 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 209
2021.03.24 11:22:48.349759 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 209 0x7faa0daae020 sleep
2021.03.24 11:22:48.349780 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.349827 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.355225 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.355252 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 210
2021.03.24 11:22:48.355272 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 210 0x7faa0dbd7820 sleep
2021.03.24 11:22:48.355295 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.355313 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.355350 [ 1094 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.355365 [ 1094 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=209
2021.03.24 11:22:48.355374 [ 1094 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.355383 [ 1094 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 209
2021.03.24 11:22:48.355391 [ 1094 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.355399 [ 1094 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 208, new matched idx: 208
2021.03.24 11:22:48.355409 [ 1094 ] {} <Trace> RaftInstance: quorum idx 1, 210 208 204 
2021.03.24 11:22:48.355417 [ 1094 ] {} <Trace> RaftInstance: local log idx 210, target_commit_idx 208, quick_commit_index_ 208, state_->get_commit_idx() 208
2021.03.24 11:22:48.355425 [ 1094 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.355432 [ 1094 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.355441 [ 1094 ] {} <Trace> RaftInstance: last_log_idx: 208, starting_idx: 191, cur_nxt_idx: 211
2021.03.24 11:22:48.355449 [ 1094 ] {} <Debug> RaftInstance: last sent log (209) to peer 1 is not applied, cnt 1
2021.03.24 11:22:48.355460 [ 1094 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=208, LastLogTerm=1, EntriesLength=2, CommitIndex=208, Term=1, peer_last_sent_idx 209
2021.03.24 11:22:48.355477 [ 1094 ] {} <Debug> RaftInstance: idx range: 209-210
2021.03.24 11:22:48.355486 [ 1094 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.355540 [ 1094 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.357608 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.357640 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 211
2021.03.24 11:22:48.357673 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.357683 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.357695 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 211 0x7faa0dbd7920 sleep
2021.03.24 11:22:48.362213 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.362267 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=211
2021.03.24 11:22:48.362278 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.362288 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 211
2021.03.24 11:22:48.362296 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.362304 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 208, new matched idx: 210
2021.03.24 11:22:48.362316 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 211 210 204 
2021.03.24 11:22:48.362323 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 210
2021.03.24 11:22:48.362338 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.362345 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.362353 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 210, starting_idx: 191, cur_nxt_idx: 212
2021.03.24 11:22:48.362365 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=210, LastLogTerm=1, EntriesLength=1, CommitIndex=210, Term=1, peer_last_sent_idx 209
2021.03.24 11:22:48.362372 [ 1092 ] {} <Debug> RaftInstance: idx: 211
2021.03.24 11:22:48.362381 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.362445 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.362458 [ 1092 ] {} <Trace> RaftInstance: local log idx 211, target_commit_idx 210, quick_commit_index_ 210, state_->get_commit_idx() 208
2021.03.24 11:22:48.362465 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.362476 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.362485 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.362516 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.362527 [ 1096 ] {} <Debug> RaftInstance: commit upto 210, current idx 208
2021.03.24 11:22:48.362536 [ 1096 ] {} <Trace> RaftInstance: commit upto 210, current idx 209
2021.03.24 11:22:48.362686 [ 1096 ] {} <Debug> RaftInstance: notify cb 209 0x7faa0daae020
2021.03.24 11:22:48.362712 [ 1096 ] {} <Trace> RaftInstance: commit upto 210, current idx 210
2021.03.24 11:22:48.362749 [ 1108 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 2
2021.03.24 11:22:48.362804 [ 1108 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 209 wake up (12990 us), return value (nil)
2021.03.24 11:22:48.362822 [ 1096 ] {} <Debug> RaftInstance: notify cb 210 0x7faa0dbd7820
2021.03.24 11:22:48.362846 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 210, current idx 210
2021.03.24 11:22:48.362889 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.362858 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.362924 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 210 wake up (7593 us), return value (nil)
2021.03.24 11:22:48.363356 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.363374 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=209
2021.03.24 11:22:48.363382 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.363391 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 209
2021.03.24 11:22:48.363398 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.363406 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 204, new matched idx: 208
2021.03.24 11:22:48.363414 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 211 210 208 
2021.03.24 11:22:48.363422 [ 1092 ] {} <Trace> RaftInstance: local log idx 211, target_commit_idx 210, quick_commit_index_ 210, state_->get_commit_idx() 210
2021.03.24 11:22:48.363429 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.363437 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.363444 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 208, starting_idx: 191, cur_nxt_idx: 212
2021.03.24 11:22:48.363454 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=208, LastLogTerm=1, EntriesLength=3, CommitIndex=210, Term=1, peer_last_sent_idx 205
2021.03.24 11:22:48.363461 [ 1092 ] {} <Debug> RaftInstance: idx range: 209-211
2021.03.24 11:22:48.363469 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.363508 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.369238 [ 1095 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.369278 [ 1095 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=212
2021.03.24 11:22:48.369289 [ 1095 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.369299 [ 1095 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 212
2021.03.24 11:22:48.369309 [ 1095 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.369317 [ 1095 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 210, new matched idx: 211
2021.03.24 11:22:48.369329 [ 1095 ] {} <Trace> RaftInstance: quorum idx 1, 211 211 208 
2021.03.24 11:22:48.369337 [ 1095 ] {} <Debug> RaftInstance: trigger commit upto 211
2021.03.24 11:22:48.369346 [ 1095 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.369354 [ 1095 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.369363 [ 1095 ] {} <Trace> RaftInstance: last_log_idx: 211, starting_idx: 191, cur_nxt_idx: 212
2021.03.24 11:22:48.369374 [ 1095 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=211, LastLogTerm=1, EntriesLength=0, CommitIndex=211, Term=1, peer_last_sent_idx 211
2021.03.24 11:22:48.369381 [ 1095 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2021.03.24 11:22:48.369389 [ 1095 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.369438 [ 1095 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.369451 [ 1095 ] {} <Trace> RaftInstance: local log idx 211, target_commit_idx 211, quick_commit_index_ 211, state_->get_commit_idx() 210
2021.03.24 11:22:48.369463 [ 1095 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.369495 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.369532 [ 1096 ] {} <Debug> RaftInstance: commit upto 211, current idx 210
2021.03.24 11:22:48.369542 [ 1096 ] {} <Trace> RaftInstance: commit upto 211, current idx 211
2021.03.24 11:22:48.369602 [ 1096 ] {} <Debug> RaftInstance: notify cb 211 0x7faa0dbd7920
2021.03.24 11:22:48.369623 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 211, current idx 211
2021.03.24 11:22:48.369631 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.369679 [ 1094 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 0
2021.03.24 11:22:48.369704 [ 1094 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 211 wake up (12019 us), return value (nil)
2021.03.24 11:22:48.376325 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.376366 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=212
2021.03.24 11:22:48.376376 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.376386 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 212
2021.03.24 11:22:48.376394 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.376402 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 211, new matched idx: 211
2021.03.24 11:22:48.376412 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 211 211 208 
2021.03.24 11:22:48.376421 [ 1092 ] {} <Trace> RaftInstance: local log idx 211, target_commit_idx 211, quick_commit_index_ 211, state_->get_commit_idx() 211
2021.03.24 11:22:48.377958 [ 1095 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.378002 [ 1095 ] {} <Debug> RaftInstance: append at log_idx 212
2021.03.24 11:22:48.378032 [ 1095 ] {} <Debug> RaftInstance: commit_ret_cv 212 0x7faa1f070f20 sleep
2021.03.24 11:22:48.378061 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.378081 [ 1097 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.378092 [ 1097 ] {} <Trace> RaftInstance: last_log_idx: 211, starting_idx: 191, cur_nxt_idx: 213
2021.03.24 11:22:48.378105 [ 1097 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=211, LastLogTerm=1, EntriesLength=1, CommitIndex=211, Term=1, peer_last_sent_idx 212
2021.03.24 11:22:48.378114 [ 1097 ] {} <Debug> RaftInstance: idx: 212
2021.03.24 11:22:48.378125 [ 1097 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.378181 [ 1097 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.382704 [ 1094 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.382735 [ 1094 ] {} <Debug> RaftInstance: append at log_idx 213
2021.03.24 11:22:48.382769 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.382795 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.382809 [ 1094 ] {} <Debug> RaftInstance: commit_ret_cv 213 0x7faa0dbd7920 sleep
2021.03.24 11:22:48.383850 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 3, type append_entries_request, OK
2021.03.24 11:22:48.383888 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=212
2021.03.24 11:22:48.383897 [ 1092 ] {} <Trace> RaftInstance: src: 3, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.383908 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 212
2021.03.24 11:22:48.383915 [ 1092 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2021.03.24 11:22:48.383923 [ 1092 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 208, new matched idx: 211
2021.03.24 11:22:48.383933 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 213 211 211 
2021.03.24 11:22:48.383941 [ 1092 ] {} <Trace> RaftInstance: local log idx 213, target_commit_idx 211, quick_commit_index_ 211, state_->get_commit_idx() 211
2021.03.24 11:22:48.383950 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 3
2021.03.24 11:22:48.383979 [ 1092 ] {} <Trace> RaftInstance: send request to 3
2021.03.24 11:22:48.383988 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 211, starting_idx: 191, cur_nxt_idx: 214
2021.03.24 11:22:48.384000 [ 1092 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=211, LastLogTerm=1, EntriesLength=2, CommitIndex=211, Term=1, peer_last_sent_idx 209
2021.03.24 11:22:48.384007 [ 1092 ] {} <Debug> RaftInstance: idx range: 212-213
2021.03.24 11:22:48.384025 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 3, type append_entries_request
2021.03.24 11:22:48.384084 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.384838 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.384854 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=213
2021.03.24 11:22:48.384862 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.384871 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 213
2021.03.24 11:22:48.384878 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.384886 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 211, new matched idx: 212
2021.03.24 11:22:48.384894 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 213 212 211 
2021.03.24 11:22:48.384901 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 212
2021.03.24 11:22:48.384910 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.384917 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.384924 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 212, starting_idx: 191, cur_nxt_idx: 214
2021.03.24 11:22:48.384933 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=212, LastLogTerm=1, EntriesLength=1, CommitIndex=212, Term=1, peer_last_sent_idx 212
2021.03.24 11:22:48.384940 [ 1092 ] {} <Debug> RaftInstance: idx: 213
2021.03.24 11:22:48.384948 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.384980 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.384992 [ 1092 ] {} <Trace> RaftInstance: local log idx 213, target_commit_idx 212, quick_commit_index_ 212, state_->get_commit_idx() 211
2021.03.24 11:22:48.384999 [ 1092 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2021.03.24 11:22:48.385011 [ 1092 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 1
2021.03.24 11:22:48.385019 [ 1092 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.385044 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ wake up
2021.03.24 11:22:48.385057 [ 1096 ] {} <Debug> RaftInstance: commit upto 212, current idx 211
2021.03.24 11:22:48.385065 [ 1096 ] {} <Trace> RaftInstance: commit upto 212, current idx 212
2021.03.24 11:22:48.385151 [ 1096 ] {} <Debug> RaftInstance: notify cb 212 0x7faa1f070f20
2021.03.24 11:22:48.385185 [ 1096 ] {} <Debug> RaftInstance: DONE: commit upto 212, current idx 212
2021.03.24 11:22:48.385194 [ 1096 ] {} <Trace> RaftInstance: commit_cv_ sleep
2021.03.24 11:22:48.385222 [ 1095 ] {} <Debug> RaftInstance: remaining elems in waiting queue: 1
2021.03.24 11:22:48.385253 [ 1095 ] {} <Debug> RaftInstance: [OK] commit_ret_cv 212 wake up (7201 us), return value (nil)
2021.03.24 11:22:48.387858 [ 1108 ] {} <Debug> RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.03.24 11:22:48.387888 [ 1108 ] {} <Debug> RaftInstance: append at log_idx 214
2021.03.24 11:22:48.387911 [ 1108 ] {} <Debug> RaftInstance: commit_ret_cv 214 0x7faa0daae120 sleep
2021.03.24 11:22:48.387927 [ 1097 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.387949 [ 1097 ] {} <Debug> RaftInstance: Server 1 is busy, skip the request
2021.03.24 11:22:48.391790 [ 1092 ] {} <Trace> RaftInstance: resp of req 2 -> 1, type append_entries_request, OK
2021.03.24 11:22:48.391862 [ 1092 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 1 with Result=1, Term=1, NextIndex=214
2021.03.24 11:22:48.391873 [ 1092 ] {} <Trace> RaftInstance: src: 1, dst: 2, resp->get_term(): 1
2021.03.24 11:22:48.391884 [ 1092 ] {} <Trace> RaftInstance: handle append entries resp (from 1), resp.get_next_idx(): 214
2021.03.24 11:22:48.391892 [ 1092 ] {} <Trace> RaftInstance: peer 1 batch size hint: 0 bytes
2021.03.24 11:22:48.391900 [ 1092 ] {} <Trace> RaftInstance: peer 1, prev matched idx: 212, new matched idx: 213
2021.03.24 11:22:48.391911 [ 1092 ] {} <Trace> RaftInstance: quorum idx 1, 214 213 211 
2021.03.24 11:22:48.391919 [ 1092 ] {} <Debug> RaftInstance: trigger commit upto 213
2021.03.24 11:22:48.391928 [ 1092 ] {} <Debug> RaftInstance: Server 3 is busy, skip the request
2021.03.24 11:22:48.391936 [ 1092 ] {} <Trace> RaftInstance: send request to 1
2021.03.24 11:22:48.391943 [ 1092 ] {} <Trace> RaftInstance: last_log_idx: 213, starting_idx: 191, cur_nxt_idx: 215
2021.03.24 11:22:48.391955 [ 1092 ] {} <Debug> RaftInstance: append_entries for 1 with LastLogIndex=213, LastLogTerm=1, EntriesLength=1, CommitIndex=213, Term=1, peer_last_sent_idx 213
2021.03.24 11:22:48.391962 [ 1092 ] {} <Debug> RaftInstance: idx: 214
2021.03.24 11:22:48.391971 [ 1092 ] {} <Trace> RaftInstance: send req 2 -> 1, type append_entries_request
2021.03.24 11:22:48.392049 [ 1092 ] {} <Trace> RaftInstance: sent
2021.03.24 11:22:48.392067 [ 1092 ] {} <Trace> RaftInstance: local log idx 214, target_commit_idx 213, quick_commit_index_ 213, state_->get_commit_idx() 212
2021.03.24 11:22:48.3