Skip to content

Instantly share code, notes, and snippets.

@chrisLeeTW

chrisLeeTW/gist:c27ea060f82b2c07df74

Created September 12, 2015 07:55
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save chrisLeeTW/c27ea060f82b2c07df74 to your computer and use it in GitHub Desktop.
Save chrisLeeTW/c27ea060f82b2c07df74 to your computer and use it in GitHub Desktop.
Download ZIP
Percona Xtradb Cluster IST Received Failed.
Raw
gistfile1.txt
There is partial log that IST Received Failed
[CODE]
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:07:20 16752 [Note] WSREP: gcomm/src/gmcast.cpp:check_liveness():1128: (94043ae8, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.88.3.20:4567 tcp://10.88.3.21:4567 tcp://10.88.3.22:4567 tcp://10.88.3.23:4567 tcp://10.88.3.24:4567
2015-09-12 15:07:21 16752 [Note] WSREP: gcomm/src/gmcast.cpp:reconnect():991: (94043ae8, 'tcp://0.0.0.0:4567') reconnecting to 90978887 (tcp://10.88.3.21:4567), attempt 0
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/gmcast.cpp:reconnect():991: (94043ae8, 'tcp://0.0.0.0:4567') reconnecting to 6ca07d8f (tcp://10.88.3.20:4567), attempt 0
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/gmcast.cpp:reconnect():991: (94043ae8, 'tcp://0.0.0.0:4567') reconnecting to de5f6602 (tcp://10.88.3.22:4567), attempt 0
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/gmcast.cpp:reconnect():991: (94043ae8, 'tcp://0.0.0.0:4567') reconnecting to 928c0cfb (tcp://10.88.3.23:4567), attempt 0
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/gmcast.cpp:reconnect():991: (94043ae8, 'tcp://0.0.0.0:4567') reconnecting to 99b311c5 (tcp://10.88.3.24:4567), attempt 0
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():862: evs::proto(94043ae8, OPERATIONAL, view_id(REG,6ca07d8f,305)) suspecting node: 6ca07d8f
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():873: evs::proto(94043ae8, OPERATIONAL, view_id(REG,6ca07d8f,305)) suspected node without join message, declaring inactive
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():862: evs::proto(94043ae8, OPERATIONAL, view_id(REG,6ca07d8f,305)) suspecting node: 90978887
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():873: evs::proto(94043ae8, OPERATIONAL, view_id(REG,6ca07d8f,305)) suspected node without join message, declaring inactive
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():862: evs::proto(94043ae8, OPERATIONAL, view_id(REG,6ca07d8f,305)) suspecting node: 928c0cfb
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():873: evs::proto(94043ae8, OPERATIONAL, view_id(REG,6ca07d8f,305)) suspected node without join message, declaring inactive
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():862: evs::proto(94043ae8, OPERATIONAL, view_id(REG,6ca07d8f,305)) suspecting node: 99b311c5
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():873: evs::proto(94043ae8, OPERATIONAL, view_id(REG,6ca07d8f,305)) suspected node without join message, declaring inactive
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():862: evs::proto(94043ae8, OPERATIONAL, view_id(REG,6ca07d8f,305)) suspecting node: de5f6602
2015-09-12 15:07:22 16752 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():873: evs::proto(94043ae8, OPERATIONAL, view_id(REG,6ca07d8f,305)) suspected node without join message, declaring inactive
2015-09-12 15:07:23 16752 [Note] WSREP: gcs/src/gcs_group.cpp:gcs_group_state_t gcs_group_handle_comp_msg(gcs_group_t*, const gcs_comp_msg_t*)():398: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2015-09-12 15:07:23 16752 [Note] WSREP: gcs/src/gcs.cpp:void _set_fc_limits(gcs_conn_t*)():752: Flow-control interval: [16, 16]
2015-09-12 15:07:23 16752 [Note] WSREP: gcs/src/gcs.cpp:void gcs_handle_act_conf(gcs_conn_t*, const void*)():856: Received NON-PRIMARY.
2015-09-12 15:07:23 16752 [Note] WSREP: gcs/src/gcs.cpp:bool gcs_shift_state(gcs_conn_t*, gcs_conn_state_t)():558: Shifting SYNCED -> OPEN (TO: 98971078)
2015-09-12 15:07:23 16752 [Note] WSREP: New cluster view: global state: 4b463d90-e491-11e3-b958-0e9322f22cf6:98971078, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
2015-09-12 15:07:23 16752 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-09-12 15:07:23 16752 [Note] WSREP: gcs/src/gcs_group.cpp:gcs_group_state_t gcs_group_handle_comp_msg(gcs_group_t*, const gcs_comp_msg_t*)():398: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2015-09-12 15:07:23 16752 [Note] WSREP: gcs/src/gcs.cpp:void _set_fc_limits(gcs_conn_t*)():752: Flow-control interval: [16, 16]
2015-09-12 15:07:23 16752 [Note] WSREP: gcs/src/gcs.cpp:void gcs_handle_act_conf(gcs_conn_t*, const void*)():856: Received NON-PRIMARY.
2015-09-12 15:07:23 16752 [Note] WSREP: New cluster view: global state: 4b463d90-e491-11e3-b958-0e9322f22cf6:98971078, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
2015-09-12 15:07:23 16752 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-09-12 15:08:01 16752 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():842: (94043ae8, 'tcp://0.0.0.0:4567') address 'tcp://10.88.3.25:4567' pointing to uuid 94043ae8 is blacklisted, skipping
2015-09-12 15:08:01 16752 [Note] WSREP: gcomm/src/gmcast.cpp:handle_stable_view():1575: declaring 6ca07d8f at tcp://10.88.3.20:4567 stable
2015-09-12 15:08:01 16752 [Note] WSREP: gcomm/src/gmcast.cpp:handle_stable_view():1575: declaring 90978887 at tcp://10.88.3.21:4567 stable
2015-09-12 15:08:01 16752 [Note] WSREP: gcomm/src/gmcast.cpp:handle_stable_view():1575: declaring 928c0cfb at tcp://10.88.3.23:4567 stable
2015-09-12 15:08:01 16752 [Note] WSREP: gcomm/src/gmcast.cpp:handle_stable_view():1575: declaring 99b311c5 at tcp://10.88.3.24:4567 stable
2015-09-12 15:08:01 16752 [Note] WSREP: gcomm/src/gmcast.cpp:handle_stable_view():1575: declaring de5f6602 at tcp://10.88.3.22:4567 stable
2015-09-12 15:08:01 16752 [Note] WSREP: gcomm/src/pc_proto.cpp:is_prim():778: Node 6ca07d8f state prim
2015-09-12 15:08:01 16752 [Note] WSREP: gcomm/src/pc.cpp:handle_up():29: save pc into disk
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:gcs_group_state_t gcs_group_handle_comp_msg(gcs_group_t*, const gcs_comp_msg_t*)():398: New COMPONENT: primary = yes, bootstrap = no, my_idx = 3, memb_num = 6
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_core.cpp:ssize_t core_handle_comp_msg(gcs_core_t*, gcs_recv_msg*, gcs_act*)():775: STATE EXCHANGE: Waiting for state UUID.
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_core.cpp:ssize_t core_handle_uuid_msg(gcs_core_t*, gcs_recv_msg_t*)():860: STATE EXCHANGE: sent state msg: 016301ce-591d-11e5-bf69-7ea72e99f7b8
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:gcs_group_state_t gcs_group_handle_state_msg(gcs_group_t*, const gcs_recv_msg_t*)():571: STATE EXCHANGE: got state msg: 016301ce-591d-11e5-bf69-7ea72e99f7b8 from 0 (mall-db-cluster-001.ak.hiiir)
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:void group_print_state_debug(gcs_state_msg_t*)():553:
Version : 3
Flags : 0x3
Protocols : 0 / 6 / 3
State : SYNCED
Prim state : SYNCED
Prim UUID : ea897772-591c-11e5-8f15-534ef7cbd2c0
Prim seqno : 303
First seqno : 94785548
Last seqno : 98971110
Prim JOINED : 5
State UUID : 016301ce-591d-11e5-bf69-7ea72e99f7b8
Group UUID : 4b463d90-e491-11e3-b958-0e9322f22cf6
Name : 'mall-db-cluster-001.ak.hiiir'
Incoming addr: '10.88.3.20:3306'
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:gcs_group_state_t gcs_group_handle_state_msg(gcs_group_t*, const gcs_recv_msg_t*)():571: STATE EXCHANGE: got state msg: 016301ce-591d-11e5-bf69-7ea72e99f7b8 from 1 (mall-db-cluster-002.ak.hiiir)
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:void group_print_state_debug(gcs_state_msg_t*)():553:
Version : 3
Flags : 0x2
Protocols : 0 / 6 / 3
State : SYNCED
Prim state : SYNCED
Prim UUID : ea897772-591c-11e5-8f15-534ef7cbd2c0
Prim seqno : 303
First seqno : 94785543
Last seqno : 98971110
Prim JOINED : 5
State UUID : 016301ce-591d-11e5-bf69-7ea72e99f7b8
Group UUID : 4b463d90-e491-11e3-b958-0e9322f22cf6
Name : 'mall-db-cluster-002.ak.hiiir'
Incoming addr: '10.88.3.21:3306'
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:gcs_group_state_t gcs_group_handle_state_msg(gcs_group_t*, const gcs_recv_msg_t*)():571: STATE EXCHANGE: got state msg: 016301ce-591d-11e5-bf69-7ea72e99f7b8 from 2 (mall-db-cluster-005.ak.hiiir)
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:void group_print_state_debug(gcs_state_msg_t*)():553:
Version : 3
Flags : 0x2
Protocols : 0 / 6 / 3
State : SYNCED
Prim state : SYNCED
Prim UUID : ea897772-591c-11e5-8f15-534ef7cbd2c0
Prim seqno : 303
First seqno : 96308810
Last seqno : 98971110
Prim JOINED : 5
State UUID : 016301ce-591d-11e5-bf69-7ea72e99f7b8
Group UUID : 4b463d90-e491-11e3-b958-0e9322f22cf6
Name : 'mall-db-cluster-005.ak.hiiir'
Incoming addr: '10.88.3.23:3306'
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:gcs_group_state_t gcs_group_handle_state_msg(gcs_group_t*, const gcs_recv_msg_t*)():571: STATE EXCHANGE: got state msg: 016301ce-591d-11e5-bf69-7ea72e99f7b8 from 3 (mall-db-cluster-006.ak.hiiir)
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:void group_print_state_debug(gcs_state_msg_t*)():553:
Version : 3
Flags : 0x2
Protocols : 0 / 6 / 3
State : NON-PRIMARY
Prim state : SYNCED
Prim UUID : 37ce84c3-590d-11e5-a4b5-9ba1a6f6aeea
Prim seqno : 302
First seqno : 98962840
Last seqno : 98971078
Prim JOINED : 6
State UUID : 016301ce-591d-11e5-bf69-7ea72e99f7b8
Group UUID : 4b463d90-e491-11e3-b958-0e9322f22cf6
Name : 'mall-db-cluster-006.ak.hiiir'
Incoming addr: '10.88.3.25:3306'
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:gcs_group_state_t gcs_group_handle_state_msg(gcs_group_t*, const gcs_recv_msg_t*)():571: STATE EXCHANGE: got state msg: 016301ce-591d-11e5-bf69-7ea72e99f7b8 from 4 (mall-db-cluster-004.ak.hiiir)
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:void group_print_state_debug(gcs_state_msg_t*)():553:
Version : 3
Flags : 0x2
Protocols : 0 / 6 / 3
State : SYNCED
Prim state : SYNCED
Prim UUID : ea897772-591c-11e5-8f15-534ef7cbd2c0
Prim seqno : 303
First seqno : 94882462
Last seqno : 98971110
Prim JOINED : 5
State UUID : 016301ce-591d-11e5-bf69-7ea72e99f7b8
Group UUID : 4b463d90-e491-11e3-b958-0e9322f22cf6
Name : 'mall-db-cluster-004.ak.hiiir'
Incoming addr: '10.88.3.24:3306'
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:gcs_group_state_t gcs_group_handle_state_msg(gcs_group_t*, const gcs_recv_msg_t*)():571: STATE EXCHANGE: got state msg: 016301ce-591d-11e5-bf69-7ea72e99f7b8 from 5 (mall-db-cluster-003.ak.hiiir)
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:void group_print_state_debug(gcs_state_msg_t*)():553:
Version : 3
Flags : 0x2
Protocols : 0 / 6 / 3
State : SYNCED
Prim state : SYNCED
Prim UUID : ea897772-591c-11e5-8f15-534ef7cbd2c0
Prim seqno : 303
First seqno : 94785543
Last seqno : 98971110
Prim JOINED : 5
State UUID : 016301ce-591d-11e5-bf69-7ea72e99f7b8
Group UUID : 4b463d90-e491-11e3-b958-0e9322f22cf6
Name : 'mall-db-cluster-003.ak.hiiir'
Incoming addr: '10.88.3.22:3306'
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:void group_post_state_exchange(gcs_group_t*)():357: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 303,
members = 5/6 (joined/total),
act_id = 98971110,
last_appl. = 98971029,
protocols = 0/6/3 (gcs/repl/appl),
group UUID = 4b463d90-e491-11e3-b958-0e9322f22cf6
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs.cpp:void _set_fc_limits(gcs_conn_t*)():752: Flow-control interval: [39, 39]
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs.cpp:bool gcs_shift_state(gcs_conn_t*, gcs_conn_state_t)():558: Shifting OPEN -> PRIMARY (TO: 98971110)
2015-09-12 15:08:01 16752 [Note] WSREP: galera/src/replicator_smm.cpp:process_conf_change():1363: State transfer required:
Group state: 4b463d90-e491-11e3-b958-0e9322f22cf6:98971110
Local state: 4b463d90-e491-11e3-b958-0e9322f22cf6:98971078
2015-09-12 15:08:01 16752 [Note] WSREP: New cluster view: global state: 4b463d90-e491-11e3-b958-0e9322f22cf6:98971110, view# 304: Primary, number of nodes: 6, my index: 3, protocol version 3
2015-09-12 15:08:01 16752 [Warning] WSREP: Gap in state sequence. Need state transfer.
2015-09-12 15:08:01 16752 [Note] WSREP: You have configured 'xtrabackup-v2' state snapshot transfer method which cannot be performed on a running server. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server.
2015-09-12 15:08:01 16752 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-09-12 15:08:01 16752 [Note] WSREP: galera/src/replicator_smm.cpp:establish_protocol_versions():1288: REPL Protocols: 6 (3, 2)
2015-09-12 15:08:01 16752 [Note] WSREP: galera/src/galera_service_thd.cpp:thd_func():39: Service thread queue flushed.
2015-09-12 15:08:01 16752 [Note] WSREP: galera/src/certification.cpp:assign_initial_position():897: Assign initial position for certification: 98971110, protocol version: 3
2015-09-12 15:08:01 16752 [Note] WSREP: galera/src/galera_service_thd.cpp:thd_func():39: Service thread queue flushed.
2015-09-12 15:08:01 16752 [Note] WSREP: galera/src/ist.cpp:prepare():367: Prepared IST receiver, listening at: tcp://10.88.3.25:4568
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_group.cpp:int group_select_donor(gcs_group_t*, int, int, const char*, const gu_uuid_t*, gcs_seqno_t, bool)():1232: Member 3.0 (mall-db-cluster-006.ak.hiiir) requested state transfer from '*any*'. Selected 2.0 (mall-db-cluster-005.ak.hiiir)(SYNCED) as donor.
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs.cpp:bool gcs_shift_state(gcs_conn_t*, gcs_conn_state_t)():558: Shifting PRIMARY -> JOINER (TO: 98971110)
2015-09-12 15:08:01 16752 [Note] WSREP: galera/src/replicator_str.cpp:send_state_request():613: Requesting state transfer: success, donor: 2
2015-09-12 15:08:01 16752 [Note] WSREP: gcache/src/gcache_rb_store.cpp:seqno_reset():389: GCache DEBUG: RingBuffer::seqno_reset(): full reset
2015-09-12 15:08:01 16752 [Note] WSREP: galera/src/replicator_str.cpp:request_state_transfer():721: Receiving IST: 32 writesets, seqnos 98971078-98971110
2015-09-12 15:08:01 16752 [Warning] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():716: 2.0 (mall-db-cluster-005.ak.hiiir): State transfer to 3.0 (mall-db-cluster-006.ak.hiiir) failed: -111 (Connection refused)
2015-09-12 15:08:01 16752 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():722: Will never receive state. Need to abort.
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_gcomm.cpp:close():265: gcomm: terminating thread
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_gcomm.cpp:close():267: gcomm: joining thread
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_gcomm.cpp:close():269: gcomm: closing backend
2015-09-12 15:08:01 16752 [Note] WSREP: gcs/src/gcs_gcomm.cpp:close():281: gcomm: closed
2015-09-12 15:08:01 16752 [Note] WSREP: galerautils/src/gu_abort.c:gu_abort():29: /usr/sbin/mysqld: Terminated.
150912 15:08:01 mysqld_safe Number of processes running now: 0
150912 15:08:01 mysqld_safe WSREP: not restarting wsrep node automatically
150912 15:08:01 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
[/CODE]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment