2016-12-02 shard3 crash log

mysqld.log

2016-12-02  8:40:33 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') connection to peer 4ceca44b with addr tcp://10.1.10.91:40047 timed out, no
 messages seen in PT3S
2016-12-02  8:40:33 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') connection to peer dee5e459 with addr tcp://10.1.8.195:40047 timed out, no
 messages seen in PT3S
2016-12-02  8:40:33 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') turning message relay requesting on, nonlive peers: tcp://10.1.10.91:40047
 tcp://10.1.8.195:40047 
2016-12-02  8:40:34 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') connection established to 4ceca44b tcp://10.1.10.91:40047
2016-12-02  8:40:34 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') reconnecting to dee5e459 (tcp://10.1.8.195:40047), attempt 0
2016-12-02  8:40:34 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') connection established to dee5e459 tcp://10.1.8.195:40047
2016-12-02  8:40:37 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') turning message relay requesting off
2016-12-02  8:45:33 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') connection to peer dee5e459 with addr tcp://10.1.8.195:40047 timed out, no
 messages seen in PT3S
2016-12-02  8:45:33 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') connection to peer 4ceca44b with addr tcp://10.1.10.91:40047 timed out, no
 messages seen in PT3S
2016-12-02  8:45:33 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') turning message relay requesting on, nonlive peers: tcp://10.1.10.91:40047
 tcp://10.1.8.195:40047 
2016-12-02  8:45:34 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') reconnecting to 4ceca44b (tcp://10.1.10.91:40047), attempt 0
2016-12-02  8:45:34 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') reconnecting to dee5e459 (tcp://10.1.8.195:40047), attempt 0
2016-12-02  8:45:35 139992479299328 [Note] WSREP: evs::proto(173c558a, OPERATIONAL, view_id(REG,173c558a,79)) suspecting node: 4ceca44b
2016-12-02  8:45:35 139992479299328 [Note] WSREP: evs::proto(173c558a, OPERATIONAL, view_id(REG,173c558a,79)) suspected node without join message, declaring i
nactive
2016-12-02  8:45:35 139992479299328 [Note] WSREP: evs::proto(173c558a, OPERATIONAL, view_id(REG,173c558a,79)) suspecting node: dee5e459
2016-12-02  8:45:35 139992479299328 [Note] WSREP: evs::proto(173c558a, OPERATIONAL, view_id(REG,173c558a,79)) suspected node without join message, declaring i
nactive
2016-12-02  8:45:36 139992479299328 [Note] WSREP: view(view_id(NON_PRIM,173c558a,79) memb {
        173c558a,0
} joined {
} left {
} partitioned {
        4ceca44b,0
        dee5e459,0
})
2016-12-02  8:45:36 139992479299328 [Note] WSREP: view(view_id(NON_PRIM,173c558a,80) memb {
        173c558a,0
} joined {
} left {
} partitioned {
        4ceca44b,0
        dee5e459,0
})
2016-12-02  8:45:36 139992470906624 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2016-12-02  8:45:36 139992433154816 [Warning] WSREP: Send action {(nil), 302, TORDERED} returned -107 (Transport endpoint is not connected)
2016-12-02  8:45:36 139992470906624 [Note] WSREP: Flow-control interval: [16, 16]
2016-12-02  8:45:36 139992470906624 [Note] WSREP: Received NON-PRIMARY.
2016-12-02  8:45:36 139992470906624 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 3755308)                                                            [3211/8494]
2016-12-02  8:45:36 139992711740160 [Note] WSREP: New cluster view: global state: f8a8c602-aeeb-11e6-8054-5f205afd57e7:3755308, view# -1: non-Primary, number 
of nodes: 1, my index: 0, protocol version 3
2016-12-02  8:45:36 139992470906624 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2016-12-02  8:45:36 139992470906624 [Note] WSREP: Flow-control interval: [16, 16]
2016-12-02  8:45:36 139992470906624 [Note] WSREP: Received NON-PRIMARY.
2016-12-02  8:45:36 139992711740160 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-12-02  8:45:36 139992711740160 [Note] WSREP: New cluster view: global state: f8a8c602-aeeb-11e6-8054-5f205afd57e7:3755308, view# -1: non-Primary, number 
of nodes: 1, my index: 0, protocol version 3
2016-12-02  8:45:36 139992711740160 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-12-02  8:45:36 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') connection established to dee5e459 tcp://10.1.8.195:40047
2016-12-02  8:45:37 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') connection established to 4ceca44b tcp://10.1.10.91:40047
2016-12-02  8:45:39 139992479299328 [Note] WSREP: declaring 4ceca44b at tcp://10.1.10.91:40047 stable
2016-12-02  8:45:39 139992479299328 [Note] WSREP: declaring dee5e459 at tcp://10.1.8.195:40047 stable
2016-12-02  8:45:39 139992479299328 [Note] WSREP: Node 4ceca44b state prim
2016-12-02  8:45:39 139992479299328 [Note] WSREP: view(view_id(PRIM,173c558a,81) memb {
        173c558a,0
        4ceca44b,0
        dee5e459,0
} joined {
} left {
} partitioned {
})
2016-12-02  8:45:39 139992479299328 [Note] WSREP: save pc into disk
2016-12-02  8:45:39 139992470906624 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
2016-12-02  8:45:39 139992470906624 [Note] WSREP: STATE_EXCHANGE: sent state UUID: b3818398-b86b-11e6-b626-6ac3566dda94
2016-12-02  8:45:39 139992470906624 [Note] WSREP: STATE EXCHANGE: sent state msg: b3818398-b86b-11e6-b626-6ac3566dda94
2016-12-02  8:45:39 139992470906624 [Note] WSREP: STATE EXCHANGE: got state msg: b3818398-b86b-11e6-b626-6ac3566dda94 from 0 (5c4edde39751)
2016-12-02  8:45:39 139992470906624 [Note] WSREP: STATE EXCHANGE: got state msg: b3818398-b86b-11e6-b626-6ac3566dda94 from 1 (342952a51790)
2016-12-02  8:45:39 139992470906624 [Note] WSREP: STATE EXCHANGE: got state msg: b3818398-b86b-11e6-b626-6ac3566dda94 from 2 (a7277b6fbf77)
2016-12-02  8:45:39 139992470906624 [Note] WSREP: Quorum results:
        version    = 4,
        component  = PRIMARY,
        conf_id    = 54,
        members    = 2/3 (joined/total),
        act_id     = 3755311,
        last_appl. = 3755245,
        protocols  = 0/7/3 (gcs/repl/appl),
        group UUID = f8a8c602-aeeb-11e6-8054-5f205afd57e7
2016-12-02  8:45:39 139992470906624 [Note] WSREP: Flow-control interval: [28, 28]
2016-12-02  8:45:39 139992470906624 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 3755311)
2016-12-02  8:45:39 139992711740160 [Note] WSREP: State transfer required: 
        Group state: f8a8c602-aeeb-11e6-8054-5f205afd57e7:3755311
        Local state: f8a8c602-aeeb-11e6-8054-5f205afd57e7:3755308
2016-12-02  8:45:39 139992711740160 [Note] WSREP: New cluster view: global state: f8a8c602-aeeb-11e6-8054-5f205afd57e7:3755311, view# 55: Primary, number of n
odes: 3, my index: 0, protocol version 3
2016-12-02  8:45:39 139992711740160 [Warning] WSREP: Gap in state sequence. Need state transfer.
2016-12-02  8:45:39 139992711740160 [Note] WSREP: You have configured 'xtrabackup-v2' state snapshot transfer method which cannot be performed on a running se
rver. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server.
2016-12-02  8:45:39 139992711740160 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-12-02  8:45:39 139992711740160 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-12-02  8:45:39 139992533407488 [Note] WSREP: Service thread queue flushed.
2016-12-02  8:45:39 139992711740160 [Note] WSREP: Assign initial position for certification: 3755311, protocol version: 3
2016-12-02  8:45:39 139992533407488 [Note] WSREP: Service thread queue flushed.
2016-12-02  8:45:39 139992711740160 [Note] WSREP: IST receiver addr using tcp://10.1.6.242:40048
2016-12-02  8:45:39 139992711740160 [Warning] WSREP: Failed to prepare for incremental state transfer: Failed to open IST listener at tcp://10.1.6.242:40048',
 asio error 'Already open.': 1 (Operation not permitted)
         at galera/src/ist.cpp:prepare():325. IST will be unavailable.
2016-12-02  8:45:39 139992470906624 [Note] WSREP: Member 0.0 (5c4edde39751) requested state transfer from '*any*'. Selected 1.0 (342952a51790)(SYNCED) as dono
r.
2016-12-02  8:45:39 139992470906624 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 3755311)
2016-12-02  8:45:39 139992711740160 [Note] WSREP: Requesting state transfer: success, donor: 1
2016-12-02  8:45:39 139992711740160 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): full reset
2016-12-02  8:45:39 139992470906624 [Note] WSREP: 0.0 (5c4edde39751): State transfer from 1.0 (342952a51790) complete.
2016-12-02  8:45:39 139992470906624 [Note] WSREP: Shifting JOINER -> JOINED (TO: 3755311)
2016-12-02  8:45:39 139992470906624 [Warning] WSREP: 1.0 (342952a51790): State transfer to 0.0 (5c4edde39751) failed: -125 (Operation canceled)
2016-12-02  8:45:39 139992470906624 [Note] WSREP: Member 0.0 (5c4edde39751) synced with group.
2016-12-02  8:45:39 139992470906624 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 3755311)
2016-12-02  8:45:39 139992470906624 [Note] WSREP: Member 1.0 (342952a51790) synced with group.
2016-12-02  8:45:40 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') turning message relay requesting off
2016-12-02  8:46:32 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') turning message relay requesting on, nonlive peers: tcp://10.1.8.195:40047
 
2016-12-02  8:46:33 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') connection established to dee5e459 tcp://10.1.8.195:40047
2016-12-02  8:46:36 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') turning message relay requesting off
2016-12-02  8:48:33 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') turning message relay requesting on, nonlive peers: tcp://10.1.10.91:40047
 
2016-12-02  8:48:33 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') connection established to 4ceca44b tcp://10.1.10.91:40047
2016-12-02  8:48:37 139992479299328 [Note] WSREP: (173c558a, 'tcp://0.0.0.0:40047') turning message relay requesting off