mysql01

MySQL thread id 24117450, OS thread handle 0x7fa39bf7d700, query id 2292559679 mysql03-int' 10.10.5.103 jira wsrep in pre-commit stage
*** WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1128591 page no 3 n bits 96 index `PRIMARY` of table `jira`.`AO_E8B6CC_MESSAGE` trx id 77362570549 lock_mode X locks rec but not gap
2014-07-12 07:40:30 14371 [Note] WSREP: cluster conflict due to high priority abort for threads:
2014-07-12 07:40:30 14371 [Note] WSREP: Winning thread:
   THD: 58, mode: applier, state: executing, conflict: no conflict, seqno: 115701758
   SQL: (null)
2014-07-12 07:40:30 14371 [Note] WSREP: Victim thread:
   THD: 24117450, mode: local, state: committing, conflict: no conflict, seqno: -1
   SQL: DELETE FROM AO_E8B6CC_MESSAGE WHERE ID IN (32806)
2014-07-12 07:40:30 14371 [Note] WSREP: declaring f2b4f82a-f7db-11e3-8770-27cfa45294b0 stable
2014-07-12 07:40:30 14371 [Note] WSREP: (ca04d333-fa74-11e3-94e2-7f5202c0e10c, 'tcp://0.0.0.0:4567') address 'tcp://10.10.5.101:4567' pointing to uuid ca04d333-fa74-11e3-94e2-7f5202c0e10c is blacklisted, skipping
2014-07-12 07:40:30 14371 [Note] WSREP: Node ca04d333-fa74-11e3-94e2-7f5202c0e10c state prim
2014-07-12 07:40:30 14371 [Note] WSREP: (ca04d333-fa74-11e3-94e2-7f5202c0e10c, 'tcp://0.0.0.0:4567') address 'tcp://10.10.5.101:4567' pointing to uuid ca04d333-fa74-11e3-94e2-7f5202c0e10c is blacklisted, skipping
2014-07-12 07:40:30 14371 [Note] WSREP: (ca04d333-fa74-11e3-94e2-7f5202c0e10c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.5.102:4567
2014-07-12 07:40:30 14371 [Note] WSREP: view(view_id(PRIM,ca04d333-fa74-11e3-94e2-7f5202c0e10c,16) memb {
        ca04d333-fa74-11e3-94e2-7f5202c0e10c,1
        f2b4f82a-f7db-11e3-8770-27cfa45294b0,1
} joined {
} left {
} partitioned {
        a50756ff-fa5e-11e3-9768-b3b1383329fb,1
})
2014-07-12 07:40:30 14371 [Note] WSREP: forgetting a50756ff-fa5e-11e3-9768-b3b1383329fb (tcp://10.10.5.102:4567)
2014-07-12 07:40:30 14371 [Note] WSREP: (ca04d333-fa74-11e3-94e2-7f5202c0e10c, 'tcp://0.0.0.0:4567') address 'tcp://10.10.5.101:4567' pointing to uuid ca04d333-fa74-11e3-94e2-7f5202c0e10c is blacklisted, skipping
2014-07-12 07:40:30 14371 [Note] WSREP: (ca04d333-fa74-11e3-94e2-7f5202c0e10c, 'tcp://0.0.0.0:4567') turning message relay requesting off
2014-07-12 07:40:30 14371 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2014-07-12 07:40:30 14371 [Note] WSREP: STATE_EXCHANGE: sent state UUID: b569b7fe-09c1-11e4-8e80-7f270c3e0455
2014-07-12 07:40:30 14371 [Note] WSREP: STATE EXCHANGE: sent state msg: b569b7fe-09c1-11e4-8e80-7f270c3e0455
2014-07-12 07:40:30 14371 [Note] WSREP: STATE EXCHANGE: got state msg: b569b7fe-09c1-11e4-8e80-7f270c3e0455 from 0 (mysql01')
2014-07-12 07:40:30 14371 [Note] WSREP: STATE EXCHANGE: got state msg: b569b7fe-09c1-11e4-8e80-7f270c3e0455 from 1 (mysql03')
2014-07-12 07:40:30 14371 [Note] WSREP: Quorum results:
        version    = 3,
        component  = PRIMARY,
        conf_id    = 14,
        members    = 2/2 (joined/total),
        act_id     = 115701781,
        last_appl. = 115701744,
        protocols  = 0/5/2 (gcs/repl/appl),
        group UUID = 4ba9f64d-f7c6-11e3-8948-9f388938aeef
2014-07-12 07:40:30 14371 [Note] WSREP: Flow-control interval: [23, 23]
2014-07-12 07:40:30 14371 [ERROR] Slave SQL: Could not execute Delete_rows event on table jira.AO_E8B6CC_MESSAGE; Cannot delete or update a parent row: a foreign key constraint fails (`jira`.`AO_E8B6CC_MESSAGE_TAG`, CONSTRAINT `fk_ao_e8b6cc_message_tag_message_id` FOREIGN KEY (`MESSAGE_ID`) REFERENCES `AO_E8B6CC_MESSAGE` (`ID`)), Error_code: 1451; handler error HA_ERR_ROW_IS_REFERENCED; the event's master log FIRST, end_log_pos 541, Error_code: 1451
2014-07-12 07:40:30 14371 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 152, 115701779
2014-07-12 07:40:30 14371 [Warning] WSREP: Failed to apply app buffer: seqno: 115701779, status: 1
         at galera/src/trx_handle.cpp:apply():340
Retrying 2th time
[snip]

2014-07-12 07:40:30 14371 [Warning] WSREP: Failed to apply trx 115701779 4 times
2014-07-12 07:40:30 14371 [ERROR] WSREP: trx_replay failed for: 6, query: void
2014-07-12 07:40:30 14371 [ERROR] Aborting

2014-07-12 07:40:31 14371 [Note] WSREP: Node ca04d333-fa74-11e3-94e2-7f5202c0e10c state prim
2014-07-12 07:40:31 14371 [Note] WSREP: view(view_id(PRIM,ca04d333-fa74-11e3-94e2-7f5202c0e10c,17) memb {
        ca04d333-fa74-11e3-94e2-7f5202c0e10c,1
} joined {
} left {
} partitioned {
        f2b4f82a-f7db-11e3-8770-27cfa45294b0,1
})
2014-07-12 07:40:31 14371 [Note] WSREP: forgetting a50756ff-fa5e-11e3-9768-b3b1383329fb (tcp://10.10.5.102:4567)
2014-07-12 07:40:31 14371 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2014-07-12 07:40:31 14371 [Note] WSREP: (ca04d333-fa74-11e3-94e2-7f5202c0e10c, 'tcp://0.0.0.0:4567') address 'tcp://10.10.5.101:4567' pointing to uuid ca04d333-fa74-11e3-94e2-7f5202c0e10c is blacklisted, skipping
2014-07-12 07:40:31 14371 [Note] WSREP: forgetting f2b4f82a-f7db-11e3-8770-27cfa45294b0 (tcp://10.10.5.103:4567)
2014-07-12 07:40:31 14371 [Note] WSREP: STATE_EXCHANGE: sent state UUID: b603ff37-09c1-11e4-8277-520228ed5564
2014-07-12 07:40:31 14371 [Note] WSREP: STATE EXCHANGE: sent state msg: b603ff37-09c1-11e4-8277-520228ed5564
2014-07-12 07:40:31 14371 [Note] WSREP: STATE EXCHANGE: got state msg: b603ff37-09c1-11e4-8277-520228ed5564 from 0 (mysql01')
2014-07-12 07:40:31 14371 [Note] WSREP: Quorum results:
        version    = 3,
        component  = PRIMARY,
        conf_id    = 15,
        members    = 1/1 (joined/total),
        act_id     = 115701784,
        last_appl. = 115701744,
        protocols  = 0/5/2 (gcs/repl/appl),
        group UUID = 4ba9f64d-f7c6-11e3-8948-9f388938aeef
2014-07-12 07:40:31 14371 [Note] WSREP: Flow-control interval: [16, 16]
2014-07-12 07:40:32 14371 [Note] WSREP: Closing send monitor...
2014-07-12 07:40:32 14371 [Note] WSREP: cluster conflict due to certification failure for threads:
2014-07-12 07:40:32 14371 [Note] WSREP: cluster conflict due to certification failure for threads:
2014-07-12 07:40:32 14371 [Note] WSREP: cluster conflict due to certification failure for threads:
2014-07-12 07:40:32 14371 [Note] WSREP: cluster conflict due to certification failure for threads:
[snipped unrelated query]
2014-07-12 07:40:32 14371 [Note] WSREP: gcomm: terminating thread
2014-07-12 07:40:32 14371 [Note] WSREP: cluster conflict due to certification failure for threads:
[snipped unrelated query]
2014-07-12 07:40:32 14371 [Note] WSREP: gcomm: joining thread
2014-07-12 07:40:32 14371 [Note] WSREP: gcomm: closing backend
2014-07-12 07:40:32 14371 [Note] WSREP: view((empty))
2014-07-12 07:40:32 14371 [Note] WSREP: Received self-leave message.
2014-07-12 07:40:32 14371 [Note] WSREP: gcomm: closed
2014-07-12 07:40:32 14371 [Note] WSREP: cluster conflict due to certification failure for threads:
2014-07-12 07:40:32 14371 [Note] WSREP: Flow-control interval: [0, 0]
2014-07-12 07:40:32 14371 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2014-07-12 07:40:32 14371 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 115701786)
[snipped unrelated query]
2014-07-12 07:40:32 14371 [Note] WSREP: RECV thread exiting 0: Success
2014-07-12 07:40:32 14371 [Note] WSREP: recv_thread() joined.
2014-07-12 07:40:32 14371 [Note] WSREP: Closing replication queue.
2014-07-12 07:40:32 14371 [Note] WSREP: Closing slave action queue.
2014-07-12 07:40:32 14371 [Note] WSREP: Service disconnected.
[snipped unrelated query]
2014-07-12 07:40:32 14371 [Note] WSREP: rollbacker thread exiting
[snipped lots more (100+) cluster conflicts on unrelated tables, etc]

12:40:33 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=25165824
read_buffer_size=131072
max_used_connections=996
max_threads=100001
thread_count=214
connection_count=134
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 38485662 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fd6f8872c80
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa39b647d48 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x7fd5c20ffb7c]
/usr/sbin/mysqld(handle_fatal_signal+0x3cb)[0x7fd5c1e465fb]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fd5c02f1340]
/usr/sbin/mysqld(_Z14ip_to_hostnameP16sockaddr_storagePKcPPcPj+0x1cf)[0x7fd5c1d7daef]
/usr/sbin/mysqld(+0x4dc598)[0x7fd5c1ea2598]
/usr/sbin/mysqld(_Z16login_connectionP3THD+0x42)[0x7fd5c1ea3bd2]
/usr/sbin/mysqld(_Z25threadpool_add_connectionP3THD+0x168)[0x7fd5c1f79788]
/usr/sbin/mysqld(+0x5b526e)[0x7fd5c1f7b26e]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0x7fd5c2378a60]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fd5c02e9182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd5bf7f630d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 24119638
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
140712 07:40:42 mysqld_safe Number of processes running now: 0
140712 07:40:42 mysqld_safe WSREP: not restarting wsrep node automatically
140712 07:40:42 mysqld_safe mysqld from pid file /var/lib/mysql/mysql01'.pid ended