container2 (succeeded)
160413 8:41:19 [Note] WSREP: Member 0.0 (container1) synced with group.
WSREP_SST: [INFO] Index compaction detected (20160413 08:41:19.826)
WSREP_SST: [INFO] Rebuilding during prepare with 4 threads (20160413 08:41:19.832)
WSREP_SST: [INFO] Compressed qpress files found (20160413 08:41:19.835)
WSREP_SST: [INFO] Decompression with 8 threads (20160413 08:41:19.838)
WSREP_SST: [INFO] Evaluating find /var/lib/mysql//.sst -type f -name '*.qp' -printf '%p\n%h\n' | xargs -n 2 qpress -T8d (20160413 08:41:19.840)
WSREP_SST: [INFO] Removing qpress files after decompression (20160413 08:41:19.891)
WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20160413 08:41:19.900)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20160413 08:41:19.902)
rm: cannot remove ‘/var/lib/mysql//innobackup.prepare.log’: No such file or directory
rm: cannot remove ‘/var/lib/mysql//innobackup.move.log’: No such file or directory
WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20160413 08:41:32.444)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/mysql/my.cnf --defaults-group=mysqld --no-version-check --move-back --force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log (20160413 08:41:32.446)
WSREP_SST: [INFO] Move successful, removing /var/lib/mysql//.sst (20160413 08:41:32.637)
WSREP_SST: [INFO] Galera co-ords from recovery: 25c5daf8-0153-11e6-86a0-73a5fa32321c:7 (20160413 08:41:32.643)
WSREP_SST: [INFO] Total time on joiner: 0 seconds (20160413 08:41:32.645)
WSREP_SST: [INFO] Removing the sst_in_progress file (20160413 08:41:32.648)
160413 8:41:32 [Note] WSREP: SST complete, seqno: 7
container3 (failed because mysqld_safe kicked in and tried to start mysql separately which screwed the SST halfway through)
160413 8:44:04 [Note] WSREP: Member 0.0 (container1) synced with group.
WSREP_SST: [INFO] Index compaction detected (20160413 08:44:04.715)
WSREP_SST: [INFO] Rebuilding during prepare with 4 threads (20160413 08:44:04.722)
WSREP_SST: [INFO] Compressed qpress files found (20160413 08:44:04.725)
WSREP_SST: [INFO] Decompression with 8 threads (20160413 08:44:04.729)
WSREP_SST: [INFO] Evaluating find /var/lib/mysql//.sst -type f -name '*.qp' -printf '%p\n%h\n' | xargs -n 2 qpress -T8d (20160413 08:44:04.731)
WSREP_SST: [INFO] Removing qpress files after decompression (20160413 08:44:04.784)
WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20160413 08:44:04.794)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20160413 08:44:04.796)
>>>>>> 160413 08:44:16 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
>>>>>> 160413 08:44:16 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.LPJnpq' --pid-file='/var/lib/mysql/container3-recover.pid'**
>>>>>> 160413 8:44:16 [Note] /usr/sbin/mysqld (mysqld 10.0.24-MariaDB-1~trusty-wsrep) starting as process 6521 ...
rm: cannot remove ‘/var/lib/mysql//innobackup.prepare.log’: No such file or directory
rm: cannot remove ‘/var/lib/mysql//innobackup.move.log’: No such file or directory
WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20160413 08:44:24.659)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/mysql/my.cnf --defaults-group=mysqld --no-version-check --move-back --force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log (20160413 08:44:24.661)
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20160413 08:44:25.244)
160413 8:44:25 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.100.100.103' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '5563' --binlog '/var/lib/mysql/mariadb-bin' : 1 (Operation not permitted)
160413 8:44:25 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
160413 8:44:25 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
160413 8:44:25 [ERROR] Aborting
The reason this causes the SST to fail halfway through is that part of the SST is to move the synced databases from .sst into the main data dir (/var/lib/mysql) - this is what the --move-back flag is for. Since mysqld_safe appears to be starting a new mysql daemon, it does what it would normally do on startup and starts to create the necessary files. One of these is ibdata1. It looks like the move of the databases from .sst is dieing because it can't overwrite the ibdata1 (that mysqld_safe has just created) with the one from it's .sst dir. This is the relevant portion of the innobackup.move.log:
...
innobackupex: Moving '/var/lib/mysql/.sst/performance_schema/file_instances.frm' to '/var/lib/mysql/performance_schema/file_instances.frm'
innobackupex: Moving '/var/lib/mysql/.sst/performance_schema/users.frm' to '/var/lib/mysql/performance_schema/users.frm'
innobackupex: Moving '/var/lib/mysql/.sst/performance_schema/events_waits_summary_by_user_by_event_name.frm' to '/var/lib/mysql/performance_schema/events_waits_summary_by_user_by_event_name.frm'
innobackupex: Moving '/var/lib/mysql/.sst/performance_schema/setup_objects.frm' to '/var/lib/mysql/performance_schema/setup_objects.frm'
innobackupex: Moving '/var/lib/mysql/.sst/performance_schema/file_summary_by_instance.frm' to '/var/lib/mysql/performance_schema/file_summary_by_instance.frm'
innobackupex: Moving '/var/lib/mysql/.sst/performance_schema/events_statements_summary_by_user_by_event_name.frm' to '/var/lib/mysql/performance_schema/events_statements_summary_by_user_by_event_name.frm'
innobackupex: Moving '/var/lib/mysql/.sst/performance_schema/performance_timers.frm' to '/var/lib/mysql/performance_schema/performance_timers.frm'
innobackupex: Starting to move InnoDB system tablespace
innobackupex: in '/var/lib/mysql/.sst'
innobackupex: back to original InnoDB data directory '/var/lib/mysql'
innobackupex: got a fatal error with the following stacktrace: at /usr//bin/innobackupex line 2216.
main::die_if_exists('/var/lib/mysql/ibdata1') called at /usr//bin/innobackupex line 2550
main::copy_back(1) called at /usr//bin/innobackupex line 1578
innobackupex: Error: Cannot overwrite file: /var/lib/mysql/ibdata1 at /usr//bin/innobackupex line 2216.