Skip to content

Instantly share code, notes, and snippets.

@derekperkins
Created October 24, 2018 04:49
Show Gist options
  • Save derekperkins/877f23c7e8bedaab009c332275b7402e to your computer and use it in GitHub Desktop.
Save derekperkins/877f23c7e8bedaab009c332275b7402e to your computer and use it in GitHub Desktop.
Vitess can't restore from a backup
2018-10-24T04:35:51.582575Z 11 [System] [MY-013172] [Server] Received SHUTDOWN from user vt_dba. Shutting down mysqld (Version: 8.0.12).
2018-10-24T04:35:53.958610Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 10 user: 'vt_dba'.
2018-10-24T04:35:55.453253Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.12) MySQL Community Server - GPL.
2018-10-24T04:35:55.481609Z mysqld_safe mysqld from pid file /vtdataroot/tabletdata/mysql.pid ended
2018-10-24T04:36:39.551390Z mysqld_safe Logging to '/vtdataroot/tabletdata/error.log'.
2018-10-24T04:36:39.570842Z mysqld_safe Starting mysqld daemon with databases from /vtdataroot/tabletdata/data
mysqld: Can't change dir to '/vtdataroot/tabletdata/data/' (OS errno 2 - No such file or directory)
2018-10-24T04:36:39.869371Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2018-10-24T04:36:39.869500Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2018-10-24T04:36:39.869545Z 0 [Warning] [MY-010915] [Server] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2018-10-24T04:36:39.869604Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.12) starting as process 2826
2018-10-24T04:36:39.872593Z 0 [Warning] [MY-010091] [Server] Can't create test file /vtdataroot/tabletdata/data/test2-unsharded2-0-replica-1.lower-test
2018-10-24T04:36:39.872607Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /vtdataroot/tabletdata/data/ is case insensitive
2018-10-24T04:36:39.872661Z 0 [ERROR] [MY-010172] [Server] failed to set datadir to /vtdataroot/tabletdata/data/
2018-10-24T04:36:39.872694Z 0 [ERROR] [MY-010119] [Server] Aborting
2018-10-24T04:36:39.872951Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.12) MySQL Community Server - GPL.
I1024 04:41:59.857255 1 mysqlctld.go:91] mycnf file (/vtdataroot/tabletdata/my.cnf) already exists, starting without init
I1024 04:41:59.857599 1 dbconfigs.go:271] DBConfigs: {
"Conn": {
"host": "",
"port": 0,
"uname": "vt_dba",
"pass": "****",
"dbname": "",
"unix_socket": "/vtdataroot/tabletdata/mysql.sock",
"charset": "utf8",
"flags": 0,
"ssl_ca": "",
"ssl_ca_path": "",
"ssl_cert": "",
"ssl_key": "",
"server_name": "",
"DeprecatedDBName": ""
},
"Users": {
"dba": "vt_dba"
}
}
I1024 04:41:59.857736 1 mysqld.go:676] Checking for updates to my.cnf
I1024 04:41:59.857823 1 mysqld.go:600] make_mycnf hook doesn't exist, reading template files
I1024 04:41:59.858296 1 mysqld.go:702] No changes to my.cnf. Continuing.
I1024 04:41:59.858331 1 mysqld.go:205] Mysqld.Start(1540356119): No mysqld_start hook, running mysqld_safe directly
I1024 04:41:59.858368 1 mysqld.go:229] Mysqld.Start(1540356119) &exec.Cmd{Path:"/usr/bin/mysqld_safe", Args:[]string{"/usr/bin/mysqld_safe", "--defaults-file=/vtdataroot/tabletdata/my.cnf"}, Env:[]string{"LD_LIBRARY_PATH=/usr/lib/mysql"}, Dir:"/usr", Stdin:io.Reader(nil), Stdout:io.Writer(nil), Stderr:io.Writer(nil), ExtraFiles:[]*os.File(nil), SysProcAttr:(*syscall.SysProcAttr)(nil), Process:(*os.Process)(nil), ProcessState:(*os.ProcessState)(nil), ctx:context.Context(nil), lookPathErr:error(nil), finished:false, childFiles:[]*os.File(nil), closeAfterStart:[]io.Closer(nil), closeAfterWait:[]io.Closer(nil), goroutine:[]func() error(nil), errch:(chan error)(nil), waitDone:(chan struct {})(nil)}
I1024 04:41:59.858693 1 mysqld.go:296] Waiting for mysqld socket file (/vtdataroot/tabletdata/mysql.sock) to be ready...
I1024 04:42:00.437817 1 mysqld.go:247] Mysqld.Start(1540356119) stdout: 2018-10-24T04:42:00.437365Z mysqld_safe Logging to '/vtdataroot/tabletdata/error.log'.
I1024 04:42:00.460237 1 mysqld.go:247] Mysqld.Start(1540356119) stdout: 2018-10-24T04:42:00.459847Z mysqld_safe Starting mysqld daemon with databases from /vtdataroot/tabletdata/data
I1024 04:42:00.767336 1 mysqld.go:247] Mysqld.Start(1540356119) stdout: 2018-10-24T04:42:00.766893Z mysqld_safe mysqld from pid file /vtdataroot/tabletdata/mysql.pid ended
I1024 04:42:00.768278 1 mysqld.go:260] Mysqld.Start(1540356119) exit: <nil>
I1024 04:34:13.628076 1 grpc_server.go:196] Listening for gRPC calls on port 16002
I1024 04:34:13.628179 1 unix_socket.go:36] Not listening on socket file
E1024 04:34:13.645538 1 syslogger.go:84] no connection, dropping syslog event: &events.TabletChange{Tablet:topodata.Tablet{Alias:(*topodata.TabletAlias)(0xc420574100), Hostname:"test2-unsharded2-0-replica-1.vttablet", PortMap:map[string]int32{"grpc":16002, "vt":15002}, Keyspace:"unsharded2", Shard:"0", KeyRange:(*topodata.KeyRange)(nil), Type:7, DbNameOverride:"", Tags:map[string]string(nil), MysqlHostname:"test2-unsharded2-0-replica-1.vttablet", MysqlPort:0, XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:129}, Status:"updated"}
I1024 04:34:13.645607 1 state_change.go:128] Executing post-action state refresh: restore from backup
W1024 04:34:13.650776 1 action_agent.go:715] Cannot get current mysql port, not checking it (will retry at healthcheck interval): net.Dial(/vtdataroot/tabletdata/mysql.sock) to local server failed: dial unix /vtdataroot/tabletdata/mysql.sock: connect: no such file or directory (errno 2002) (sqlstate HY000)
I1024 04:34:13.650808 1 state_change.go:165] Running tablet callback because: restore from backup
I1024 04:34:13.650823 1 state_change.go:279] Disabling query service on type change, reason: not a serving tablet type(RESTORE)
I1024 04:34:13.650836 1 state_change.go:154] Done with post-action state refresh
I1024 04:34:13.650861 1 mysqld.go:296] Waiting for mysqld socket file (/vtdataroot/tabletdata/mysql.sock) to be ready...
I1024 04:34:13.663709 1 rebuild_keyspace.go:71] rebuildKeyspace unsharded2
I1024 04:34:13.667123 1 rebuild_keyspace.go:143] updating keyspace serving graph in cell test2 for unsharded2
I1024 04:34:13.674814 1 locks.go:249] Unlocking keyspace unsharded2 for successful action RebuildKeyspace
I1024 04:35:50.673002 1 backup.go:757] Restore: checking no existing data is present
I1024 04:35:50.701960 1 backup.go:772] Restore: looking for a suitable backup to restore
I1024 04:35:51.569496 1 backup.go:811] Restore: found backup unsharded2/0 2018-10-23.213816.test2-0094246201 to restore with 4 files
I1024 04:35:51.569519 1 backup.go:824] Restore: shutdown mysqld
I1024 04:35:51.569525 1 mysqld.go:329] Mysqld.Shutdown
I1024 04:35:51.569529 1 mysqld.go:333] executing Mysqld.Shutdown() remotely via mysqlctld server: /vtdataroot/mysqlctl.sock
I1024 04:35:55.584636 1 backup.go:830] Restore: deleting existing files
I1024 04:35:55.584770 1 backup.go:728] Restore: removing files in InnodbLogGroupHomeDir (/vtdataroot/tabletdata/innodb/logs)
I1024 04:35:56.551263 1 backup.go:710] Restore: removing files in RelayLogPath.* (/vtdataroot/tabletdata/relay-logs/vt-0094246201-relay-bin.*)
I1024 04:35:56.551405 1 backup.go:725] Restore: skipping removal of nonexistent RelayLogIndexPath (/vtdataroot/tabletdata/relay-logs/vt-0094246201-relay-bin.index)
I1024 04:35:56.551414 1 backup.go:725] Restore: skipping removal of nonexistent RelayLogInfoPath (/vtdataroot/tabletdata/relay-logs/relay-log.info)
I1024 04:35:56.551420 1 backup.go:710] Restore: removing files in BinLogPath.* (/vtdataroot/tabletdata/bin-logs/vt-0094246201-bin.*)
I1024 04:35:56.551503 1 backup.go:728] Restore: removing files in DataDir (/vtdataroot/tabletdata/data)
I1024 04:35:56.553675 1 backup.go:728] Restore: removing files in InnodbDataHomeDir (/vtdataroot/tabletdata/innodb/data)
I1024 04:35:56.553858 1 backup.go:835] Restore: reinit config file
I1024 04:35:56.553872 1 mysqld.go:725] Mysqld.ReinitConfig
I1024 04:35:56.553876 1 mysqld.go:729] executing Mysqld.ReinitConfig() remotely via mysqlctld server: /vtdataroot/mysqlctl.sock
I1024 04:35:56.555423 1 backup.go:841] Restore: copying all files
I1024 04:36:38.881763 1 backup.go:853] Restore: starting mysqld for mysql_upgrade
I1024 04:36:38.881787 1 mysqld.go:177] executing Mysqld.Start() remotely via mysqlctld server: /vtdataroot/mysqlctl.sock
W1024 04:36:39.925487 1 clientconn.go:1166] grpc: addrConn.createTransport failed to connect to {/vtdataroot/mysqlctl.sock 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial unix /vtdataroot/mysqlctl.sock: connect: connection refused". Reconnecting...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment