Last active
August 16, 2017 16:43
-
-
Save masv/73ef9cc07cf98ccf8f0aa28225c55689 to your computer and use it in GitHub Desktop.
Problems while promoting a MySQL master with Orchestrator
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| orchestrator -c topology -i db1 | |
| # I would expect to be able to make db1 and db2 co-master, but this fails | |
| # with error 1776 (can not set binlog pos when auto-positioning is active) | |
| orchestrator -c make-co-master -i db2 -d db1 | |
| # Promoting db2 with graceful master takeover works, but requires some cleanup | |
| orchestrator -c graceful-master-takeover -i db1 | |
| orchestrator -c topology -i db1 | |
| orchestrator -c set-writeable -i db2 | |
| orchestrator -c start-slave -i db1 | |
| orchestrator -c detach-replica -i db2 # This also fails with error 1776 | |
| orchestrator -c reset-slave -i db2 # This makes db2 the single master | |
| orchestrator -c topology -i db1 | |
| # Moving replicas around for another attempt | |
| orchestrator -c move-replicas-gtid -i db2 -d db3 | |
| orchestrator -c topology -i db1 | |
| # Fails with error 1776 | |
| orchestrator -c make-co-master -i db3 -d db2 | |
| # Fails with "too many replicas" | |
| orchestrator -c graceful-master-takeover -i db2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ubuntu@admin:~$ orchestrator -c topology -i db1 | |
| 2017-08-16 15:50:42 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:50:42 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:50:42 DEBUG Initializing orchestrator | |
| 2017-08-16 15:50:42 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:50:42 DEBUG instanceKey: db1:3306 | |
| 2017-08-16 15:50:42 DEBUG instanceKey: db2:3306 | |
| 2017-08-16 15:50:42 DEBUG instanceKey: db3:3306 | |
| 2017-08-16 15:50:42 DEBUG instanceKey: db4:3306 | |
| db1:3306 [0s,ok,5.6.36-82.1-log,rw,ROW,>>] | |
| + db2:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| + db3:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| + db4:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| ubuntu@admin:~$ orchestrator -c make-co-master -i db2 -d db1 | |
| 2017-08-16 15:50:57 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:50:57 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:50:57 DEBUG Initializing orchestrator | |
| 2017-08-16 15:50:57 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:50:57 DEBUG Will check whether db2:3306's master (db1:3306) can become its co-master | |
| 2017-08-16 15:50:57 INFO Will make db2:3306 co-master of db1:3306 | |
| 2017-08-16 15:50:57 INFO auditType:begin-maintenance instance:db2:3306 cluster:db1:3306 message:maintenanceToken: 1, owner: ubuntu, reason: make co-master of db1:3306 | |
| 2017-08-16 15:50:57 INFO auditType:begin-maintenance instance:db1:3306 cluster:db1:3306 message:maintenanceToken: 2, owner: ubuntu, reason: db2:3306 turns into co-master of this | |
| 2017-08-16 15:50:57 DEBUG ChangeMasterTo: will attempt changing master on db1:3306 to db2:3306, mysqld-bin.000001:665 | |
| 2017-08-16 15:50:57 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| 2017-08-16 15:50:57 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| 2017-08-16 15:50:57 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| 2017-08-16 15:50:57 INFO auditType:end-maintenance instance:db1:3306 cluster:db1:3306 message:maintenanceToken: 2 | |
| 2017-08-16 15:50:57 INFO auditType:end-maintenance instance:db2:3306 cluster:db1:3306 message:maintenanceToken: 1 | |
| 2017-08-16 15:50:57 FATAL Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| ubuntu@admin:~$ orchestrator -c graceful-master-takeover -i db1 | |
| 2017-08-16 15:51:11 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:51:11 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:51:11 DEBUG Initializing orchestrator | |
| 2017-08-16 15:51:11 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:51:11 INFO Will demote db1:3306 and promote db2:3306 instead | |
| 2017-08-16 15:51:11 INFO Stopped slave on db2:3306, Self:mysqld-bin.000001:665, Exec:mysqld-bin.000001:408 | |
| 2017-08-16 15:51:11 INFO Will set db1:3306 as read_only | |
| 2017-08-16 15:51:11 INFO instance db1:3306 read_only: true | |
| 2017-08-16 15:51:11 INFO auditType:read-only instance:db1:3306 cluster:db1:3306 message:set as true | |
| 2017-08-16 15:51:11 INFO Will advance db2:3306 to master coordinates mysqld-bin.000001:408 | |
| 2017-08-16 15:51:11 INFO Will start slave on db2:3306 until coordinates: mysqld-bin.000001:408 | |
| 2017-08-16 15:51:11 INFO Stopped slave on db2:3306, Self:mysqld-bin.000001:665, Exec:mysqld-bin.000001:408 | |
| 2017-08-16 15:51:11 INFO executeCheckAndRecoverFunction: proceeding with DeadMaster detection on db1:3306; isActionable?: true; skipProcesses: false | |
| 2017-08-16 15:51:11 INFO topology_recovery: detected DeadMaster failure on db1:3306 | |
| 2017-08-16 15:51:11 INFO topology_recovery: Running 1 OnFailureDetectionProcesses hooks | |
| 2017-08-16 15:51:11 INFO topology_recovery: Running OnFailureDetectionProcesses hook 1 of 1: echo 'Detected DeadMaster on db1:3306. Affected replicas: 1' >> /tmp/recovery.log | |
| 2017-08-16 15:51:11 INFO CommandRun(echo 'Detected DeadMaster on db1:3306. Affected replicas: 1' >> /tmp/recovery.log,[]) | |
| 2017-08-16 15:51:11 INFO CommandRun/running: bash /tmp/orchestrator-process-cmd-781658707 | |
| 2017-08-16 15:51:11 INFO CommandRun successful. exit status 0 | |
| 2017-08-16 15:51:11 INFO topology_recovery: Completed OnFailureDetectionProcesses hook 1 of 1 in 3.672934ms | |
| 2017-08-16 15:51:11 INFO Completed OnFailureDetectionProcesses hook 1 of 1 in 3.672934ms | |
| 2017-08-16 15:51:11 INFO topology_recovery: done running OnFailureDetectionProcesses hooks | |
| 2017-08-16 15:51:11 INFO executeCheckAndRecoverFunction: proceeding with DeadMaster recovery on db1:3306; isRecoverable?: true; skipProcesses: false | |
| 2017-08-16 15:51:11 INFO topology_recovery: will handle DeadMaster event on db1:3306 | |
| 2017-08-16 15:51:11 INFO auditType:recover-dead-master instance:db1:3306 cluster:db1:3306 message:problem found; will recover | |
| 2017-08-16 15:51:11 INFO topology_recovery: Running 1 PreFailoverProcesses hooks | |
| 2017-08-16 15:51:11 INFO topology_recovery: Running PreFailoverProcesses hook 1 of 1: echo 'Will recover from DeadMaster on db1:3306' >> /tmp/recovery.log | |
| 2017-08-16 15:51:11 INFO CommandRun(echo 'Will recover from DeadMaster on db1:3306' >> /tmp/recovery.log,[]) | |
| 2017-08-16 15:51:11 INFO CommandRun/running: bash /tmp/orchestrator-process-cmd-519328150 | |
| 2017-08-16 15:51:11 INFO CommandRun successful. exit status 0 | |
| 2017-08-16 15:51:11 INFO topology_recovery: Completed PreFailoverProcesses hook 1 of 1 in 2.403677ms | |
| 2017-08-16 15:51:11 INFO Completed PreFailoverProcesses hook 1 of 1 in 2.403677ms | |
| 2017-08-16 15:51:11 INFO topology_recovery: done running PreFailoverProcesses hooks | |
| 2017-08-16 15:51:11 INFO topology_recovery: RecoverDeadMaster: will recover db1:3306 | |
| 2017-08-16 15:51:11 INFO topology_recovery: RecoverDeadMaster: masterRecoveryType=MasterRecoveryGTID | |
| 2017-08-16 15:51:11 INFO topology_recovery: RecoverDeadMaster: regrouping replicas via GTID | |
| 2017-08-16 15:51:11 DEBUG Stopping 1 replicas via StopReplicationNicely | |
| 2017-08-16 15:51:11 INFO Stopped slave nicely on db2:3306, Self:mysqld-bin.000001:665, Exec:mysqld-bin.000001:408 | |
| 2017-08-16 15:51:11 INFO Stopped slave on db2:3306, Self:mysqld-bin.000001:665, Exec:mysqld-bin.000001:408 | |
| 2017-08-16 15:51:11 DEBUG - sorted replica: db2:3306 mysqld-bin.000001:408 | |
| 2017-08-16 15:51:11 DEBUG GetCandidateReplica: candidate: db2:3306, ahead: 0, equal: 0, late: 0, break: 0 | |
| 2017-08-16 15:51:11 DEBUG RegroupReplicasGTID: working on 0 replicas | |
| 2017-08-16 15:51:11 INFO Started slave on db2:3306 | |
| 2017-08-16 15:51:11 DEBUG RegroupReplicasGTID: done | |
| 2017-08-16 15:51:11 INFO auditType:regroup-replicas-gtid instance:db1:3306 cluster:db1:3306 message:regrouped replicas of db1:3306 via GTID; promoted db2:3306 | |
| 2017-08-16 15:51:11 INFO auditType:begin-downtime instance:db1:3306 cluster:db1:3306 message:owner: ubuntu, reason: lost-in-recovery | |
| 2017-08-16 15:51:11 INFO topology_recovery: RecoverDeadMaster: 0 postponed functions | |
| 2017-08-16 15:51:11 INFO topology_recovery: promoted replica: db2:3306 | |
| 2017-08-16 15:51:11 INFO auditType:recover-dead-master instance:db1:3306 cluster:db1:3306 message:promoted replica: db2:3306 | |
| 2017-08-16 15:51:11 INFO topology_recovery: checking if should replace promoted replica with a better candidate | |
| 2017-08-16 15:51:11 INFO topology_recovery: RecoverDeadMaster: successfully promoted db2:3306 | |
| 2017-08-16 15:51:11 INFO topology_recovery: Running 1 PostMasterFailoverProcesses hooks | |
| 2017-08-16 15:51:11 INFO topology_recovery: Running PostMasterFailoverProcesses hook 1 of 1: echo 'Recovered from DeadMaster on db1:3306. Failed: db1:3306; Promoted: db2:3306' >> /tmp/recovery.log | |
| 2017-08-16 15:51:11 INFO CommandRun(echo 'Recovered from DeadMaster on db1:3306. Failed: db1:3306; Promoted: db2:3306' >> /tmp/recovery.log,[]) | |
| 2017-08-16 15:51:11 INFO CommandRun/running: bash /tmp/orchestrator-process-cmd-728553469 | |
| 2017-08-16 15:51:11 INFO CommandRun successful. exit status 0 | |
| 2017-08-16 15:51:11 INFO topology_recovery: Completed PostMasterFailoverProcesses hook 1 of 1 in 2.056175ms | |
| 2017-08-16 15:51:11 INFO Completed PostMasterFailoverProcesses hook 1 of 1 in 2.056175ms | |
| 2017-08-16 15:51:11 INFO topology_recovery: done running PostMasterFailoverProcesses hooks | |
| 2017-08-16 15:51:11 INFO topology_recovery: - RecoverDeadMaster: updating cluster_alias: db1:3306 -> db2:3306 | |
| 2017-08-16 15:51:11 ERROR Error 1062: Duplicate entry 'db2:3306' for key 'PRIMARY' | |
| 2017-08-16 15:51:11 ERROR Error 1062: Duplicate entry 'db2:3306' for key 'PRIMARY' | |
| 2017-08-16 15:51:11 INFO Topology recovery: {"Id":1,"UID":"379355970:e5dca94c6b319d861e58fe6829d7500a62134f421c14dde04f8b708e5f83d2b1","AnalysisEntry":{"AnalyzedInstanceKey":{"Hostname":"db1","Port":3306},"AnalyzedInstanceMasterKey":{"Hostname":"","Port":0},"ClusterDetails":{"ClusterName":"db1:3306","ClusterAlias":"db1","ClusterDomain":"","CountInstances":4,"HeuristicLag":0,"HasAutomatedMasterRecovery":false,"HasAutomatedIntermediateMasterRecovery":false},"IsMaster":true,"IsCoMaster":false,"LastCheckValid":true,"CountReplicas":1,"CountValidReplicas":1,"CountValidReplicatingReplicas":0,"CountReplicasFailingToConnectToMaster":0,"CountStaleReplicas":0,"ReplicationDepth":0,"SlaveHosts":[{"Hostname":"db2","Port":3306}],"IsFailingToConnectToMaster":false,"Analysis":"DeadMaster","Description":"Master is reachable but its single slave is not replicating","StructureAnalysis":null,"IsDowntimed":false,"DowntimeEndTimestamp":"","DowntimeRemainingSeconds":0,"IsBinlogServer":false,"PseudoGTIDImmediateTopology":false,"OracleGTIDImmediateTopology":true,"MariaDBGTIDImmediateTopology":false,"BinlogServerImmediateTopology":false,"CountStatementBasedLoggingReplicas":0,"CountMixedBasedLoggingReplicas":0,"CountRowBasedLoggingReplicas":1,"CountDistinctMajorVersionsLoggingReplicas":1,"IsActionableRecovery":true},"SuccessorKey":{"Hostname":"db2","Port":3306},"SuccessorAlias":"","IsActive":false,"IsSuccessful":false,"LostReplicas":[],"ParticipatingInstanceKeys":[],"AllErrors":[],"RecoveryStartTimestamp":"","RecoveryEndTimestamp":"","ProcessingNodeHostname":"","ProcessingNodeToken":"","Acknowledged":false,"AcknowledgedAt":"","AcknowledgedBy":"","AcknowledgedComment":"","LastDetectionId":0,"RelatedRecoveryId":0,"RecoveryType":"MasterRecoveryGTID"} | |
| 2017-08-16 15:51:11 INFO topology_recovery: Running 1 PostFailoverProcesses hooks | |
| 2017-08-16 15:51:11 INFO topology_recovery: Running PostFailoverProcesses hook 1 of 1: echo '(for all types) Recovered from DeadMaster on db1:3306. Failed: db1:3306; Successor: db2:3306' >> /tmp/recovery.log | |
| 2017-08-16 15:51:11 INFO CommandRun(echo '(for all types) Recovered from DeadMaster on db1:3306. Failed: db1:3306; Successor: db2:3306' >> /tmp/recovery.log,[]) | |
| 2017-08-16 15:51:11 INFO CommandRun/running: bash /tmp/orchestrator-process-cmd-410464568 | |
| 2017-08-16 15:51:11 INFO CommandRun successful. exit status 0 | |
| 2017-08-16 15:51:11 INFO topology_recovery: Completed PostFailoverProcesses hook 1 of 1 in 1.843223ms | |
| 2017-08-16 15:51:11 INFO Completed PostFailoverProcesses hook 1 of 1 in 1.843223ms | |
| 2017-08-16 15:51:11 INFO topology_recovery: done running PostFailoverProcesses hooks | |
| 2017-08-16 15:51:11 INFO topology_recovery: Waiting for 0 postponed functions | |
| 2017-08-16 15:51:11 DEBUG PostponedFunctionsContainer: waiting on 0 postponed functions | |
| 2017-08-16 15:51:11 DEBUG PostponedFunctionsContainer: done waiting | |
| 2017-08-16 15:51:11 INFO topology_recovery: Executed 0 postponed functions | |
| 2017-08-16 15:51:11 DEBUG ChangeMasterTo: will attempt changing master on db1:3306 to db2:3306, mysqld-bin.000001:665 | |
| 2017-08-16 15:51:11 INFO ChangeMasterTo: Changed master on db1:3306 to: db2:3306, mysqld-bin.000001:665. GTID: true | |
| db2:3306 | |
| mysqld-bin.000001:665 | |
| 2017-08-16 15:51:11 DEBUG Promoted db2:3306 as new master. Binlog coordinates at time of promotion: mysqld-bin.000001:665 | |
| ubuntu@admin:~$ orchestrator -c topology -i db1 | |
| 2017-08-16 15:51:28 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:51:28 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:51:28 DEBUG Initializing orchestrator | |
| 2017-08-16 15:51:28 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:51:28 DEBUG instanceKey: db1:3306 | |
| 2017-08-16 15:51:28 DEBUG instanceKey: db2:3306 | |
| 2017-08-16 15:51:28 DEBUG instanceKey: db3:3306 | |
| 2017-08-16 15:51:28 DEBUG instanceKey: db4:3306 | |
| - db1:3306 [null,nonreplicating,5.6.36-82.1-log,ro,ROW,>>,GTID,downtimed] | |
| + db2:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| + db3:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| + db4:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| ubuntu@admin:~$ orchestrator -c set-writeable -i db2 | |
| 2017-08-16 15:51:59 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:51:59 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:51:59 DEBUG Initializing orchestrator | |
| 2017-08-16 15:51:59 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:51:59 INFO instance db2:3306 read_only: false | |
| 2017-08-16 15:51:59 INFO auditType:read-only instance:db2:3306 cluster:db1:3306 message:set as false | |
| db2:3306 | |
| ubuntu@admin:~$ orchestrator -c start-slave -i db1 | |
| 2017-08-16 15:52:13 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:52:13 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:52:13 DEBUG Initializing orchestrator | |
| 2017-08-16 15:52:13 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:52:13 INFO Started slave on db1:3306 | |
| db1:3306 | |
| ubuntu@admin:~$ orchestrator -c detach-replica -i db2 | |
| 2017-08-16 15:52:39 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:52:39 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:52:39 DEBUG Initializing orchestrator | |
| 2017-08-16 15:52:39 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:52:39 INFO Will detach db2:3306 | |
| 2017-08-16 15:52:39 INFO auditType:begin-maintenance instance:db2:3306 cluster:db1:3306 message:maintenanceToken: 3, owner: ubuntu, reason: detach replica | |
| 2017-08-16 15:52:39 INFO Stopped slave on db2:3306, Self:mysqld-bin.000001:665, Exec:mysqld-bin.000001:665 | |
| 2017-08-16 15:52:39 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| 2017-08-16 15:52:39 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| 2017-08-16 15:52:40 INFO Started slave on db2:3306 | |
| 2017-08-16 15:52:40 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| 2017-08-16 15:52:40 INFO auditType:end-maintenance instance:db2:3306 cluster:db1:3306 message:maintenanceToken: 3 | |
| 2017-08-16 15:52:40 FATAL Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| ubuntu@admin:~$ orchestrator -c reset-slave -i db2 | |
| 2017-08-16 15:53:02 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:53:02 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:53:02 DEBUG Initializing orchestrator | |
| 2017-08-16 15:53:02 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:53:02 INFO Will reset replica on db2:3306 | |
| 2017-08-16 15:53:02 INFO auditType:begin-maintenance instance:db2:3306 cluster:db1:3306 message:maintenanceToken: 4, owner: ubuntu, reason: reset replica | |
| 2017-08-16 15:53:02 INFO Stopped slave on db2:3306, Self:mysqld-bin.000001:665, Exec:mysqld-bin.000001:665 | |
| 2017-08-16 15:53:02 INFO Reset slave db2:3306 | |
| 2017-08-16 15:53:02 INFO auditType:reset-slave instance:db2:3306 cluster:db1:3306 message:db2:3306 replication reset | |
| 2017-08-16 15:53:02 INFO auditType:end-maintenance instance:db2:3306 cluster:db1:3306 message:maintenanceToken: 4 | |
| db2:3306 | |
| ubuntu@admin:~$ orchestrator -c topology -i db1 | |
| 2017-08-16 15:53:06 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:53:06 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:53:06 DEBUG Initializing orchestrator | |
| 2017-08-16 15:53:06 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:53:06 DEBUG instanceKey: db1:3306 | |
| 2017-08-16 15:53:06 DEBUG instanceKey: db2:3306 | |
| 2017-08-16 15:53:06 DEBUG instanceKey: db3:3306 | |
| 2017-08-16 15:53:06 DEBUG instanceKey: db4:3306 | |
| db2:3306 [0s,ok,5.6.36-82.1-log,rw,ROW,>>] | |
| + db1:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| + db3:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| + db4:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| ubuntu@admin:~$ orchestrator -c move-replicas-gtid -i db2 -d db3 | |
| 2017-08-16 15:54:31 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:54:31 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:54:31 DEBUG Initializing orchestrator | |
| 2017-08-16 15:54:31 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:54:31 INFO Will move 2 replicas below db3:3306 via GTID | |
| 2017-08-16 15:54:31 INFO Will move db4:3306 below db3:3306 via GTID | |
| 2017-08-16 15:54:31 INFO Will move db1:3306 below db3:3306 via GTID | |
| 2017-08-16 15:54:31 INFO auditType:begin-maintenance instance:db4:3306 cluster:db2:3306 message:maintenanceToken: 7, owner: ubuntu, reason: move below db3:3306 | |
| 2017-08-16 15:54:31 INFO auditType:begin-maintenance instance:db1:3306 cluster:db2:3306 message:maintenanceToken: 8, owner: ubuntu, reason: move below db3:3306 | |
| 2017-08-16 15:54:31 INFO Stopped slave on db1:3306, Self:mysqld-bin.000001:665, Exec:mysqld-bin.000001:665 | |
| 2017-08-16 15:54:31 INFO Stopped slave on db4:3306, Self:mysqld-bin.000001:922, Exec:mysqld-bin.000001:665 | |
| 2017-08-16 15:54:31 DEBUG ChangeMasterTo: will attempt changing master on db1:3306 to db3:3306, mysqld-bin.000001:922 | |
| 2017-08-16 15:54:31 DEBUG ChangeMasterTo: will attempt changing master on db4:3306 to db3:3306, mysqld-bin.000001:922 | |
| 2017-08-16 15:54:31 INFO ChangeMasterTo: Changed master on db1:3306 to: db3:3306, mysqld-bin.000001:922. GTID: true | |
| 2017-08-16 15:54:31 INFO ChangeMasterTo: Changed master on db4:3306 to: db3:3306, mysqld-bin.000001:922. GTID: true | |
| 2017-08-16 15:54:31 INFO Started slave on db1:3306 | |
| 2017-08-16 15:54:31 INFO Started slave on db4:3306 | |
| 2017-08-16 15:54:31 INFO auditType:move-below-gtid instance:db1:3306 cluster:db2:3306 message:moved db1:3306 below db3:3306 | |
| 2017-08-16 15:54:31 INFO auditType:end-maintenance instance:db1:3306 cluster:db2:3306 message:maintenanceToken: 8 | |
| 2017-08-16 15:54:31 INFO auditType:move-below-gtid instance:db4:3306 cluster:db2:3306 message:moved db4:3306 below db3:3306 | |
| 2017-08-16 15:54:31 INFO auditType:end-maintenance instance:db4:3306 cluster:db2:3306 message:maintenanceToken: 7 | |
| 2017-08-16 15:54:31 INFO auditType:move-replicas-gtid instance:db3:3306 cluster:db2:3306 message:moved 2/2 replicas below db3:3306 via GTID | |
| db1:3306 | |
| db4:3306 | |
| ubuntu@admin:~$ orchestrator -c topology -i db1 | |
| 2017-08-16 15:54:35 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:54:35 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:54:35 DEBUG Initializing orchestrator | |
| 2017-08-16 15:54:35 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:54:35 DEBUG instanceKey: db1:3306 | |
| 2017-08-16 15:54:35 DEBUG instanceKey: db2:3306 | |
| 2017-08-16 15:54:35 DEBUG instanceKey: db3:3306 | |
| 2017-08-16 15:54:35 DEBUG instanceKey: db4:3306 | |
| db2:3306 [0s,ok,5.6.36-82.1-log,rw,ROW,>>] | |
| + db3:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| + db1:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| + db4:3306 [0s,ok,5.6.36-82.1-log,ro,ROW,>>,GTID] | |
| ubuntu@admin:~$ orchestrator -c make-co-master -i db3 -d db2 | |
| 2017-08-16 15:54:52 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:54:52 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:54:52 DEBUG Initializing orchestrator | |
| 2017-08-16 15:54:52 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:54:52 DEBUG Will check whether db3:3306's master (db2:3306) can become its co-master | |
| 2017-08-16 15:54:52 INFO Will make db3:3306 co-master of db2:3306 | |
| 2017-08-16 15:54:52 INFO auditType:begin-maintenance instance:db3:3306 cluster:db2:3306 message:maintenanceToken: 9, owner: ubuntu, reason: make co-master of db2:3306 | |
| 2017-08-16 15:54:52 INFO auditType:begin-maintenance instance:db2:3306 cluster:db2:3306 message:maintenanceToken: 10, owner: ubuntu, reason: db3:3306 turns into co-master of this | |
| 2017-08-16 15:54:52 DEBUG ChangeMasterTo: will attempt changing master on db2:3306 to db3:3306, mysqld-bin.000001:922 | |
| 2017-08-16 15:54:52 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| 2017-08-16 15:54:52 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| 2017-08-16 15:54:52 ERROR Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| 2017-08-16 15:54:52 INFO auditType:end-maintenance instance:db2:3306 cluster:db2:3306 message:maintenanceToken: 10 | |
| 2017-08-16 15:54:52 INFO auditType:end-maintenance instance:db3:3306 cluster:db2:3306 message:maintenanceToken: 9 | |
| 2017-08-16 15:54:52 FATAL Error 1776: Parameters MASTER_LOG_FILE, MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_POS cannot be set when MASTER_AUTO_POSITION is active. | |
| ubuntu@admin:~$ orchestrator -c graceful-master-takeover -i db2 | |
| 2017-08-16 15:55:10 DEBUG Connected to orchestrator backend: orc_server_user:?@tcp(127.0.0.1:3306)/orchestrator?timeout=1s | |
| 2017-08-16 15:55:10 DEBUG Orchestrator pool SetMaxOpenConns: 128 | |
| 2017-08-16 15:55:10 DEBUG Initializing orchestrator | |
| 2017-08-16 15:55:10 INFO Connecting to backend: maxConnections: 128, maxIdleConns: 32 | |
| 2017-08-16 15:55:10 FATAL GracefulMasterTakeover: master db2:3306 should only have one replica (making the takeover safe and simple), but has 3. Aborting |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment