Skip to content

Instantly share code, notes, and snippets.

@mattlord
Last active July 10, 2021 00:28
Show Gist options
  • Save mattlord/3afe4f920d9a59d39efcf84564bf4de2 to your computer and use it in GitHub Desktop.
Save mattlord/3afe4f920d9a59d39efcf84564bf4de2 to your computer and use it in GitHub Desktop.
The Pos column in the _vt.vreplication table is the vreplication equivalent of mysqld's @@global.gtid_executed value
The hard part is finding the source GTID you think you want to skip, verifying the contents of it and that it matches the error, and finally that it's safe to skip
You can find the source tablet for that errored vstream in the workflow output, e.g.:
...
{
"Shard": "-",
"Tablet": "us_central1_a-3612345650",
"ID": 16,
"Bls": {
"keyspace": "foob",
"shard": "80-",
...
From there, you can find the mysqld instance, e.g.:
$ vtctl ListAllTablets | awk '$2 == "foob" && $3 == "80-" && $4 == "master" {print $6; exit}'
dbfoobhost:3306
Then you can see that instance's server_uuid, e.g.:
[root@dbfoobhost:/home/mlord] $ mysql --defaults-file=/file/to/creds.cnf -BNe "select @@global.server_uuid"
70d613d1-678c-b18b-7704-64b6591be455
And then come back to the workflow's vreplication record to find that UUID in the Pos field, e.g.:
"Pos": "MySQL56/70d613d1-678c-b18b-7704-64b6591be455:1-2646817,fb827d7a-696b-7ad1-988c-c77ff601c177:1-28"
The problematic GTID is likely the high value +1 -- so in this case 70d613d1-678c-b18b-7704-64b6591be455:2646818 -- but you can verify by finding that GTID in the source hosts's binlogs e.g.:
[root@dbfoobhost:/home/mlord] $ for file in /mysql/*-bin.*; do mysqlbinlog -vvv --base64-output=DECODE-ROWS ${file} | grep -A30 "70d613d1-678c-b18b-7704-64b6591be455:2646818" && echo "Found in ${file}"; done
Then you can verify that's the right GTID by looking at the values shown for the write in the binlog events and compare it to the error seen for that vreplication stream
Then you can work with the app team to be sure it's safe to skip it on the target side...
And if so, add that value to the Pos field on the target mysqld intance, e.g.:
[root@dbfoobhost:/home/mlord] $ mysql --defaults-file=/file/to/creds.cnf -e "UPDATE _vt.vreplication SET Pos='MySQL56/70d613d1-678c-b18b-7704-64b6591be455:1-2646818,fb827d7a-696b-7ad1-988c-c77ff601c177:1-28' WHERE id=16"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment