OM-migration-review.md

Reproducing the steps:

❯ python -m venv venv-012
❯ source venv-012/bin/activate
❯ pip install "openmetadata-ingestion[snowflake,mysql,bigquery]~=0.12.3"
# Pin SQLAlchemy to 1.4 for old release without requirement fix
❯ pip install SQLAlchemy~=1.4
❯ metadata --version
metadata, version metadata 0.12.3.0 from /Users/pmbrull/tests/0123-0131-upgrade/venv-012/lib/python3.9 (python 3.9)

Get the 0.12.3 compose file, map the MySQL ports are and

docker compose down -v && docker compose up mysql elasticsearch

Start the 0.12.3 server from tar:

./bootstrap/bootstrap_storage.sh migrate-all
./bin/openmetadata.sh start

Ingested a few sources and created teams and users. Now check the MySQL counts:

Getting the counts from running:

SELECT CONCAT(
    'SELECT "', 
    table_name, 
    '" AS table_name, COUNT(*) AS exact_row_count FROM `', 
    table_schema,
    '`.`',
    table_name, 
    '` UNION '
) 
FROM INFORMATION_SCHEMA.TABLES 
WHERE table_schema = 'openmetadata_db';

TOTAL 536
DATABASE_CHANGE_LOG	5
bot_entity	1
change_event	153
chart_entity	0
dashboard_entity	0
dashboard_service_entity	0
database_entity	3
database_schema_entity	8
dbservice_entity	3
entity_extension	1
entity_extension_time_series	2
entity_relationship	156
entity_usage	0
field_relationship	0
glossary_entity	0
glossary_term_entity	0
ingestion_pipeline_entity	0
location_entity	0
messaging_service_entity	0
metric_entity	0
ml_model_entity	0
mlmodel_service_entity	0
openmetadata_settings	1
pipeline_entity	0
pipeline_service_entity	0
policy_entity	4
report_entity	0
role_entity	2
storage_service_entity	0
table_entity	131
tag	12
tag_category	4
tag_usage	3
task_sequence	1
team_entity	3
test_case	0
test_definition	22
test_suite	0
thread_entity	0
topic_entity	0
type_entity	16
user_entity	5
user_tokens	0
webhook_entity	0

Stop the 0.12.3 server

❯ ./bin/openmetadata.sh stop

Activate the 0.13.1 openmetadata-ingestion

pip install "openmetadata-ingestion==0.13.1.9"
❯ python -m metadata --version
metadata 0.13.1.0 from /Users/pmbrull/tests/0123-0131-upgrade/venv/lib/python3.9 (python 3.9)

❯ metadata backup -u openmetadata_user -p openmetadata_password -H localhost -d openmetadata_db --port 3306
Creating OpenMetadata backup for localhost:3306/openmetadata_db...
Backup stored locally under openmetadata_202301291209_backup.sql
❯ wc -l openmetadata_202301291209_backup.sql
     574 openmetadata_202301291209_backup.sql
# Note that this is the number of 536 + 43 truncate statements - ignoring 5 DATABASE_CHANGE_LOG records

Now we stop MySQL and start it up again from scratch

docker compose down -v && docker compose up mysql elasticsearch

I had a completely empty MySQL, so I am running the migrations for 0.12.3 again to create the base tables.

With the server still down, run the restore:

~/tests/0123-0131-upgrade ❯ metadata restore -H localhost -u openmetadata_user 
-p openmetadata_password -d openmetadata_db --port 3306 
--input openmetadata_202301291209_backup.sql

Restoring OpenMetadata backup for localhost:3306/openmetadata_db
Queries to process for restore: 574
Restore finished. 0 queries failed from 574.
Backup restored from openmetadata_202301291209_backup.sql

Rerun the total count. We still have 536 entries.

With the server still down, run the migrations for 0.13.1

❯ cd openmetadata-0.13.1
❯ ./bootstrap/bootstrap_storage.sh migrate-all

Start the 0.13.1 server

~/tests/01/openmetadata-0.13.1 ❯ ./bin/openmetadata.sh start

Reindex from the UI. We can run through the tables.