Embedded migration issue and solution options

gistfile1.txt

Steps to reproduce the issue (program enrolment is used as example, it will happen with all entities)
1. Start with old schema database (181) on the client and then start with new version of the app (185, 7.0.x).
2. Let the migration run for a few seconds into the ProgramEncounter migration.
3. make stop_app
4. make open_db (see the obs in program enrolments are empty) and the schema for it is Embedded.
5. make start_app (see the schema version is 181) and migration runs again but it fails to make program enrolments observations non empty.

Reason.
In step 2 we do commit transaction. This commits the new (185) schema changes but not the schema version. Due the commit realm file gets changed. This was verified by throwing an error before transaction commit and open_db, the schema is not migrated in that case and data is all there. So because the schema gets changed, the observations becomes empty - treated like a new field.
In step 5 since the realm file has already changes the migration doesn't do anything for program enrolments, just runs through them.


Solution Tradeoffs.
1. Whether multiple tries are provided to user.
2. Saving entity queue data to server.
3. Incremental migration.
4. Storage space availability (minor tradeoff, given we already assume users use fast sync).
5. Code complexity.

Solution 1 (Multiple tries. No data loss. No incremental migration. Code=Complex.)
- if FlagInDb=null Create a backup of realm file before starting migration if space exists.
- if FlagInDb=null Run migration and immediately set FlagInDb=Started and on completion set FlagInDb=Completed.
- if FlagInDb=Started then replace realm file with backup file and run migration again. If backup file doesn't exist then throw error to user to reinstall.
- if FlagInDb=Completed delete the backup file.


Solution 2. (Single try. Data loss possible. No incremental migration. Code=Simple)
Same as above except the back up file. Only one try allowed and unsaved data can get lost.

Solution 3. (Multiple tries. No data loss. Incremental migration. Code=Complex.)
Do the same as solution one at application level (comprehensive, complex)
- Serialise schema JSON from from 6.1 and commit to 7.0.x branch
- TrySafe opening old db with 6.1. If it has EntityQueue items. Push the data to server. Switch to schema definition in current code.
- Create a new realm file (like we do in fast sync) and perform migration outside realm's control and then replace the file on success. Save incremental state like Individual.obs, PE.obs, PE.cancelObs, etc.
Avni migration - to allow repeatability and potentially incremental migration.

Additionally.
- Display the white screen message in multiple languages. Default Hindi and give option to switch language.

1. Create newObservations with embeddedObservations. Maintain this value, but don't use it anywhere in code
2. Copy over observations every time we save observations
3. Add an on-start verification and migration in code (outside realm)
4. Add analytics on sync_telemetry that verifies the consistency of the data
5. Delete observations later. Alternatively, we have a backout mechanism if things don't look good

Advantage

Safe
Rerunnable
Ability to back out if things don't work as per plan

Consequence

Doubling of data size on device