schema-updates.md

Proposal:

1. The schema version becomes an abridged semver: [major].[minor]. A major version change means something in code needs to be changed to support the new data (to the extent that we think it matters — we could decide that something wasn't a big deal if slightly misformatted). Minor version changes are safe to apply — they might add a field to something, but not in a way that's expected to cause any problems.

2. A given client version comes bundled with a given schema version, and it can fetch minor schema updates for its major version, on daily checks or when it sees a Zotero-Schema-Version header when starting a sync. This would be client-based, not server-based, so for the daily check we'd make schema versions available as /schema/4 or something (which might serve version 4.2). If a sync returned Zotero-Schema-Version and it was a higher major version, the client would ignore it.

3. When uploading data to a library, the client would include its current schema version in a Zotero-Schema-Version request header, and the server would store the version with each library if it was a known schema version (to avoid mischief) and greater than the currently stored version for that library.

4. When starting a sync for a library, the client would check the Zotero-Library-Schema-Version header, which would be the stored version for that library, meaning it's the highest-possible version for data in that library. If it was higher than the client's current schema version, the client would stop syncing that library and say that syncing that library required a newer version of the app. (If it was only a higher minor version, it might mean that the minor schema update (from Zotero-Schema-Version at the start of the main sync process) failed for some reason, and it could just show a temporary error rather than saying the client needs to be updated.)

5. Unknown properties shouldn't ever happen under this scheme, so they would cause the object download to fail. The items would be retried on a backoff schedule (in case there was a server-side problem) or after an upgrade (in case there was a client bug), as they already are now (at least in the desktop app).

Issues:

1. Just because a client with a given schema version writes to a library, it doesn't necessarily mean the data is incompatible, but we have no good way of knowing that, so it requires a sync cut-off. (I think it would be crazy for the API to start comparing the data to all past schema versions, for example.)

2. This cut-off would happen even when we added new object types (e.g., annotations) that an older client wouldn't try to download anyway because it didn't know about them. (I think this would, in fact, mean that there was no reason to track library versions separately for different object types, as Michal said he was doing, becuase the client wouldn't even try to sync the library if it didn't support the new object type.)

3. This only partly solves the beta problem. It means that we can make a new major version available on the server and also bundle it with a beta, which is necessary for testing new sync-dependent features (good), but if the beta writes to a library, no non-beta clients will be able to sync with that library (bad).

4. We'd still want as much as possible in the schema, to minimize major versions. So as Michal says, item type image URLs (of various sizes) should be in there, and we'd want to think about other things that might help avoid major versions.

Bonus Proposal:

1. The best way to keep the cut-off from affecting too many people would be to roll out app versions that could support a new major schema version but that didn't expose the associated functionality in the UI until they were offered the new major schema version from the API. That would let us remotely turn on features after most users had upgraded to a compatible version. Unfortunately, the semver approach on its own prevents that, because it means the client, rather than the server, decides which clients to send a new major version to. (Doing it server-side also wouldn't be very nice to unofficial clients.)

2. A hybrid approach could be to do semver but also set a maximum major version in the client that it could upgrade to if available, and hide features until the major version was offered. So if the client had schema 2.4 but it had a maxSchemaVersion of 3, it would check /schema/3 before checking /schema/2, and only use /2 if /3 was a 404. Similarly, Zotero-Schema-Version from the API would offer a comma-separated list of the latest available version for each major version, and if the client with 2.4 and maxSchemaVersion of 3 saw that a 3.2 was available, it could upgrade to that and expose the hidden functionality.

3. We would test this by dropping in a 3.2 schema file locally and/or by adding 3.2 to apidev responses.

4. It's a little weird to turn on functionality remotely — and it does increase the chances of a bug that suddenly appeared even though someone hadn't upgraded (perhaps purposely) — but I think it'd be the best way to minimize sync cut-offs.

5. I'm not sure if Apple has some app store rule against enabling new functionality like this.

The whole idea of ever getting a message that says you need to upgrade to sync is sort of unpleasant, and a major departure from our historical practice (where we didn't cut off anything for many years and then cut off 4.0 only after 5.0 had been out for about a year), but I don't see a better option, and this last part would at least keep most regular upgraders from seeing such a message, at least when we went to the trouble of adding forward-compatibility.

Issues:

So if beta testers suddenly decided they want to quit for some reason, they would have to remove the app and reinstall from app store for it to work.

To clarify, I mean the normal beta track of the desktop app too, which you don't need to sign up for. And it's not just the person's personal library — I agree that should be expected. It's that if they make any changes in a group library using a higher major schema version, they'll lock out anyone from that library who's not using the beta.

Maybe just a dialog box when a major schema version update happens while you're using a beta version, warning that your Zotero database was updated and that making changes in a group library may prevent non-beta users from syncing until the next official release? It'd get a little annoying, but 1) we hopefully won't make major version updates too often and 2) it'd be better than locking your whole class/team/company out of your group library.

Bonus:

1. In what I'm proposing, pushing the next major schema version would still be something we'd switch on on the server only when enough people had upgraded to a version that could support it and not give them the upgrade message. It's just that the clients would know the maximum versions they could accept — and they'd have to anyway, because they'd have to have been coded to do so.

2. It's fairly common to have "feature toggles"

Yes, though not necessarily ones that are enabled without user interaction. E.g., you can do an in-app purchase and enable new functionality, but that's something you chose to do.

Anyway if we notice increase in crashes after we publish a new schema, we can always just remove the schema again. It shouldn't be a problem because even if for example we added a new object and we remove the schema, the (possibly broken object which causes crash) would still be stored in DB, but it wouldn't appear to the user because the schema doesn't support it. Broken UI elements would be hidden again. I guess it should/could work.

I don't think we want to support schema downgrading — a schema upgrade could change the DB, and it's not clear what would happen to existing data. It's also just as likely that the new data alone could cause a problem even if the UI elements became hidden again. I think this is just something we try to avoid with testing, and do emergency updates if a problem emerges.

Or we could have feature toggles separated from schema. The new schema would update immediately, but big new features could be enabled by another endpoint (for example new objects, ui components or whole screens).

That's an interesting idea, though disabling a feature that we've already announced wouldn't be great, and having a feature appear that we hadn't announced also wouldn't be great. And, as above, it's possible that the bug that would appear would be a result of the data rather than the schema itself or the UI, and that data couldn't be created (on your device or a synced one) until the UI was enabled. So I doubt the ability to disable a UI feature remotely is worth it.

In general, I'm not sure to what extent the App Store requires a change in thinking. If something broke in the desktop app, we'd just fix it immediately and make a new version available. It seems like App Store review has gotten to the point where it's pretty quick (one day?), and isn't there some emergency mechanism you can use every so often? Also, auto-updates are enabled by default on iOS and semi-automatic on desktop, so except for the paranoid, who can disable whatever pref controls schema updates too, it's not like there aren't essentially remote-triggered updates all the time.

So maybe breakage is breakage, and we just try to test and avoid it and put out quick fixes when things break.

3. It should be fine with Apple. As I mentioned in previous point, we'll have all the code in place and they can inspect it.

I mean, Apple certainly reviews functionality, too — and this would be functionality that literally wouldn't appear without a server trigger. We obviously wouldn't be the first to do something like this, and it's unlikely that anything we added this way would be problematic, but I wouldn't be surprised if the guidelines somehow required you to let them test any new functionality contained in a given release with a demo account. (As an example, if we passed icon URLs in the schema, and there was an "mobile app" item type, and the icon was an Apple logo or an iPhone logo, they would certainly reject if it had shown up when they tested it.)

If people don't want to update their apps then we can't really help them.

Yup.

Maybe just a dialog box when a major schema version update happens while you're using a beta version, warning that your Zotero database was updated and that making changes in a group library may prevent non-beta users from syncing until the next official release? It'd get a little annoying, but 1) we hopefully won't make major version updates too often and 2) it'd be better than locking your whole class/team/company out of your group library.

Not really annoyinig, you inform the user about the thing which is very important and you can have "Don't show again" option in case someone wants to keep using betas all the time. And we can have some small red indicator somewhere as well which would mean that you're going to break it for non-beta users if you make changes.

1. In what I'm proposing, pushing the next major schema version would still be something we'd switch on on the server only when enough people had upgraded to a version that could support it and not give them the upgrade message. It's just that the clients would know the maximum versions they could accept — and they'd have to anyway, because they'd have to have been coded to do so.

Right, I agree that this should work fine.

1. It's fairly common to have "feature toggles"

Yes, though not necessarily ones that are enabled without user interaction. E.g., you can do an in-app purchase and enable new functionality, but that's something you chose to do.

Even without user interaction and in-app purchases I believe many apps have them. It's commonly used for A/B testing to serve different layouts or features. I've had apps that exposed functionality after app review and we weren't rejected for it. Also we can make a tester account for app review team and just bundle the next major schema with it so that it would be exposed for app review and we would be just fine.

In general, I'm not sure to what extent the App Store requires a change in thinking. If something broke in the desktop app, we'd just fix it immediately and make a new version available. It seems like App Store review has gotten to the point where it's pretty quick (one day?), and isn't there some emergency mechanism you can use every so often? Also, auto-updates are enabled by default on iOS and semi-automatic on desktop, so except for the paranoid, who can disable whatever pref controls schema updates too, it's not like there aren't essentially remote-triggered updates all the time.

Yeah app review times are now taking about a day. They've improved on this a lot. There is also an "expedited review" for which you can ask. It's not stated how many times, but unless we do it pretty much every release we should be able to ask for those if needed.

1. It should be fine with Apple. As I mentioned in previous point, we'll have all the code in place and they can inspect it.

I mean, Apple certainly reviews functionality, too — and this would be functionality that literally wouldn't appear without a server trigger. We obviously wouldn't be the first to do something like this, and it's unlikely that anything we added this way would be problematic, but I wouldn't be surprised if the guidelines somehow required you to let them test any new functionality contained in a given release with a demo account. (As an example, if we passed icon URLs in the schema, and there was an "mobile app" item type, and the icon was an Apple logo or an iPhone logo, they would certainly reject if it had shown up when they tested it.)

As I mentioned above we can provide a special account for them and just bundle the new schema which would be used for that account so that they can test the new features. But I've had apps that didn't do anything extra and just passed the review. They can't always control all the content anyway, so if they catch you with an apple logo that appears in the app after review you've got a problem, if they don't then well... You tricked them this time. But of course it'll be safer to just have them review everything and we won't have to worry.