Self deploying schema

idea.md

The problem

"Roll out non-breaking changes to biz ops schema without requiring us to redeploy any applications"

More detailed requirements

1. Schema change propagates to all consumers on creation of git semver tag
2. No dramatic differences in how biz-ops-schema library is used in each app
3. API updates before others
4. Able to easily work with local data when npm linking/working offline
5. Nothing picks up major releases of schema without installing new version of node module
6. If the API is on v2 and other apps still on v1, nothing breaks (major releases will typically mean the structure of the data in the schema files/library has changed, but the actual API endpoints won't have changed, so should still work fine for all consumers)

Proposal

• A semver release x.y.z of biz-ops-schema releases a new version of the packaage and deploys aschema file to {s3 bucket}/vx/latest.json
• The file's structure will be:

     {
        schema: //the entire schema as json
        version: //git commit hash
     }

• The biz-ops-schema library can be configured with a BASE_URL to poll for schema updates
  • The API will be configured with {s3 bucket}
  • All other apps will be configured with biz-ops-{staging}.api.ft.com/schema (or could go via api.ft.com/biz-ops, but want to avoid having to pass creds. *new thought - coudl the API post to {s3 bucket}/{prod|staging}/{version}? How would that work?)
• Any app with the library installed will poll ${BASE_URL}/v${version of library, read from package.json}/latest.json
• When the content retrieved from the app is detected to have a different version property, the cached data is replaced wit the new
• The library also fires an event on update (might need an on before update event too). In the API this is listened to and used to
  1. generate a new graphQL API, which replaces the last one
  2. serve the new schema from its /schema/vN/latest.json endpoint
• If a request comes in for a different version of the schema to the one the api is using, it is proxied through to s3

So for all apps except the API, there's no change really, other than having to add a little config to the biz-ops-schema library In the API there's a bit to do, but it's not ridiculously difficult, though perhaps a little unintuitive and weird

For managing particularly tricky things I'm thinking there should also be an easy way to put an application into 'just use the schema you have installed' mode. That might actually be a better way to manage major version changes than proxying through to s3 for previous schemas: disable autoupdate until everything is on the same version, then turn back on... but that requires a level of coordination this whole project is designed to avoid

Will apps (other than the API) use any of the schema data from the node library they've installed? It sounds like once they start polling the API, they'll always be using a local cache of the central data. In which case, is there any benefit in distributing that info via the package? Having two sources of the similar data, one of which is stale, is a recipe for confusion.
Perhaps only the API needs a package with the schema in and the other apps should use a different lightweight package, which does the polling and caching?

Cheers for giving it ht once over. The local copy of the schema is mainly for local development (npm linking and such like). You're right that I should prevent/discourage using it in prod. In general, even the API will be polling for fresh data too, so sharing the library but varying the endpoint makes sense to me.

We have a balance here between each app having direct access to the schema (and needing to be rebuilt each time it changes) and the api being the host of the schema and each consumer calling the api for meta data.
The benefit of the former is that anyone can experiment with the schema at a data level or at a meta level without needing a private version of the api (they just need a local copy of the schema). The effect of the latter is the api owns everything and becomes a bottleneck to quick local development and latency for consumers due to the need for network requests to the api.

What we really need is dynamic npm inclusion at runtime! We would then have the new schema automatically included into the api and its consumers each time we published the npm.

Oooh. Someone has been looking at dynamic import recently: https://github.com/tc39/proposal-dynamic-import Could we use it?

And a very highly used npm module! https://www.npmjs.com/package/babel-plugin-dynamic-import-node

One big consideration for having it set up as I plan is

If the API is on v2 and other apps still on v1, nothing breaks (major releases will typically mean the structure of the data in the schema files/library has changed, but the actual API endpoints won't have changed, so should still work fine for all consumers)

For this to work there needs to be some persistent source of schema info outside of the api runtime, otherwise other apps become too coupled to the API, and we can never make breaking changes to the schema structure without downtime and coordinating releases

dynamic npm inclusion at runtime

Not sure this is workable, but even if it was, how would this be simpler than publishing to s3 and sharing from there? Also doesn't solve requirement 3 - need to orchestrate the API updating before everything else