Self deploying schema

idea.md

The problem

"Roll out non-breaking changes to biz ops schema without requiring us to redeploy any applications"

More detailed requirements

1. Schema change propagates to all consumers on creation of git semver tag
2. No dramatic differences in how biz-ops-schema library is used in each app
3. API updates before others
4. Able to easily work with local data when npm linking/working offline
5. Nothing picks up major releases of schema without installing new version of node module
6. If the API is on v2 and other apps still on v1, nothing breaks (major releases will typically mean the structure of the data in the schema files/library has changed, but the actual API endpoints won't have changed, so should still work fine for all consumers)

Proposal

• A semver release x.y.z of biz-ops-schema releases a new version of the packaage and deploys aschema file to {s3 bucket}/vx/latest.json
• The file's structure will be:

     {
        schema: //the entire schema as json
        version: //git commit hash
     }

• The biz-ops-schema library can be configured with a BASE_URL to poll for schema updates
  • The API will be configured with {s3 bucket}
  • All other apps will be configured with biz-ops-{staging}.api.ft.com/schema (or could go via api.ft.com/biz-ops, but want to avoid having to pass creds. *new thought - coudl the API post to {s3 bucket}/{prod|staging}/{version}? How would that work?)
• Any app with the library installed will poll ${BASE_URL}/v${version of library, read from package.json}/latest.json
• When the content retrieved from the app is detected to have a different version property, the cached data is replaced wit the new
• The library also fires an event on update (might need an on before update event too). In the API this is listened to and used to
  1. generate a new graphQL API, which replaces the last one
  2. serve the new schema from its /schema/vN/latest.json endpoint
• If a request comes in for a different version of the schema to the one the api is using, it is proxied through to s3

So for all apps except the API, there's no change really, other than having to add a little config to the biz-ops-schema library In the API there's a bit to do, but it's not ridiculously difficult, though perhaps a little unintuitive and weird

For managing particularly tricky things I'm thinking there should also be an easy way to put an application into 'just use the schema you have installed' mode. That might actually be a better way to manage major version changes than proxying through to s3 for previous schemas: disable autoupdate until everything is on the same version, then turn back on... but that requires a level of coordination this whole project is designed to avoid

One big consideration for having it set up as I plan is

If the API is on v2 and other apps still on v1, nothing breaks (major releases will typically mean the structure of the data in the schema files/library has changed, but the actual API endpoints won't have changed, so should still work fine for all consumers)

For this to work there needs to be some persistent source of schema info outside of the api runtime, otherwise other apps become too coupled to the API, and we can never make breaking changes to the schema structure without downtime and coordinating releases

dynamic npm inclusion at runtime

Not sure this is workable, but even if it was, how would this be simpler than publishing to s3 and sharing from there? Also doesn't solve requirement 3 - need to orchestrate the API updating before everything else