vega-polaris

Keybase proof

I hereby claim:

• I am featherweightdev on github.
• I am featherweight (https://keybase.io/featherweight) on keybase.
• I have a public key ASBD0rxdyp3nYUBuUtBRXu4rDOgn2brmE8iDfxBLFs95Mgo

To claim this, I am signing this object:

vega-polaris

In my previous post I explained the logic of starting our cluster small, and scaling it up as we go along. Now that we know the rules to follow, how do we actually scale?

Some notes about when and how to scale

Just a quick summary of the previous post: we're going to have logging set up around the bottleneck metrics discussed there. Once a bottleneck threshold is reached, the on-call (which, hopefully, is one of us) will get an alert that mentions which cluster is hitting its bottleneck threshold. At that point, we'll put a freeze on any new migrations, until your work in scaling up the cluster is done.

Here's the catch: you have to wait until current migrations are done before you start scaling the cluster. We don't know for sure what will happen because there's too many factors involved (feel free to update this doc if you just found out what happens...) so we will just avoid messing with it entirely.

vega-polaris

Restoring a snapshot in ES6

A lot can happen during a migration, for example, we could somehow mess up our original (ES6) cluster. Luckily, even if we do, it's not the end of the world. We can always restore the cluster from a snapshot.

I've already written about doing that on ES8, since we keep taking that whole cluster down. This post is the same idea, only for ES6, in case we need to do that during the migration. But if you're not sure what a snapshot is, or how it helps, please skim through that previous post - it's already there.

We have a regularly-running snapshot creation job for ES6, that runs weekly on both crawldb prod and trackdb prod. We may be running it on a different schedule during the migration, but either way, we wil

vega-polaris

Restoring a snapshot in ES6

A lot can happen during a migration, for example, we could somehow mess up our original (ES6) cluster. Luckily, even if we do, it's not the end of the world. We can always restore the cluster from a snapshot.

I've already written about doing that on ES8, since we keep taking that whole cluster down. This post is the same idea, only for ES6, in case we need to do that during the migration. But if you're not sure what a snapshot is, or how it helps, please skim through that previous post - it's already there.

We have a regularly-running snapshot creation job for ES6, that runs weekly on both crawldb prod and trackdb prod. We may be running it on a different schedule during the migration, but either way, we wil