Rocks

Rocks.md

Rocks

Sync V3 Infrastructure Abstraction

Basics

At its core, the system is event driven.

ex: "a record changed", "an Entity changed", or [future event type]

In response to events, we need to take different Actions. Also, Events don't just magically come into the system - we have to listen for them, and sometimes go out and get them.

[Source] --Emit Event--> [Handler] --Emit Action--> [Actor]

Components

A sync process can be constructed from 3 different items

Sources

Processes that listen for (or actively find) conditions under which events should be triggered

ex: tail (a process that searches for updated records), webhooks (a listening process that triggers events on incoming reqs)

Mechanics

Sources are long-lived processes that are built to emit Events.

Processes can listen for incoming events (like webhooks) or go looking for new events (like the Tail process). It's not clear yet whether we'll put Sources onto EC2, Fargate, or some other service for long-lived processes.

Actors

The outputs of our system - actually do something in response to Events.

ex: "update an Entity", "write out to a third party system", or "write to the warehouse"

Mechanics

Calls for Actions are pushed into an SQS queue. Lambdas are triggered with the requests for work.

Before they're pushed into the SQS queue, a record of the call for action is record into DynamoDB. Once the task completes, it removes its own entry from DynamoDB. Any tasks that haven't marked themselves completed within 15 minutes (the max timeout of a Lambda) will be assumed failed and be re-added to the queue.

Handlers

Pieces of code that wait for Events to happen. When something happens in the system that they care about, Event Listeners will trigger Actions based on what the customer's configuration (their Workflow) says should happen.

ex: "a record changed, go update the corresponding Entity" or "an Entity changed, sync the changes"

Mechanics

Events come into the system via SQS queue, triggering a generic Event Lambda. Handlers are units of code that are called by the generic Event Lambda depending on the type of the Event.

basic blueprint:

module.exports = {
  listeners: {
    lookForChanges: {
      type: 'tail',
      loopIntervalMs: 5 * 60 * 1000
    }
  },

  actors: {
    syncChanges: {
      type: 'write',
      rateLimits: [
        {
          type: 'LIMIT_PER_PERIOD',
          agent: 'hubspot1',
          periodMs: 5 * 60 * 1000
        }
      ]
    }
  },

  handlers: {
    updateEntites: {
      type: 'handleRecordChange'
      event: 'RECORD_CHANGED',
      blockers: [
        {
          id: 'tail',
          type: 'listener'
        }
      ]
    },
    syncChanges: {
      type: 'handleEntityChange'
      event: 'ENTITY_EHANGED',
      blockers: [
        {
          id: 'tail',
          type: 'listener'
        },
        {
          id: 'updateEntities',
          type: 'handler'
        }
      ]
    }
  }
}