Sync V3 Infrastructure Abstraction
At its core, the system is event driven.
ex: "a record changed", "an Entity changed", or [future event type]
In response to events, we need to take different Actions. Also, Events don't just magically come into the system - we have to listen for them, and sometimes go out and get them.
[Source] --Emit Event--> [Handler] --Emit Action--> [Actor]
A sync process can be constructed from 3 different items
Processes that listen for (or actively find) conditions under which events should be triggered
ex: tail (a process that searches for updated records), webhooks (a listening process that triggers events on incoming reqs)
Sources are long-lived processes that are built to emit Events.
Processes can listen for incoming events (like webhooks) or go looking for new events (like the Tail process). It's not clear yet whether we'll put Sources onto EC2, Fargate, or some other service for long-lived processes.
The outputs of our system - actually do something in response to Events.
ex: "update an Entity", "write out to a third party system", or "write to the warehouse"
Calls for Actions are pushed into an SQS queue. Lambdas are triggered with the requests for work.
Before they're pushed into the SQS queue, a record of the call for action is record into DynamoDB. Once the task completes, it removes its own entry from DynamoDB. Any tasks that haven't marked themselves completed within 15 minutes (the max timeout of a Lambda) will be assumed failed and be re-added to the queue.
Pieces of code that wait for Events to happen. When something happens in the system that they care about, Event Listeners will trigger Actions based on what the customer's configuration (their Workflow) says should happen.
ex: "a record changed, go update the corresponding Entity" or "an Entity changed, sync the changes"
Events come into the system via SQS queue, triggering a generic Event Lambda. Handlers are units of code that are called by the generic Event Lambda depending on the type
of the Event.
basic blueprint: