In this architecture, microservices boundaries map to Domain Aggregates that encapsulate a given transaction within their boundary. Domain Aggregates are the collection of domain objects that must be read and/or updated within the boundary of a single database trasaction (to guarantee the read/write consistency for the involved etities, under concurrency.)
A domain model contains clusters of different data entities and processes that can control a significant area of functionality, such as order fulfilment or inventory. A more fine-grained Domain Driven Design (DDD) unit is the Aggregate, which describes a cluster or group of entities and behaviors that can be treated as a cohesive unit from a transactional perspective.
We usually define an aggregate based on the transactions that we need. A classic example is an order that also contains a list of order items. An order item will usually be an entity. But it will be a child entity within the order aggregate, which will also contain the order entity as its root entity, typically called an aggregate root.
An aggregate is a group of objects that must be consistent together, but we cannot just pick a group of objects and label them an aggregate. We must start with a domain concept and think about the entities that are used in the most common transactions related to that concept. Those entities that need to be transactionally consistent are what forms an aggregate. Thinking about transaction operations is the best way to identify aggregates.
An aggregate is composed of at least one entity: the aggregate root, also called root entity or primary entity. Additionally, it can have multiple child entities and value objects, with all entities and objects working together to implement required behavior and transactions.
The purpose of an aggregate root is to ensure the consistency of the aggregate; it should be the only entry point for updates to the aggregate through methods or operations in the aggregate root class. We should make changes to entities within the aggregate only via the aggregate root. It is the aggregate’s consistency guardian, taking into account all the invariants and consistency rules you might need to comply with in your aggregate. If you change a child entity or value object independently, the aggregate root cannot ensure that the aggregate is in a valid state. It would be like a table with a loose leg. Maintaining consistency is the main purpose of the aggregate root.
In the figure below, we can see sample aggregates like the buyer aggregate, which contains a single entity (the aggregate root Buyer). The order aggregate contains multiple entities and a value object.
Given that our Microservices boundaries are defined based on Domain Aggregates Pattern (as described above), the architecture is already highly decoupled and lends itself to asynchronous, transactional, persistent messaging for driving business process flows.
Such an architecture is optimized for scale, performance, and fault tolerance. The following are its operating tenets and benefits:
-
All writes (from clients and services) are sent as events via the Kafka Streams API and are consumed with 'back pressure' support by the subscribed services, which then update the persistence layer. The client receives an ACK right away and releases the connection. The response from the subscribed service and any subsequent change to a any of the Kafka connected data stores is sent via the Kafka Streams API so the client(s), e.g. UI, can then update. This provides for increased reliability and scalability compared to the tightly coupled request/response model.
-
All reads (aside from reads that are made in the course of an update) are made directly from client (via cache layer) to the services where the services are connected to read-only DB replicas, outside the transaction path. This assures the reads are not slowed down by heavy transactional loads, and given the use of realtime, ordered change notifications via the Kafka Streams API, the UI is able to update to the latest consistent app state, without transactions blocking reads.
-
To aggregate/filter data for display/analytics purposes across domain aggregates we use GraphQL allowing us to have the desired view of the data in the UI without having to write ad-hoc service APIs. This was PoC'd separately.
-
Services are modeled around Domain Aggregates that encapsulate a given transaction within themselves, which means that they do not invoke any other services directly. This avoids two common problems in event-driven microservices architecture:
- (No) Implicit control/data flow (i.e. flow that's not captured in code)
- (No) Distributed transactions that would require 2PC (which may not be possible across different data stores)
- Clients (incl. UI) may coordinate services to implement a given business process flow with each service/transaction as a node in the process, using transactional messaging and service response/db change notifications to drive the flow. Kafka's Transactional Messaging (via Streams API) is ideal for coordinating business process flows at scale, based on the following features:
- Guaranteed order of events in stream (per partition)
- Guaranteed "Exactly Once" processing of events in stream
- Support for atomic batching of events in streams across multiple topics and partitions
- Distributed, Externally Consistent Append-Only Log (client keeps index of last message and can fetch at any time, within the retention window)
-
Fault tolerance is easily achieved in this scenario as crashed service instances can restart without missing any write events.
-
Zero downtown when upgrading clients, services and data stores by having the new services take over the existing topics after data store migration. Requests from different versions of the given client can be sent to different versions of the Kafka topic so the ones from updated and existing clients will be persisted in Kafka and handled by the new service as it comes online to enable hot swapping of services, including after database migration. If there is no database migration then two or more versions of the same service can operate concurrently and each listen to a different version of the Kafka topic, e.g. during A/B testing when/if services need to be modified.