Being apart of several different architecture teams throughout my career, I have learned alot about data organization from the core data to the frontend interfaces to everything in-between. Generally, every single architecture survives off what I am coining as the Five Star Architecture (FSA). The FSA essentially encompasses five different tiers, each with their specific resposibilities and methodologies.
The following is a high-level diagram of the tiers including the communication protocol(s):
Client Tier [ Mobile Clients ] [ Web Browser Clients ] [ Partners ]
APIs HTML REST
| | |
\ v /
BFF Tier ----> [ Backend for Frontends (BFF) ] <--
GraphQL, REST
|
v
Federation Tier [ Federated Composition, Registry and Routing Tier ]
GraphQL, gRPC
----------/ | \---------------
| | |
v v v
Core Tier [ Core Service 1 ] <---> [ Core Service 2 ] <---> [ Core Service 3 ]
gRPC, DB gRPC, DB gRPC, DB
\ /
Data Tier ---> [ Database 1 ] [ Database 2 ] <----
The client tier is made up of the various clients that interact with the business. Predominantly, this is composed of the mobile clients and web interfaces that make up a product. However, it may also include partner feeds that interface with the business.
The mobile clients will generally use REST-based APIs to deliver view-focused data from the BFFs. The BFFs are client focused and provide view-specific data. This is important since this data will often be easily exposed in the applications, enabling reverse engineering. However, because these APIs target very specific needs, the APIs themselves are not bound to backwards compatibility so the dependability of reverse engineering will be less reliable. It is, however, critical that these APIs employ versioning since varying levels of mobile client versions will exist.
The web clients are based on browsers and will generally be HTML focused with Javascript frontend applications. These clients may or may not interface with a BFF. In some cases, the web server will be the BFF itself but will translate the API data into view data via HTML. In other cases, the web server may provide view-based APIs as a true BFF using a complete Javascript application on the frontend.
The BFF tier is where the view logic of an application lives. The BFF tier is responsible for exposing business data in view specific ways. This helps to minimize the necessity for mobile clients to translate data. It also helps to reduce the simplicity of reverse engineering APIs since the APIs are likely to, and often should, change. The BFF tier should support retrieving data from core systems, often through the use of a Federation tier. This helps to reduce the effort on the BFF of knowing where data lives or the need to understand relationships to compose multiple sources of data. Instead, the BFF should be able to purely fetch all the data it needs in order to translate that data to view-specific models.
BFFs, by principle, are often targeted at specific device types. In some cases these may be shared where the clients are highly similar such as Android and iOS. In other cases, they should be highly separate such as Web vs Mobile vs Set Top. View data is often rarely shared so the need for a common BFF is unlikely. Instead, business data that is shared should act as its own core system and provide that data.
The Federation tier is critically important to the entire data flow. The Federation tier is essentially the routing
system of all data. This tier may be a full service providing rich composition of data or may be more virtual system
offering access points to or a registry of data. The particular type of Federation depends on the business and backend
systems. The most important point is that it provides a way for BFFs to access any data from any system without having
to know specifics. This allows the frontend systems to change and evolve as well as the backend systems all while keeping
everything in sync. The next important factor is to provide data relationships and even data composition. This allows cores
to be purpose driven as a set of microservices while still having relationships to other data. The Federation tier helps
to align those relationships and expose them simply to the BFF. The final important factor for Federation is tracking data.
It is critical to know and understand how data is used and accessed from top-to-bottom. This allows data evolution to occur
safely by always knowing access patterns. By managing the flow of data from BFF to cores, Federation is able to keep everything
in sync and understand those patterns.
One of the most popular forms of Federation is GraphQL, in paticular the Federation platform by Apollo. GraphQL Federation is able to composite multiple backend systems and define relationships across those systems to define a complete data schema. This allows BFFs to access any data across the entire system without having to worry about where that data exists or how the relationships are managed. Further, it allows the core systems to evolve without impacting relationships. The backend may start with 1 monolith and evolve into several microservices, or vice-versa.
Businesses, may instead elect to build their own Federation systems. For example, the company could build a custom DSL that allows the BFF to fetch all data and all relationships and use a central gRPC registry that is able to automatically fetch the data and relationships behind the scenes. Thus, core services would provide their gRPC schemas along with relationships to the central repository. The federated router would then take a BFF query and invoke the necessary cores. This, once again, allows the gateway to track the access patterns and manage the internal routing keep clients agnostic and allowing cores to evolve independently.
It is highly recommended that the Federation tier be very lightweight with little to no logic. This allows it to focus purely on data access, data composition and data routing. Logic, instead, should live within the BFFs (view logic) or the core data systems (business and data logic).
The Core tier is essentially where all business logic lives. However, it is important to note that view logic does not live here. The goal of the core tier is to expose business data, not product data. In other words, core services should be product agnostic, but also product enabling. This means that the core services should think about how to expose data in a meaningful way without being tied to specific products. This allows the cores to evolve independently of products while allowing future projects to easily onboard.
The core services should interface with the Federation system as designed by that particular system. This may be pure GraphQL, or it may be gRPC, or it may be simple RESTful APIs. Regardless, these data models and access rules should be published to a central schema registry. That registry may be built into the gateway itself or it may be something more static such as a Nexus repository or even a Swagger repository.
The core services should also interface to the particular data store(s) they are associated with. It is important to remember that core services are not purely CRUD APIs on top of a data store. Core services are meant to empower data access. Thus, while CRUD may be one facet of a core service, it is often desirable to have more elaborate queries on top of the data. For example, a user service might have simple CRUD access patterns, but it is likely those same users will need to support filtering, grouping, and other query mechanisms. These allow the work to happen at the most performant tier and simplifies the higher tiers that are then able to focus on the view rules.
In some cases, core tiers will need to talk to other core tiers. This should generally be avoided in favor of maintaining a strict top-to-bottom data flow. However, it is likely these cases will arise, especially in order to empower data access and create more dynamic data relationships. In this capacity, services should talk to each other through IPC. Often times this will be gRPC, but in other cases it may be RESTful APIs. It may seem advantageous to communicate back up the stack through Federation. However, that can quickly run into circular dependencies as well as chicken and the egg data issues. For this reason it is important to keep traffic at the same level as peer relationships. Services may, and probably should, still use and share registries such as gRPC, Swagger, GraphQL, etc. However, the goal of Federation is to define data relationships and composition for BFF consumption. The goal of cores is to fetch additional data, not relationships.
The data tier is the underlying data storage. This includes common data stores such as SQL, key-value stores, document stores, etc. The data tier should be organized in a business-centric manner focused on the specific data needs. This data will be queried and managed by the core tiers that utilize it. Generally, it is recommended that the number of cores that interact with a data store should be minimal, often a single store. This reduces the number of dependencies that would be impacted if the data store were evolved. It is much easier to evolve services than data stores. For that reason, it is advisable to have a single store and then other systems interact with those cores directly as peer relationships.