Some thoughts on Microservices and databases
One of the recommended best practices for microservices is that each service have its own database - when practical. The goal is to reduce the potential impact that a single database migration change might have across a collection of services. Think of a relational database change, which isn't schema-less by nature, and how each depending service might be impacted prior to a production release. In complex systems identifying impact across subsystems becomes a project in and of itself.
By ensuring that each microservices has its own database, the impact of change may be isolated or at least kept at bay.
However, this does raise an import question and opportunity. At the level of a microservices, what is the right database to use? The reason this question is also an opportunity is because we can think about database requirements in the context of a what an individual or group microservices might require. This is something that would be virtually out of the question when relying on monolithic applications and data stores. However, for Microservices this option is back on the table.
Keep in mind that a database per service might not be practical for your organization and architectural requirements - however, the goal is to realize the benefits of isolating the impacts of change. After all, isn't that one of the core benefits of moving to microservices?
When it comes to data stores, in a purest sense, there really isn't a one size fits all - one that addresses all possible needs. And this remains true unless:
- you don't want to consider the right database for the job
- performance isn't a huge concern or at least one that can be addressed at higher absorbed costs
- you don't want the extra burden of learning new technologies.
The right tool for the job - a concept most of us firmly believe in, but don't often practice.
There's no free lunch right? That's the understanding that everything has an associated cost - at some level. So naturally, the cost of having many, production ready, database environments - can quickly become prohibitive. After all, each database instance would have to be replicated. When it comes to this, it's perhaps time to take a closer look at your true data requirements:
- how much of your data needs is transactional and needs to be joined in relational queries?
- can some data be stored in NoSQL document data stores?
- would some of the data need to be in permanent in-memory caches?
Understanding your data requirements can lead to the following benefits:
- lower physical costs - smaller database tiers - through decentralization.
- lower maintanance costs through less expensive database operational cost such as those relating to schema migration time and effort.
- increased throughput as some data would be faster to fetch.
Measure twice, cut once. Take the time to consider the data requirements of your services, coupled with the performance characteristics you owe your customers.
At a time when relational databases reigned supreme and big iron roamed the earth, the choice of database types was seemingly clear. Map your data to a set of relationships and call it a day.
Today we have lots of options. Let's consider a few which Amazon, a leader in cloud based infrastructure, has to offer:
| Database | Type | Description |
|---|---|---|
| RDS | Relational | Amazon RDS (Relational Database Service) makes it easy to set up, operate, and scale a MySQL, Oracle, SQL Server, or PostgreSQL database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks. |
| DynamoDB | NoSQL - Key/Value, Document | DynamoDB is a highly scalable, fully managed NoSQL database service. Its seamless throughput and storage scaling, as well as automatic 3-way replication frees you from time-consuming database administration tasks and allows you to focus on your application and business. |
| Redis / ElasticCache | Key/Value | Amazon ElasticCache is a web service that makes it easy to deploy, operate, and scale an in-memory cache in the cloud. |
In the context of services, large relational databases can be trimmed down to smaller, more specialized databases. In some (and at times: many) cases the breakdown maps nicely to a related service.
A key factor to consider is data access patterns which will in turn help dictate information granularity.
For many services, having access to all the data isn't useful. Additionally, some data may not need to be heavily cross-queried. This is where document databases shine. Documents can be sub-queried, providing "views" into data. A good example is a user profile that might be accessed via a user profile service. Document databases still allow you to ask questions such as "let me see a list of customers in New York".
DynamoDB offers both key/value and document storage features.
In some cases having data in memory and accessible in the fastest possible time is absolutely necessary. Redis makes this possible and is ideal when data needs to be shared among a number of services. This allows high-speed access between services.
The goal of a database per service can be made practical by the use of cloud-based managed data services. Those services free you from the complexities inherent in backup, replication and scalability. Yes, you do still have to pay some attention to those concerns, but a reliable and performant solution is often a few clicks away. That's because managed solutions are designed for the eventuality of data growth and increased usage patterns. Naturally that presumes your company is actually growing!
Microservices and Managed data services are among the primary drivers for how a db per service might be realized. Otherwise the alternative might require a team of database administrators and system admins just to manage eventual change.