Note:
- This is something to consider and discuss if we should do it now or later, and I would love a healthy debate about the pros and cons of doing it now or later.
Context:
- We are going to turn on reading and writing both from MongoDB.
- I believe that it is going to increase the load on our MongoDB Atlas clusters
- Increased load means we may be required to use the large cluster size.
- The large size of the cluster means higher costs.
- The experiment aims to validate if MongoDB is a cost-effective & performant event store for the long run.
Problem:
- we only read and write to Primary and never use Secondaries.
- To spread the load evenly across MongoDB Atlas clusters and use it effectively, the MongoDB driver provides options to enable reading from the Secondary. https://www.mongodb.com/docs/manual/core/read-preference-use-cases/
- However, when enabling read-preference secondary, we might encounter eventual consistency problems as some of the use cases, like coins/stars, rely on instantaneous consistency.
- We can enable MongoDB to write-acknowledge to all three replicas before confirming "write operation", but it might impact performance.
Experiment Idea:
- Adapt our connection strings & settings so we can measure what happens when we enable read preference to Secondary and guarantee consistent writes.
- That way, we can find the balance of costs vs performance gain. The experiment could mean that if we enable reading on secondary, we might not need a big cluster, but we might sacrifice performance.
- As we are also switching from DynamoDB to MongoDB entirely, it is hard to predict the net outcome.