How to design Horizontally scalable distributed systems?

designing_ha_systems.md

HA principles: CAP done right

• Design in isolation (Partition)
  • 2 business modules must communicate over message pattern. They are not considered business modules if one requires the other, in which case they don’t need to communicate over messages if a business process has to die, all interconnected business modules must die
  • process are stateless
  • fault tolerant implies scalability implies concurrency and concurrent business processes
• Availability
  • its better to die on fault and restart, then drag and build up the pile of error
  • when a process dies, fault detection and communicating fault is important
  • spawning millions of instances is not availability. to be available, service Cannot go down.
  • any of the system should be able to become a leader
  • Have hot reloading
• Consistency
  • having one concrete source of truth is not enough. the right leader election will guarantee how soon data is replicated.
  • Consistent replication using a consistent cord hashing is necessary
  • The right storage db will save you
  • Not defensive programming. use guards
  • use lightweight pre-emptive scheduling processes.
• fast message passing
• spawning millions of instances is not availability. to be available, any of the system should be able to become a leade
• No shared states betwen processes
• strictly functional
• dynamically typed
• No locks, Consider actor model for message passing
• Rethink The General Problem: SLOC vs YAGNI
• Data Structures behave differently in FP
• When leveraging dynamism or meta-programming, consider trade-offs with readability, ease of maintainability and from point 2, efficiency
• Domain driven logic
• Include benchmarking in CI!