Skip to content

Instantly share code, notes, and snippets.

@lilac

lilac/fstore.md

Last active October 20, 2023 03:45
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lilac/c44eeea76cac90d84e903a9aa5248cd3 to your computer and use it in GitHub Desktop.
Save lilac/c44eeea76cac90d84e903a9aa5248cd3 to your computer and use it in GitHub Desktop.
Download ZIP
A database tailored for feature store
Raw
fstore.md

Build a persistent, high availability and low latency key-value storage engine for derived data. One of such use cases is feature store. These use cases share some common characteristics, that we could utililze to build a better performing database.

  1. The data is derived, so no need to handle replication or data recovery.
  2. All the writes are through bulk loading, so no concurrency control needed.

In summary, it's like a scalable cache engine.

Features

  • High throughput bulk load
  • Low latency (especially on read), target at < 10ms 99.99%
  • Handle big data (larger than RAM)
  • Horizontal scaling
  • Stable performance on heavy access load

Non features

  • Transactions

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment