Write-ahead logging for transactions in a bytespace.
Write operations are given a monotonically increasing id and written to transaction log. Batches are processed asynchronously and added to the store. By default, query methods withheld until batch processing is complete for the commit which was current at the time the query is received.
The batch processing pipeline exposes extensions points to allow pre-commit hooks to do additional work before committing to the transaction log, and post-commit hooks to do additional work after a transaction. Work can be isolated to specific subspaces, allowing heavyweight analytical processing to proceed without blocking queries in hot-path or transaction-heavy spaces. Analytical work can be delegated to subprocesses or even remote machines. Work on multiple commits can be batched together, and monotonically increasing batch ids can be leveraged to allow reads on possibly stale data to be held, e.g. until the next "reporting" batch is completed.
This approach also allows potentially large chained batches to be created efficiently, without requiring the entire batch to fit in memory. Operations on a chained batch are written to a transient space, and moved atomically into the transaction log upon commit. When using bytewise over multilevel, this requires an additional database method to be exposed: batchRange. This is derived from createReadStream and chained batch, and allows a potentially large range to be copied or moved as a batch without copying contents over the network.
Batch logs need not exist in db storage directly. As batch logs would generally be processed sequentially, they could just as well be backed by any seekable media. They could be striped out to spinning disks and periodically log-rolled, with seek offsets kept to index the start of each batch. Each subspace can define its own batchlog configuration. For instance, related subspaces may be stored together, or certain space could be given specific TTL values. Subspaces consisting of potentially large blob records could be configured to bypass db storage entirely.
Any number of processing tasks may be bound to a given batch log. The log is a durable resource which tasks can tail from some specific seek point, possibly with some filter logic to avoid unnecessary network traffic. This type of API essentially creates a durable message bus. Generally, batches would be kept directly in the db with a short TTL (e.g. hot storage). Before being expunged, they can be streamed out to a seekable store with reasonable latency (e.g. warm storage), and/or to a more permanent archival location such as amazon glacier (e.g. cold storage).
Batches could be lost from warm storage (e.g. disk failure) or intentially removed, and automatically revived (albeit slowly) from cold storage on demand. For example, a task which was stopped perhaps months prior, upon restart, should be able to pick up and continue plugging along without missing a single commit. This may involve substantial hangs -- perhaps hours at a time while blocks of batches are retrieved from cold storage and either added back into warm storage or a scratch area (if warm storage doesn't want a given batch any longer). If a task is running too slowly it can be stopped, possibly tweaked and tested, and started again with more concurrency.
Recent data should be available with extremely low latency if it already lives in the db (e.g. within its TTL). This latency can be improved further, dialed back to approximately zero by using post-commit hooks, or can even be made negative with pre-commit hooks (although necessarily blocks the write loop, so is best avoided when possible). Starting processing tasks on older batches may not be as performant initially, but once streaming begins it ought to be about as efficient as processing fresh data. Nothing about this design requires an explicit queue that data gets popped from and subsequently lost into the ether. More than just having a single coherent API into this kind of overall architecture, this buys you the flexibility to change you mind later, and go back in time to reprocess your data with hindsight.
Batch data written as invertible operations offers some additional power. Batch operations could also be imbued with invertible context after the fact, to avoid blocking writes. ...