Skip to content

Instantly share code, notes, and snippets.

@mzgoddard

mzgoddard/append-serializer-2.md

Last active May 15, 2018 08:52
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mzgoddard/b9250f152cfce76e587e77a996cf2ceb to your computer and use it in GitHub Desktop.
Save mzgoddard/b9250f152cfce76e587e77a996cf2ceb to your computer and use it in GitHub Desktop.
Download ZIP
Raw
append-serializer-2.md

cacache is far too slow as a read and write cache to disk solution for hard-source.

On one test project on an older 13" MBP:

@latest #cacache
initial build 28s 38s
iterative build 4.7s 11s

Instead of replacing, cacache will be an optional serializer like leveldb. Replacing append-serializer will fall to a new version of its idea. Instead of so finely trying to maximize use of append mode files, write out a log file once, with the meta info at the beginning and in a separate file additionally. Reading all the files in reverse builds up the prior state.

read

  • readdir
  • filter log\d+
  • reverse
  • for each
    • read log
    • select header items not yet read
    • read items
    • deleted items are in the header with 0 length

write

  • while items
    • concat items below "max" size
      • write log with header of items
    • if item is "max" size write as its own log entry
    • remove items from set to be written

sizes

  • read but filter log\d+-meta
  • total is length of all items ignoring if its already read

compact

  • when can i compact? determined by larger plugin?
  • read
  • delete
  • write
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment