letrec-notes.md

Rough notes as I read and use https://p.hagelb.org/letrec.html:

• I'm using luarocks for the first time today, but what's bad about it and what is letrec trying to fix?
• Dang, letrec should be named something else...
• The store should slurp in modules compiled as .so/.dll files as well? Need to search under package.cpath...
• Minor point, but all the built-in modules (table, string etc.) should be added to skip-module?.
• "-meta" files should contain more metadata...? Maybe something like:

   STORE/
   +  log-meta-SHA.lua
          { name: log
          , version: 0.1.2-1 -- ^ version number of the library that contained this module
          , filename: log.lua
          , data: log-lua-SHA
          , requires: { rings-meta-SHA, ... }
          }

   +  log-lua-SHA.lua
          ...

   +  rings-meta-SHA.lua
          { name: rings
          , version: 1.1.0-2
          , filename: rings.so
          , data: rings-so-SHA
          , requires: {}
          }

   +  rings-so-SHA
          ...

This would make the store more readable and debuggable. When there are tons of SHA.lua/SHA.so files in the store, it would be really handy to look at this sort of information. I have another suggestion regarding metadata files; all SHA's, for example the one returned by store(log.lua, meta) should point to a "-meta" file and not "log.lua" file. It would add a level of indirection, another hop before the actual module is fetched, but it would also make everything more flexible.

• In order to perform lookups, letrec would need a top-level table that maps package-name:package-version:... to the SHA of it's corresponding "-meta" file.

• letrec didn't fetch a transitive dependency, date, of the module log from lua-log(1). line 47 in containment.fnl suggests that parsing requires isn't implemented right now, right?

• Splitting large files into chunks:

  Since letrec will primarily deal with .lua files, this won't be too important, and it might not be a good idea either, but there's one difference between letrec's and other CAS'. Most content-adressed stores split up the data they store into small chunks, which helps them deal with large files efficiently. bup's design notes(2) explain it really well. To be clear, I'm not suggesting we do this for letrec, but just pointing out a difference. I was really happy when I learned about this chunking strategy, so I'll leave this here in case someone isn't familiar with it.

Thanks for the feedback! I've addressed some of it in the more recent commits, and that patch you sent helped.

In order to perform lookups, letrec would need a top-level table that maps package-name:package-version:... to the SHA of it's corresponding "-meta" file.

Initially I was thinking that you would use the module key as an entry point into the system and it's up to whoever is loading the module to look that up. But at some point it we will want a mapping from human-readable names to keys, but that mapping itself will need to be provided by the project rather than letrec. But letrec can provide tools to help construct such a map for the project to use.