Ideas for Perl 6 versioned module management

design-outline.txt

The following specifies a possible design for in-filesystem storage of Perl 6 modules with associated metadata necessary to satisfy the S11 goals as well as hopefully being reasonably efficient for loading modules at runtime, without involving binary databases, and with reasonable capabilities for recovering from corrupted state.

BASIC LAYOUT
------------
* Modules are stored (as now) beneath one of the four roots specified in %*CUSTOM_LIB
* These are searched (as now) for candidates in the order: < home site vendor perl >
* Within each lib root, files are stored in the form: Module/Name/auth/ver.pm

WILDCARD AUTH OR VER
--------------------
* In each directory of auth directories, there is one special auth named 'Any'; this is unique because it is not in the proper auth format
  + The 'Any' auth directory contains all ver.pm's in all other auth directories, but with conflicts resolved according to the "first version lineage" rule
* In each auth directory containing ver.pm's, there is one special ver.pm named 'Any.pm'; again this is unique because it is not in the proper version format
  + The 'Any.pm' should be the most recent ver.pm for the default auth, chosen again according to the "first version lineage" rule

FILESYSTEM DETAILS
------------------
* On filesystems that have symlinks, shortcuts, or equivalent, the Any/ver.pm and .../Any.pm above should make use of these to minimize duplication.
* On filesystems that are case insensitive or otherwise cannot represent character differences between two module names that Perl 6 would consider different, *conflicting* files can instead be stored in the form: Module/Name/auth/ver/id.pm, where id is a unique identifier per conflicting file for this filesystem-mangled Module/Name/auth/ver path
  + These must be resolved using the install state/manifest file for this lib root; see below

STATE/MANIFEST FILES
--------------------
* Each lib root has a matching state/manifest file, containing info about all installed modules in that lib root
* This state/manifest file is managed by the module install system
* System packagers are expected to trigger updates of the state/manifest file *using the module install system* during post-install and pre-remove phases
* This state/manifest file contains in a fast-to-parse format the module name, auth, ver, and filename relative to lib root of all installed modules, plus any additional metadata available during module install time
  + As a first cut, perhaps a tab-separated file, one line per fully-specified (no "Any" parts) filename on disk, with fields as follows:
    - unmangled module name, auth, ver, relative filename as plain strings; this should be safe because tabs, newlines, and nulls are not allowed in these fields
    - remaining metadata encoded as a JSON object
* At install time, the module install system must add rows to this file, as well as update the Any/ver.pm and Any.pm links on the filesystem as appropriate
* At uninstall time, the module install system must remove rows from this file, and recompute the Any/ver.pm and Any.pm links on the filesystem as appropriate
  + If there are no remaining auth/ver.pm files for a given Module/Name/, then the entire Module/Name/ tree should be removed to avoid accumulating clutter
* It should be possible to ask the module install system to regenerate or reconcile the state/manifest file (and Any links, if needed) from the on-disk Module/Name/auth/ver.pm files in case of corruption

MODULE LOAD RESOLUTION
----------------------
1. If a full filename is requested (ending in .pm), then it is searched for in the lib roots in order
   * 'Any' is explicitly allowed for the auth and/or ver portions of the filename, and will be handled by the appropriate links
   * If no .../ver.pm is found, but instead a .../ver/ directory is found, consult the state/manifest file to pick a .../ver/id.pm file
   * If still no match is found, fail
2. If a Module::Name is provided, it is converted to Module/Name/Any/Any.pm, and the file searched as in (1)
3. If an exact ver and/or auth are provided along with a Module::Name, then these are substituted for the appropriate 'Any' in Module/Name/Any/Any.pm, and the filesystem searched as in (1)
4. If a wildcard ver and/or auth are provided, or the filesystem contains a .../ver/ directory where a .../ver.pm file is expected, then consult the state/manifest files as follows:
   * Load and parse the state/manifest files for each lib root that contained a matching Module/Name tree, in the usual order
   * Use the best-first match for the module name, auth, and ver provided (i.e. exact match if found, best match if no exact matches, first match if multiple "best" matches)
   * If an exact match is found in an early state/manifest file, the process can be short-circuited and further state/manifest files not loaded
   * During the initial application load time, or any time that the Perl runtime can determine that a state/manifest file has not changed, Perl may cache the state/manifest file parse results and reuse them when loading additional modules