beaker-dat-modules-proposal.md

Beaker/Dat code modules proposal

Like node and other platforms, Beaker will need a way to share modules of code and other assets. The Web's ES Modules give us a lot of the tooling for importing and exporting javascript, but we still need solutions for publishing, versioning, and "installing" dependencies.

This is a proposal to use Dat archives and ES Modules in Beaker to solve code-sharing.

Overview

"Module" dat archives have a folder structure that includes the current development code in /dev and any frozen versions in /v. A trivial example of that folder structure:

/dat.json
/dev/index.js
/v/1.0.0/index.js
/v/1.0.1/index.js

This predictable folder structure makes it easier for developers to import from module dats. An import can be accomplished in Javascript like this:

import { sayHello } from 'dat://trivial-module.com/v/1.0.1/index.js';

To provide integrity-checks and simplify version management, the importing site can specify a vendor.json which is similar to node's package.json:

{
  "modules": {
    "trivial-module.com": "^1.0.0"
  }
}

This is read by developer tooling and used to produce a /vendor directory which mounts the target modules at a pinned & integrity-checked version. This then enables JS imports as follows:

import { sayHello } from '/vendor/trivial-module.com/index.js';

When Import Maps land in Chromium, the tooling will be able to automatically generate an import-map for the vendor directory:

{
  "imports": {
    "trivial-module.com": "/vendor/trivial-module.com/index.js"
  }
}

Which will then enable Javascript imports that look like this:

import { sayHello } from 'trivial-module.com';

Useful prior knowledge

• Every dat archive includes a manifest file (dat.json). The manifest includes a "type" array which declares what the dat includes. The types can be any string. We frequently use URLs as types, but in this case I'm going to propose a non-URL token of "module."
• All dat archives are built upon a verifiable append-only log called hypercore. The log includes a reference to every revision to the archive, and so it's possible to reference dat archives at specific revisions in history. (That is, they're versioned by the protocol.) Versions can also include a content-hash to verify integrity.
• The upcoming Dat protocol release will add "mounts." Mounts are similar to symlinks and git submodules. They enable a dat archive to be placed as a subfolder within another dat archive. Mounts can optionally be "pinned" to a specific version.

Proposal

Module Archives (Publishing)

• Modules will be published in "Module" dat archives.
  • Modules will have the "module" type.
  • As with all dat archives, modules will have a title and description in the dat.json.
  • As with all dat archives, modules can be given a human-readable URL with DNS. (E.g. dat://jquery.com/)
• Modules will include a current "development" artifact and any number of "published version" artifacts.
  • All "published versions" will have a Semantic Version identifier.
• Module content will be placed in an expected folder structure:
  • The /dev folder will contain the current "development" artifact.
  • The /v folder will contain the "published version" artifacts with each version contained in a subfolder.
    • The pattern is /v/{semver} for each published version.
    • For example, version 1.0.0 of a module is written at /v/1.0.0.
• The flow for publishing a new version is as follows:
  • The user requests a new version be published. They provide the desired {semver}.
  • If /v/{semver} already exists, the flow aborts.
  • The /dev folder is copied to /v/{semver}.

A trivial example module with 2 published versions would have the following file structure:

/dat.json
/dev/index.js
/v/1.0.0/index.js
/v/1.0.1/index.js

Importing & Usage

It is possible for a module archive to be used directly via an import in Javascript:

import { sayHello } from 'dat://trivial-module.com/v/1.0.1/index.js';

This suffers from two issues:

1. This does a poor job of managing the import across multiple files. For example: If you decide to upgrade from 1.0.1 to 2.0.0 at some point, you will need to update each import statement in your project.
2. This provides no content-integrity guarantees. Ideally a hash-sum would be provided to ensure the import has not changed from its original version.

This proposal includes the following solution for importing:

• The importing dat archive includes a vendor.json file.
  • It includes the "modules" attribute which specifies the addresses of modules to be imported.
  • The "modules" attribute is an object which maps the dat address to a semver range (similar to the package.json "dependencies" attribute).
• The importing dat archive includes a /vendor directory.
  • All imported modules are mounted within this directory.
  • Mounts point to the subfolder of the target version. If 1.0.1 is requested, then the mount will point to the /v/1.0.1 subfolder.
  • Mounts are "pinned" to a version and include a hash-sum to verify integrity.
• The flow for installing imports is as follows:
  • For each module specified in vendor.json:
    • The file listing is fetched.
    • A folder matching the semver-range specified in vendor.json is located in /v/*.
      • If no match is found, abort with error.
    • The module is mounted in /vendor using the domain name as a subfolder. The mount target will be the matching /v/* folder and will include the current revision and content-hash of the module dat archive.
      • If a mount already exists in /vendor under the given name, it is overwritten.

A trivial example of a vendor.json is as follows:

{
  "modules": {
    "trivial-module.com": "^1.0.0"
  }
}

After running the "install flow", the file structure would include the following items:

/vendor.json
/vendor/trivial-module.com -> dat://trivial-module.com+{version}+{hash}/v/1.0.1/

The module archive would now be imported using the following statement in Javascript:

import { sayHello } from '/vendor/trivial-module.com/index.js';

Future Possibility: Import Maps

The import statement from /vendor above is still relatively lengthy. When Import Maps arrive, it will be possible to generate a vendor.importmap automatically which would then enable the following Javascript:

import { sayHello } from 'trivial-module.com';

The import map would look something like this:

{
  "imports": {
    "trivial-module.com": "/vendor/trivial-module.com/index.js"
  }
}

Alternatives: No /vendor directory

It would be possible to skip the /vendor folder and its mounts and use import-maps alone. In this case, the import-map would specify a "strong link" URL (a dat URL which includes a version and hash-sum). It would look something like this:

{
  "imports": {
    "trivial-module.com": "dat://trivial-module.com+152+9107c8778061f463889aaef94cf312872cea203194075a8c7050673fc57e7094/v/1.0.1/index.js"
  }
}

The advantage of this approach is that it's mechanically simpler (no /vendor directory with its mounts). The disadvantage of this approach is that it loses the benefits of mounts (mounts improve load time by bundling data across connections).