Dependencies in @scriptableapp

README.md

Managment of external dependencies in @scriptableapp

When writing scripts in @scriptableapp it would be great to be able to include external dependencies. Having to include e.g. a JSX-like layout library in each script is hard to maintain.

Solution

I would propose to expand the functionality of importModule (or introduce a new function) in @scriptableapp.
Inspired by the dependency managment of Deno it would be great to be able to pass a URL to importModule and have that script downloaded and cached.

Providing such a function would open up the whole npm ecosystem via a sites like https://unpkg.com/.

I build a proof-of-concept in @scriptableapp to show how this could be implemented.

Proof Of Concept

This PoC isn't using the building importModule as it is not usable inside of an imported module. It is working by using the new AsyncFunction() constructor to wrap the downloaded code in an async function and executing it. By passing down a new module object and importModuleWeb it allows for nested dependency graphs.

This approach also lends itself tp caching the resulting module, mirroring the node-require behaviour.

async function importModuleWeb(url, { forceUpdate } = { forceUpdate : false }) {
  // Cache this in globalThis.__import__ as we can't declare global variables 
  const { AsyncFunction, cache } = globalThis.__import__ || {
    AsyncFunction : Object.getPrototypeOf(async function(){}).constructor,
    cache: new Map()
  }
  // Name of the modules cache folder
  const MODULES_FOLDER = 'sb_modules';

  // Oversimplified isUrl check
  if (url.includes('://')) {
    const fm = FileManager.local();
    const moduleCacheName = getCacheFileName(url);
    const modulesFolderPath = prepareModuleCache(fm);
    const modulePath = fm.joinPath(modulesFolderPath, moduleCacheName);

    // Don't re-evaluate modules that are cached
    if (cache.has(moduleCacheName)) {
      return cache.get(moduleCacheName);
    }

    let script;
    // Download file if it is forced or not yet cached 
    if (forceUpdate || !fm.fileExists(modulePath)) {
      script = await download(url);
      fm.writeString(modulePath, script);
    } else {
      script = fm.readString(modulePath);
    }

    // Build new AsyncFunction and evaluate it 
    const fn = new AsyncFunction('module', 'importModuleWeb', script);
    const module = { exports: {}, filename: modulePath }; // module-API

    // Execute user code 
    await fn(module, importModuleWeb);

    // Cache the result;
    cache.set(moduleCacheName, module.exports);

    return module.exports;
  } else {
    throw new Error('Only URLs are supported')
  }


  async function download(url) {
    const rq = new Request(url);
    return await rq.loadString();
  }

  // Replaces "://", "/" and "." (except the last one) with "_" to use as a unique cache key. 
  function getCacheFileName(url) {
    const fileName = url.replace(/:\/\/|\.(?=[\s\S]*\.)|\//g, '_');
    return sanitizeFileName(fileName);
  }
  
  // Create the cache folder 
  function prepareModuleCache(fm) {
    const modulesPath = fm.joinPath(fm.cacheDirectory(), MODULES_FOLDER)
    !fm.isDirectory(modulesPath) &&  fm.createDirectory(modulesPath);
    return modulesPath;
  }

  // Adopted from https://github.com/parshap/node-sanitize-filename/blob/master/index.js
  // Just to be 100% save with the filename
  function sanitizeFileName(input, replacement = '') {
    const illegalRe = /[\/\?<>\\:\*\|"]/g;
    const controlRe = /[\x00-\x1f\x80-\x9f]/g;
    const reservedRe = /^\.+$/;

    var sanitized = input
      .replace(illegalRe, replacement)
      .replace(controlRe, replacement)
      .replace(reservedRe, replacement)
    return sanitized;
  }
}

Example Usage

Consumer

To use my JSX-like layout library you coud do:

const { h, render } = await importModuleWeb(
  'https://gist.githubusercontent.com/FWeinb/84e981aed913c12afb07011e670e52fa/raw/343e9705da4e3b8ab05d535d502ba2f5d24ee8f8/jsx-layout.js',
)

// use render here

Library

My JSX-like layout library could then directly depend on htm using https://unpkg.com/ without coping the source into a widget.

const htm = await importModuleWeb(
  'https://unpkg.com/htm@3.0.4/mini/index.js'
)

module.exports.render = render; // Omitted in this example 
module.exports.h = htm.bind(function(n, t, ...e) {
  return t = t || {}, e = e || [], "function" == typeof n ? (e = e.reverse(), n({
      ...t,
      children: e
  })) : {
      type: n,
      attrs: t,
      children: e
  }
});