ES6 module proposal

modules.markdown

Modules Proposal

EDIT There is a second version of this proposal down in the comments. https://gist.github.com/445adc206d51a038de4a#gistcomment-359858

I really like Isaac's module proposal. Having recently designed a module system for the luvit project (node ported to lua), I've thought a bit about this.

I think we can set a few minimal rules and reduce a ton of edge cases. One in particular is cyclic dependencies. I don't think they are worth the pain and should simply be disallowed.

To help understand things, we should make a distinction between dependencies and peers. In a large application or package there are several files containing code. Some files depend on things defined in other files before they can start. Simple dependencies that block startup can't be cyclic or you'll have a deadlock at startup. So the other type of dependency is wanting to reference an api or function from another file during runtime.

The basic syntax I propose is like the one Isaac showed (if I understand it correctly). There are two new keywords, import and export. The import keyword must be followed by a string literal that is the identifier for the file you want to import. The export keyword is like return and yield in that it sends a value. It does not affect control-flow, but it does set the current file's export value.

Consider lib/add.js:

// lib/add.js
function add(a, b) {
  return a + b;
}
export add;

And then we have a peer file that depends on this:

// lib/usesAdd.js
var add = import "./add.js";
var result = add(2, 3);

Now suppose we wanted to write a math package that contains add as well as other functions.

// math.js
export {
  add: import "./lib/add.js",
  multiply: import "./lib/multiply.js",
  // ...
};

Here is where this gets tricky. Suppose that we want to implement multiplication using addition and a loop. So our multiplication library will need to depend on add. I can't depend on the main app because that depends on it. So it has to depend on add directly.

// lib/multiply.js
var add = import "./add.js";
function multiply(a, b);
  var product = 0;
  for (var i = 0; i < b; i++) {
    product = add(product, a);
  }
  return product;
}

But not we learn later on that add needs to access multiply for some reason. This happens all the time in real applications. How would this be required?

I really want dependencies with import and export to be expressible as a DAG (Directed Acyclic Graph). There is a clear order for loading the executing the files. There are two reasons for this. One, it makes a lot of nasty edge cases with current module systems simply go away. And second, JavaScript is single threaded. Even with the exports.foo = bar syntax in node and CommonJS, the dependencies have to be executed in some serial order. This may surprise people when you're allowed to require some module, but it's methods aren't populated yet because you were executed before it was. This would be a leaky abstraction.

So then how would I solve this dilemma using acyclic dependencies? I would handle it at the application level where I control the logic and know what to expect.

I would rewrite math.js as follows:

// math.js
var math = {};
export math;
math.add = (import "./lib/add.js")(math);
math.multiply = (import "./lib/multiply.js")(math);

Then in each module I would export a function that accepts the math namespace.

// lib/add.js
export function (math) {
  return function (a, b) {
    // ... might use math.multiply
  };
};

There are several other techniques that come to mind that I could employ for other use cases. The point is that using the basic primitives that would be provided by the language, I can organize my app however I want. My code would use the same style in the browser or in node. The browser could pull my code on demand as it's needed or a server-side compiler could statically analyze the imports and generate a single js file that the browser then reads. All JS code everywhere would be written using the same simple module syntax, but without imposing huge constraints on the runtime or providing leaky abstractions to applications.

Can you maybe make a quick example of something fancier like how your proposal would work with jQuery 1.8 style modules if you have multiple things depending on the AJAX module for example. It sounds like you have to know a lot about the other modules to make it work in your proposal.

I think a DAG model would cause problems for things that lazy load modules. Plugin systems, configurable modules, etc. that are loaded via variables (ie. 'import "./plugins/"+option.pluginName') would need to reimplement the module system in order to viable, or all modules would need to be loaded every time a program runs. We use this pattern of variable requires quite often at setup-time to avoid bloating memory usage on constrained systems. If a means could be presented to avoid always loading all modules this would still get around the problem with a clever use of getter/setters.

One thing that I need clarified before commenting more, is what would you do to enforce no cyclic dependencies occur? Would it throw an error when the scripts are loaded but before running? When the dependency get imported while it is still loading?

@xjamundx, I'm afraid I don't know jQuery 1.8 style modules, how does that work? And yes, this proposed system doesn't solve everything for the programmer. It's a tool that helps them. It's better than what we have today. Incremental change is good in JavaScript because it's near impossible to remove anything or go back, ever.

Disallowing cyclic dependencies between modules is a mistake. At the level of deployed libraries, it makes sense not to have cycles, which you can enforce for example by dependency tracking in a package management system like NPM. But within an application or library, it's crucial to allow people to divide up the internal modules however they see fit. Disallowing cycles forces programmers to restructure their programs in painful ways.

You're making the programmer do the work for you in the name of simplifying the job of the language designer. That's the wrong way around. You should not force programmers to implement their own "tie-the-knot" pattern every time they have a cyclic dependency. I've been there in other languages, and it's painful.

Here's another way to look at it. A module system should stay out of your way as your program grows. It should impose a minimum of costs to splitting up your program into modules however you think is appropriate. When you take code in one module and split it into several, it should continue to work. JS already allows mutual recursion within a single scope, but it's initialized in a linear order. So typically if you have mutual recursion, you need to make sure not to invoke it until everything's been initialized. If you want to enforce an order of initialization, it's much simpler to just access the cyclic subgraph only through some single entry point. In your example, that would mean that other parts of the app would only access math. In the ES6 module system, add and multiply could be mutually recursive. The math module's import order would ensure that both add and multiply were initialized before math is initialized. There's still a deterministic order of initialization (and there's no deadlock problem).

If we ban cyclic dependencies, then when you want to pull apart mutually recursive code into separate modules, you're forced to use these painful patterns, and in practice you just end up avoiding it and keeping everything in one big module. In other words, the incentives prevent you from using modules. The language gets in your way.

Whereas, with cyclic dependencies, you can simply separate your app into separate modules without having to worry about whether the language will yell at you and force you to restructure everything. And if you want to ensure that they're loaded in a specific order, you can easily use a facade module like math. This is lighter weight than restructuring your modules and all their dependencies.

@bmeck, Modules will not be available to import till they have exported their value. Maybe I should reuse return instead of export so that it forces control-flow to end at export and have sane run-to-finish semantics. Then cycles simply cannot occour. If A imports B and then B imports A, A will still be pending and thus not available. I do this in my luvit project and it has worked well so far (though I do allow dynamic require there since it's server-side only and sync I/O is ok).

I have several independent comments.

A: I like the direction. It is fair to say that with @izs’s proposed single-exported-value, you can still have cyclic dependencies and you still need to use unfinished exports objects, but the complexity of supporting cycles is moved out of the module system and into the modules. This is well aligned with the philosophy of making simple things easy while leaving complex things possible.

B: My second proposal to TC39, after a year of CommonJS in the wild, introduced the same (import "foo.js") statically-analyzable dependency syntax. @dougcrockford’s comment at the time was that it should be import("foo.js"), presumably since eval sets a precedent in JavaScript for operators masquerading as function calls. I concur.

C: With this particular solution, export could be return.

D: For what it’s worth, import could also just be require. That is, when JavaScript is normally evaluated, require("foo.js") would be a function call, but when compiled by the system module loader would be statically analyzable syntax.

C + D: With these two alterations, we get a graceful migration path, where we can write scripts that target both old and new evaluation contexts.

However, with or without @creationix’s amendments or my C and D amendments, I gather that @izs’s proposal still utterly eliminates the possibility of importing new syntax (macros), the door @dherman’s proposal is intended to leave open, although the proposal does not walk through said door. This leaves us in a bit of a pickle; most of us are not in a position to evaluate whether macros are worth the price and too easily dismiss their value.

It might be helpful for @dherman to provide a strawman for JavaScript macros in the context of his Harmony modules proposal so that we can see why this simplification would not work, and so we might evaluate whether it would be worth the extra complexity. I gather that naming individual exports is the important feature for macros.

E: Module blocks are an independent issue. My opinion is that they provide very little value. @dherman likes to draw small circles around bits of code in files. I prefer small files. It’s a value-call. Given that "module" is proposed to be a contextual keyword that would not interfere with my "module" variable names, I could happily ignore their existence, but I would be even happier knowing they don’t exist.

F: Bulk import * is an independent issue, a value-judgement too. My objection would mostly go away if we were limited to one "import *" at the top of a module, much like we are limited to single, prototypical inheritance. That way I would have a good clue of where to look next for my locally unbound variable, perhaps following a chain of "import *" calls, but never exploding into a search of the dependency tree with a resolution order to contend with (depth-first? breadth-first? C3 monotonic?). That said, I can and do live better without "import *" at all. An equivalent to "import *" was in my original proposal to CommonJS (include("foo.js"))—they talked me out of it—we’ve lived happily without it.

G: Losing the ability to early-detect unbound free variables is an independent issue.

@kriskowal I like your A,B,C, and D. Indeed it is a very nice migration path and we can still have @izs's runtime hooks for when it becomes part of the language.

@dherman, as @kriskowal explained so much better than I did. I understand that cyclic dependencies happen in real applications, but I don't believe that can properly be handled at the language level without leaking badly at the edges. The language should provide the tools to easily do this in user-space. Also, since the author has to explicitly create the unfinished objects themselves, the behavior will be visible and obvious if implemented in user-space.

Yeah, I have to agree with @dherman and @kriskowal here. I've worked with module systems that didn't handle cyclic deps well, and it's very painful once you get above the math.add/multiply level. It requires that the app developer has a lot of understanding of the module system internals.

@isaacs, I'm confused. I thought @kriskowal was against supporting cycles within the language as was your proposal (except for the mention of having a magic value that resolves in-place later)

...but the complexity of supporting cycles is moved out of the module system and into the modules. This is well aligned with the philosophy of making simple things easy while leaving complex things possible.

I'm not saying applications won't have cycles, I'm just saying that responsibility shouldn't rest in the module system of the language.

See also discussion on the Traceur issue Issue 104: Mutually recursive modules.
http://code.google.com/p/traceur-compiler/issues/detail?id=104

@creationix Well, in my half-baked proposal blog post, I left open the problem of how to handle cycles, pointing out just that it's not handled well today in node or rjs. I also said that having a special export keyword would perhaps make it easier (that is, possible) to solve this, but I don't know how, exactly.

Thanks for all the feedback everyone. Here is my updated proposal with @isaacs's ideas assimilated inline. I like this much better than my original proposal and think that some level of cyclic dependencies can be baked into the module system by using a caching system like Lua does.

Loader

Loader is a global provided by the runtime. It is not fully implemented by the language, but an interface implemented by things like node or browsers. This provides the runtimes a place to hook I/O for their module system and allows script authors access to do advanced things like dynamic requires or module concatenation.

Loader.preload

An object containing compiled and defined module functions.

Loader.loaded

An object containing cached results from Loader.preload. Modules are only run the first time they are required. Calls after that pull the cached value from here.

Loader.resolve(requestPath, callerPath, callback) -> fullPath

This function is implemented by the runtime to map require paths to fully qualified paths (still opaque identifiers as far as the language is concerned since JS has no concept of I/O).

Loader.fetch(fullPath, callback) -> contents

This functions is implemented by the runtime. It fetches the code contents and returns it in the callback.

Loader.require(fullPath)

Execute a module and return the result (or cached result). It must be already defined and fully loaded or this will throw.

Loader.define(fullPath, contents, callback)

Defined by the language. This will parse and compile the contents. It will recursivly call resolve, fetch, and define for any dependencies mentioned in the contents. The callback will be called when all is done, but the value Loader.preload[fullPath] will be set early on so some form of cyclic dependencies can be satisfied.

---

Here is some pseudo-code to show the in-language part of this interface:

var Loader = {};      // The main Loader namespace shared by language, runtime, and userspace code.
Loader.preload = {};  // Compiled modules will live here
Loader.loaded = {};   // The cached result of compiled modules will live here

// Stub function that should be implemented by the runtime.
Loader.resolve = function (requestPath, callerPath, callback) {
    throw new Error("Must be implemented by runtime.");
};

// Stub function that should be implemented by the runtime.
Loader.fetch = function (fullPath, callback) {
    throw new Error("Must be implemented by runtime.");
};

// The language will call this function for each require it finds in source code after resolving the path.
Loader.require = function (fullPath) {
    // Check for cached module values and reuse them
    if (fullpath in Loader.loaded) return Loader.loaded[fullPath];
    // Throw if the path is not in any table yet
    if (!(fullPath in Loader.preload)) throw new Error("Bad require");
    // Otherwise execute the module for the first time and cache the result.

    return Loader.loaded[fullPath] = Loader.preload[fullPath]();
};

// Compile and store a module.  It's dependencies will be scanned for,
// resolved, and loaded before calling the callback.
Loader.define = function (fullPath, contents, callback) {
    // Note the functions `compile`, `resolveDependencies`, and
    // `loadMissingDeps` are implemented elsewhere within the language itself.

    // callback is optional, define fallback behavior.
    if (!callback) {
        callback = function (err) { if (err) throw err; };
    }

    // compile will eval the code as well as extract dependencies into the
    // outarg variable `dependencies`.
    var dependencies = [];
    try {
        // The require statements will be replaced by calls to
        // Loader.require(fullpath) where they were require(shortPath).
        var fn = compile(fullPath, contents, dependencies);
    } catch (err) {
        // Send syntax errors to the callback.
        return callback(err);
    }
    // Store the compiled function early to allow for cyclic dependencies.
    Loader.preload[fullPath] = fn; 

    // Resolve dependencies and get unique values.
    // This is pseudo-code, but the resolved paths will be stored with the fn somehow.
    // If the fn is executed before it's fully baked then it will throw.
    resolveDependencies(dependencies, function (err, newDependencies) {
        if (err) return callback(err);
        // Load any missing modules using Loader.fetch() and Loader.define()
        loadMissingDeps(newDependencies, function (err) {
            if (err) return callback(err);
            // Now `Loader.preload` has definitions for everything that was in
            // dependencies.  We can now callback.
            callback();
        });
    });
}

The only keyword added will be require. This will be used by Loader.define to scan for dependencies as well as call Loader.require at runtime with the resolved paths that were found during define.

---

This proposal works well for node style runtimes, it works for on-demand browser runtimes (where each dependency is requested till enough is loaded to start executing) and pre-compilers can simply concatenate several Loader.define(fullPath, contents) calls to prime the IO caches and prevent round-trips.

Oh, and since the only syntax change is a new require() expression, this can easily be polyfilled in today's JavaScript. And I failed to mention that each file runs in it's own shadow copy of the real global like @isaacs proposed.

Note that this modified proposal is fairly close to what I have today in Luvit. I could implement it fully there as a testbed if that was helpful and there was a chance of this landing in JavaScript.

Let me just point out some social-interaction reality (despite my complete lack of expertise in that area ;-):

The ES module proposal exists and represents a piece of work by the ES team. They are not likely to throw it over for a gist proposal, no matter how thoughtful. Instead, any new proposal has to make contact with and be compared to the ES proposal.

Similarly CommonJS and RequireJS went through a very large number of iterations and a great number of efforts at unification. So any new proposal has to explain how it differs from these efforts. The most obvious way would be a reliance on first-class parsing; simply a 'better' API is not adequate. This experience also tells that that any proposal that fails to support asynchronous loading will fail. I think Tim's proposal for example includes these issues but it call it out and do the comparison.

(Just to be clear: I'm not pointing at Tim and I am trying to be encouraging: this effort is important!).

@johnjbarton I'm aware, thanks. This thread contains my 1/3 baked and 2/3 baked ideas. I'm still bouncing ideas. Only once I know what I want do I go through the trouble of trying to convince others and gain consensus. Get it right first, then worry about being right.

Also, while this is just a gist, it's the result of many years of working with this problem.

I'm willing to make a proper proposal and address everyones concerns through the proper channels if that's desired. I'm just not ready for that yet.

@bmeck. Dynamic require can be done in my second proposal. Here is a sample that loads all modules in a folder and stores their exports in a big object.

var controllerDir = __dirname + "/controllers";
var controllers = {};
fs.readdirSync(controllerDir).forEach(function (filename) {
  var fullPath = path.join(controllerDir, filename);
  var contents = fs.readFileSync(fullPath, "utf8");
  Loader.define(fullPath, contents); // I assume node's Loader.define will use sync I/O and so don't need a callback
  controllers[file] = Loader.require(fullPath);
});

This is fully integrated into the system and any requires within the controllers/* files will work as normal.

@dherman Here is an example of cyclic dependencies using my second proposal:

Here is even.js:

var odd = require('./odd.js');
return function even(n) {
  return n == 0 || odd(n - 1);
};

And odd.js:

var even = require('./even.js');
return function odd(n) {
  return n != 0 && even(n - 1);
};

And then I kick off the process by compiling this code:

// main.js
var odd = require('./odd.js');
console.log(odd(5));

• Before compiling we have a clean state. Loader.preload is {} and so is Loader.loaded.
• The runtime loads main.js and calls Loader.define("/main.js", ..., callback)
• Inside define, the text is parsed and a dependency on "./odd.js" is discovered.
• This compiled half-baked function is stored at Loader.preload["/main.js"]
• Loader.resolve("/main.js", "./odd.js", callback) is called which outputs "/odd.js".
• Loader.fetch("/odd.js", callback) gets the text for that module
• Then Loader.define("/odd.js", ..., callback) is called.
• Inside the text, it finds the "./even.js" dependency.
• This half-baked function is stored at Loader.preload["/odd.js"]
• Then "./even.js" is resolved, fetched, and Loader.define("/even.js", ..., callback) is called.
• When the code is parsed, the only dependency found it "/odd.js", but we already have an entry for that in the preload table, so we're done.
• The compiled function is stored in the preload table.
• The async stack unwinds through chained callbacks. At each level it fully bakes the existing functions so they know the resolved paths.
• Finally the callback for the initial Loader.define("./main.js", ..., callback) is called.
• The system then executes the initial code snippet.
• When require("./odd.js") is encountered, a call to Loader.require("/odd.js") is made since this function is now fully baked.
• This call will execute that module.
• While executing that module, the require("./even.js") will be evaluated which causes that module to be executed.
• "odd.js" never finished running and so Loader.loaded["/odd.js"] is still empty, so ....

Dang, I found a bug in my proposal. This could be solved using a Proxy object that passes through to the real export value once it's known, but I'm hoping for a more elegant solution.

Though if I required within the function bodies, it would resolve just fine. It's the cyclic dependencies before exporting/returning values that's problematic. And since the require's values are resolved to full identifiers at compile time, there is little overhead in calling require in the loop.

// even.js
return function even(n) {
  var odd = require('./odd.js');
  return n == 0 || odd(n - 1);
}

Simple things made simple and complex things made possible perhaps? Plus this has the added benefit that "odd.js" never gets executed if it's never needed.

@kriskowal said:
"E: Module blocks are an independent issue. My opinion is that they provide very little value. @dherman likes to draw small circles around bits of code in files. I prefer small files. It’s a value-call. Given that "module" is proposed to be a contextual keyword that would not interfere with my "module" variable names, I could happily ignore their existence, but I would be even happier knowing they don’t exist."

Personally I like one-file === one module; I don't like all the extra syntax. However I don't understand how we can create a scope without new syntax. Top-level file scope is already defined in JS: it's global scope.

@creationix says:
"Oh, and since the only syntax change is a new require() expression, this can easily be polyfilled in today's JavaScript."

I don't understand how this can work. Simply adding unconstrained "require()" would just be a hack along the lines of Burke's require.js scanner for require. What does a programmer think when the read:
var imgs;
if (retina) {
imgs = require("daHot.js");
} else {
imgs = require("elCheapo.js");
}
?

Beyond the dynamics, the fundamental way we resolve cyclic dependency is to create a two-pass or two-phase solution where one phase builds references filled in by the second one. The first phase can be based on declare-before-use and ordering of statements or declare-before-use and a syntax that implicitly orders (like JS today). How can unconstrained require() provide that "before-use" thing if it is executeable?

I would like to attempt to illustrate how @dherman’s proposal might handle mutual dependencies, in its defense. As @johnjbarton points out, Traceur hasn’t done this yet, so this is my speculation.

Mutually dependent modules would get transpiled into a “working set” with a shared scope. Imports and exports would be hoisted to this shared scope.

odd.js

import odd from "odd.js";
export function even(n) {
  return n == 0 || odd(n - 1);
};

even.js

import even from "even.js";
export function odd(n) {
  return n != 0 && even(n - 1);
};

main.js

import odd from "odd.js";
console.log(odd(5));

transpiled.js

// hoist even.js#even as even in shared scope
// hoist odd.js#odd as odd in shared scope
(function (even, odd) {
    // from even.js
    var evenFactoryCalled;
    function evenFactory() {
        // memoize
        if (evenFactoryCalled) {
            return;
        }
        evenFactoryCalled = true;
        oddFactory(); // call site of import
        even = function even(n) { // call site of export
          return n == 0 || odd(n - 1);
        };
    }
    // from odd.js
    var oddFactoryCalled;
    function oddFactory() {
        if (oddFactoryCalled) {
            return;
        }
        oddFactoryCalled = true;
        evenFactory();
        odd = function odd(n) {
          return n != 0 && even(n - 1);
        };
    }
    // from main.js
    var mainFactoryCalled;
    function mainFactory() {
        if (mainFactoryCalled) {
            return;
        }
        mainFactoryCalled = true;
        oddFactory();
        console.log(odd(5));
    }
    mainFactory();
})();

Consider an alternative with single-value exports/imports. I've removed the names of the exported functions to clarify that the name does not apply to the exported symbol, but if they were retained, they would simply be the name of the function.

odd.js

import "odd.js" as odd;
export function (n) {
  return n == 0 || odd(n - 1);
};

even.js

import "even.js" as even;
export function (n) {
  return n != 0 && even(n - 1);
};

main.js

import "odd.js" as odd;
console.log(odd(5));

The transpiled form would be identical to the previous example.

Note that this cannot be done with let or var, because the declaration must hoist to a different scope.

This alternative impacts destructuring multiple exports.

With @dherman's proposal, each individual exported name populates a variable in shared scope. Formally ignoring nested module name-spaces for a moment, destructuring in an import clause does not occur at the site of the import clause, but at the site of the corresponding export.

mathy.js

export even = function (n) {
    return n == 0 || odd(n - 1);
};
export odd = function (n) {
    return n == 1 || even(n - 1);
};

main.js

import {even, odd} from "mathy.js";

// even and odd hoisted from mathy.js
(function (even, odd) {
    function mathyFactory() {
        even = function (n) {
            return n == 0 || odd(n - 1);
        };
        odd = function (n) {
            return n == 1 || even(n - 1);
        };
    }
    var mainFactoryCalled;
    function mainFactory() {
        if (mainFactoryCalled) {
            return;
        }
        mainFactoryCalled = true;
        mathyFactory() // call site of import {even, odd}
        console.log(odd(5));
    }
    mainFactory();
})();

Note that there is no destructuring assignment in the transpiled output. The destructuring occurs when the working set of modules is compiled, and each of the variables in the structure is mapped to a variable in the shared scope.

Consider the equivalent with single-value exports. This example applies to both return or export syntax.

mathy.js

var even = function (n) {
    return n == 0 || odd(n - 1);
};
var odd = function (n) {
    return n == 1 || even(n - 1);
};
return {even, odd};

main.js

import {even, odd} from "mathy.js";
console.log(odd(5));

transpiled.js

// even and odd hoisted from main.js instead of mathy.js
(function (mainEven, mainOdd) {
    function mathyFactory() {
        var mathyEven = function (n) {
            return n == 0 || odd(n - 1);
        };
        var mathyOdd = function (n) {
            return n == 1 || even(n - 1);
        };
        // site of return {even, odd}
        // for each corresponding import site:
        {mainEven, mainOdd} = {mathyEven, mathyOdd}; // main.js
    }
    var mainFactoryCalled;
    function mainFactory() {
        if (mainFactoryCalled) {
            return;
        }
        mainFactoryCalled = true;
        mathyFactory() // call site of import {even, odd}
        console.log(mainOdd(5));
    }
    mainFactory();
})();

The rules are slightly different. With single-value exports, we must execute destructuring for each corresponding import at the location of the export.

In both cases, the shared values stabilize after all involved factories have finished executing. Also noteworthy that single-value-exports does not need knowledge at compile time of the shape of the exports object. It also does not need to track at compile-time the boundary between static members and dynamic members.

Left as a future exercise, imports that are commuted to exports. These probably do not work so well with single-value-exports, but fine with multi-value-exports.