Lua style coroutines in nodeJS proposal

coroutine-example.js

var FS = require('fs');
var Fiber = require('./fiber'); // Not to be confused with the existing node-fibers module

// readFile is a normal non-blocking, async function, but internally it can use
// the coroutine helper fiber. These kinds of coroutines are no more dangerous
// to the caller than any other async function.
function readfile(filename, next) {
  // The call to Fiber is non-blocking.  It's contents are only run sync till
  // the first call to wait and then it returns.
  Fiber(function (resume, wait) {

    var err, fd, stat, chunks;
    
    console.log("opening...");
    // This is a normal async function.  It doesn't know anything about coroutines.
    // The FS.open call is non-blocking, even for this coroutine
    FS.open(filename, "r", "0644", resume);
    // Destructuring assignment, wait returns the arguments that resume got.
    // The call to wait will suspend this coroutine and only resume when the
    // `resume` callback eventually gets called.
    [err, fd] = wait();
    if (err) { next(err); return; }

    // Same thing, but doing a fstat on the fd
    console.log("fstatting...");
    FS.fstat(fd, resume);
    [err, stat] = wait();
    if (err) { next(err); return; }

    // Here is an example using a blocking loop to read a file in chunks
    console.log("reading...");
    var chunk, length, offset = 0;
    do {
      FS.read(fd, offset, 72, resume);
      [err, chunk, length] = wait();
      if (err) { next(err); return; }
      chunks.push(chunk);
      offset = offset + length;
    } while (length);

    console.log("pausing...");
    // Works for other async functions too, not just node ones.
    // Sleep this coroutine for 500ms
    setTimeout(resume, 500);
    wait();

    console.log("closing...");
    FS.close(fd, resume)
    [err] = wait();
    if (err) { next(err); return; }
    
    next(null, chunks, stat);

  });
}

fiber.js

// This is the implementation of the fiber helper used in the example.

// 'coroutine' is a made up addon or builtin to future javascript
// that implements lua style coroutines.
var Coroutine = require('coroutine');

module.exports = function Fiber(fn) {
  var resume = Coroutine.wrap(fn);
  resume(resume, Coroutine.yield);
};

uhoh

Where are the arguments to the function defined? Where does filename come from?

I updated it to make the helper less magical. I think it was trying to do too much.

It seems like something that could be sugared to by making the Fiber body an array with each element corresponding to the code between resume / wait calls. Each element of the array would then would be the continuation of the previous element.

This makes me think its not too far removed from existing libraries.

I dig it. For quite a while I've thought that Node should have fibers, but nobody should ever write a public function that expects to be running in a fiber. This sort of interface eliminates the running in a fiber concept.

I don't have much experience with this stuff, but this seems easier to read and understand to me. Makes the control flow logic linear which is what many programmer are used to. I wonder about nesting issues since the parent routine calling readfile(..) might be calling it inside the Fiber thing and calling wait() on the next line...

@ggoodman: the real strength of this sort of thing when compared to existing flow control libraries isn't very apparent with these simple step-by-step examples. Try throwing in some loops and complex branching logic, and suddenly an array of functions doesn't really represent your logic very well. The TameJS website (http://tamejs.org/) has some very good examples of real-world problems that are hard to express without coroutines.

I think the real strength is your stack doesn't get destroyed with each I/O call. This helps a lot in debugging. There is some validity to the syntax arguments as well, but I consider that style.

Here it is prototyped with @laverdet's node-fibers: https://gist.github.com/1349764

@creationix: if maintaining a clean stack is your number 1 priority, node-fibers is actually a better paradigm. Imagine that your readfile() example above called out to a function not written by you and not written with fibers that had a crash-y edge case in one of its callbacks. The stack trace won't even mention readfile(). You'll only get useful stack traces if every async function you rely on either a) never throws an exception or b) is implemented with Fiber.

Currently, in my largest Node project, I use node-fibers everywhere. Every function inside the project expects to be called inside a fiber, and every call to Node's built-ins is wrapped in a fiber-friendly function before being used. This totally solves the stack trace problem, but it is a huge pill to swallow. Even if one day in the distant friendly future, Node were to support coroutines, I imagine the implementation would look a lot like what you have here, which isn't really a silver bullet for stack traces precisely because it hides the fibers from client functions. My point is, I think the Node team is going to have to solve the stack trace problem in some other way.

Silver bullets are as mythical as the werewolves they kill. What I want is something minimal, incremental, and as natural to js semantics as possible. This doesn't solve all problems, but it helps a lot with the main use case where people can't handle callbacks (doing several async I/O operations serially as part of a larger task). If other code has to be fiber or co-routine aware to interop then it's not a natural fit. My proposal fits easily into existing paradigms.

The reason I mention preserving the stack is when comparing this to solutions like Step. Step is syntactically almost as nice as this, but debugging is a nightmare and things like loops or if statements are impossible.

I think we agree completely on what makes this a useful idea and how it fits better into the Node ecosystem than raw fibers. I was mostly reacting to your comment about stack traces being more important than manageable looping and branching.

It looks like this is just as easy to implement with generators, which will almost certainly make it into future versions of v8, so this is probably going to be a viable strategy for future modules.

https://gist.github.com/1364503