GELAM Test Thought Experiment

test.js

/**
 * This is a sketch of one alternative implementation of the GELAM
 * test harness. At a high level, it does three things:
 *
 * 1. It still uses the ArbPL/Loggest framework of loggers and
 *    expectations (mocked out below with slightly different naming).
 *
 * 2. It replaces the serial "test step" framework with a flow based
 *    on Promises.
 *
 * 3. It explicitly shuttles functions to frontend/backend contexts.
 *
 * Tests run in an undetermined context -- _not_ the frontend, not the
 * backend, not the server. You shouldn't have any access to any of
 * those things at test-runner level. Instead, you must 'shell out' to
 * the frontend/backend/server explicitly. This is done via
 * Function.toString() so as to avoid variable leakage.
 *
 * Instead of serial 'test steps' as before, test flow is controlled
 * through promises. The function that runs in the other context
 * should return a promise; to exit the context, you resolve the
 * promise as you would in normal JS code. In this way, a test step
 * for GELAM would follow an execution flow similar to those in Gaia;
 * a test's final 'done' callback could just be hooked onto the end of
 * the promise chain.
 *
 * Removing these serial test steps means we must have an alternate
 * way to keep logging from getting all mushed together:
 *
 * To keep the detailed level of logging with ArbPL, we use a special
 * type of promise called a LoggerPromise. This promise resolves
 * normally, like any other promise. However, it does two additional
 * things: one, it allows you to grab references to Actors; two, it
 * itself acts as a lazyLogger for convenience.
 *
 * When the LoggerPromise would resolve normally -- for instance,
 * after doing work in the Frontend context -- it checks its
 * expectations. If it succeeded, the promise resolves as normal and
 * the test continues. But if an actor's expectations went unmet, it
 * fails the Promise, logging and cascading the error as we would in
 * our current infrastructure. To mimic our current notion of a test
 * step or test group, the LoggerPromise could be named.
 *
 * In theory, this preserves a hierarchy of test steps. You can still
 * define a set of expectations that you must complete before moving
 * on to the next step, but you aren't restrained to one flat serial
 * level. Similarly, these tests could integrate seamlessly with the
 * Marionette tests, because we're explicit about which parts of the
 * code get run in different contexts.
 *
 * Example of sorts follows.
 *
 */

mail.onFrontend(function(frontend) {
  // Create a LoggerPromise, which is really just a Promise.
  var p = new frontend.LoggerPromise('Opens a Folder');
  // You can grab a reference to an Actor from a LoggerPromise,
  // and specify things that the actor must log.
  p.actor('ImapFolderConn').mustLog('syncDateRange');
  // Alternately, p could simultaneously be a lazy logger:
  p.mustLog('some thing');
  frontend.MailAPI.viewFolders(function(folders) {
    p.log('some thing');
    // Rather than implicitly completing a "step" when all
    // expectations have been resolved, the LoggerPromise will
    // explicitly raise hell if expectations are unmet when the
    // promise would have otherwise normally been resolved, here:
    p.resolve(folders);
  });
  return p;
}).onBackend(function(folders) { // << JSONified through
  // In this function, the result of the promise has been shuttled
  // through to the server context. You could then instantiate more
  // LoggerPromise expectations, etc.

}).then(function() {
  // Or we could run stuff in the testing context, e.g. if we're
  // running in Marionette and we want to do things in la-la land.

  // We know the complete test has finished after we've resolved the
  // final promise; the test is done here.
}).then(testDone, testDone);