piccolo-module-refactor.md

In order to get the piccolo framework intro a more stable state I'm refactoring major and minor parts intro standalone modules.

A major part is the node-like module system. The system should be broken down intro two modules. A client-side and a server-side module. The server-side seams necessary, since the node-core alternative is totally blocking (not ideal in dynamic server resource loading) and there might be security concerns, where a localized module system is important. The client-side is necessary since no module system exists.

The thought is to provide two modules (in result one almost-isomorfic-api-only module). They would have the following API:

var http = require('http');

var ServerModuleSystem = require('ServerModuleSystem');
var BrowserModuleSystem = require('ClientModuleSystem');

var serverRequire = ServerModuleSystem({
  root: '/localized/directory/modules/',
  
  // see: https://github.com/AndreasMadsen/localizer
  // functionallity for adding .server.js files to the require.resolve
  // search pattern, and disallowing .client.js files
  allowed: function (basename) {
    return [ path.basename(basename, '.js') + '.server.js' ]; // just example
  },
  
  // should not be resolved
  core: {
    'url': /* some filepath */,
    'path': /* some filepath */
    /* isomorfic modules */
  },
  
  // the modules will have access to these globals
  globals: {
    'setTimeout': setTimeout,
    'clearTimeout': clearTimeout,
    /* std globals continues */
    'node': {
      'require': require,
      'process': process
    }
  }
});

var browserRequire = new window.BrowserModuleSystem({
  root: '/localized/directory/modules/',
  
  // see: https://github.com/AndreasMadsen/localizer
  // functionallity for adding .client.js files to the require.resolve
  // search pattern, and disallowing .server.js files
  allowed: function (basename) {
    return [ path.basename(basename, '.js') + '.client.js' ]; // just example
  },
  
  // should not be resolved
  core: {
    'url': /* some filepath */,
    'path': /* some filepath */
    /* isomorfic modules */
  }
});

http.createServer(function (req, res) {
  if (req.url.indexOf('/moduleSystem/')) {
    
    // module was required on the client. The require call was executed
    // from `source` and require() argument was `request`. This the client
    // should be given modules it allready has, there is an `acquired`
    // parameter too, specifying relative filepaths.
    // Note it will not just send the req.query.request but also the files
    // required by req.query.request and so on.
    req.pipe(
      browserRequire.send({
        // note req.query is just a fictional API
        'acquired': req.query.acquired || [],
        'source': req.query.source,
        'request': req.query.request
      })
    ).pipe(res);
    
  } else if (req.url === '/moduleLoader/') {
    req.pipe( browserRequire.core() ).pipe(res);
  } else if (req.url === '/') {
    // Send a html file including a request to the core file and something [1] there
    // invokes browserRequire.require('anotherModule', function (err, doIt) { doIt(); })
    // The module could then construct something using domstream or something else,
    // but that is up to another module.
    serverRequire.require('startingModule', function (err, doIt) {
      // could construct a domstream, or something else.
      doIt().pipe(req);
    });
  } else {
    // since `serverRequire.require` don't need have a static argument, it could
    // depend on the request input and serve a module.
  }
});

[1] File there invokes window.browserRequire.require in the client

var browserRequire = new window.BrowserModuleSystem({
  
  // the resolve part will be performed on the server, there is
  // no need to specify root, allowed or core.
  constructUrl: function (acquired, source, request) {
    return window.location.origin + '/moduleSystem/' +
            '?acquired=' + encodeURIComponent(acquired) +
            '&source=' + encodeURIComponent(source) +
            '&request=' + encodeURIComponent(request);
  },
  
  // the modules will have access to these globals
  globals: {
    'setTimeout': setTimeout,
    'clearTimeout': clearTimeout,
    /* std globals continues */
    'browser': {
      'window': window
    }
  }
});

document.getElementsByTagName('a')[0].onclick = function () {
  browserRequire.require('anotherModule', function (err, doIt) {
    doIt();
  });
};

preliminary conclusion

There is nothing new in this. All this is basically what piccolo does, but the isomorphic modules has been removed (perhaps module should continue to exist) and there are no insomorfic page construction. Unlike piccolo (current version) this also added the possibility for isomorfic-api-only modules, since someone can require('./binding.js') where there is no binding.js but instead there are binding.client.js and binding.server.js.

more thoughts

The optimization (e.q. resolve cache) there can be made by joining the two modules might be beneficial than having the modules seperated.

The fact that node.require has to be called with the node. prefix seams really bad, consider simply providing the node native require call on the server and focus less on async preloading on the server.

There are some security concerns, simply filtering *.server.js files, is not enogth if a npm installed module exists inside the localized environment.

Too complex. Just have commonJS require

whilst i agree that its hard to get a true isomorphic module system for both client and server side,i do agree,about the separation of server and client module,now i maybe Raynos as a point in using commonjs require,but also this implementation is not bad,but though abit over complicated as towards the setup unless all that will already be available for inuse without much setting up of the server module and clientmodule by the user, I would best advice the seperation of modules being limited to development and honestly if commonJS can resolve the loading ,then it won't be bad either

@rayons, I agree that this is complex but I don't see a way to support standard sync require calls on the client, without doing sync XHR. In my (limited) experience doing sync XHR a lot worse than a reading a file synchronously on the server. Could you elaborate on how you would provide a commonJS module system in the browser?

update Ah, perhaps you where speaking of the AMD style.

I'm not sure how well I have explained myself, but the thought isn't that require('filename', callback) should be called every single time. But only when the require calls can't be predicted by very simple static analysis. In normal cases (e.q. a require('async') call within a module file) would be sufficient, since it can be prefetched.

@influx6 my experience with getting a isomorphic module system is that it is fairly easy, (e.q. piccolo has a fully isomorphic module system). The big challenge is to get give access to browser API on the browser and node.js API on the server, without making the module system more complex and not creating a security leak.

in most cases you know you'r whole dependency graph at the start of run time, but there are some cases where is is desirable to load a module by a name stored in a variable, or just to only load a tiny core initially, and more as you actually need it.

We've (myself, @maxogden, @substack and others) have had some discussion about the simplest way to achive this, with the minimum departure from node.js style (note, node.js is not strictly commonjs, but is the flavor of commonjs which won)

So, basically, we have come to the conclusion that we'd need some way to bundle A, B, C, but subtract modules we already have X, Y, Z, from a new bundle...

then you could just request those asynchronously

require.async(['A','B','C'], function (err, A, B, C) {
  //now, these modules are loaded.
  /*
   This is just PSEUDOCODE. 
   there are many fine details for the exact pattern that would work best.
  */
})

however, A, B, C are still ordinary node.js modules.

@dominictarr thats a really good point. I didn't think there would be a great need for bundling module request asynchronously, but reconsidering the benefits in bundled request when they share dependencies this would definitely be worth supporting.

NOTE: it seams like require.async is renamed to require.ensure and have a slightly different API http://wiki.commonjs.org/wiki/Modules/Async/A