Ye Olde brainstorm about IO API

gistfile1.md

Data normalization API proposal

Data API normalization/centralization in three layers

1. transport layer

2. transaction layer

3. encapsulated configuration layer (aka Resource/DataSource)

4. Transport layer

---

Goals: encapsulate one method of pipelining data with as little code and as few requirements as possible.

var nativeObject = Y.io.<transport>(source, callback(err, response) {
    /* 'this' is appropriate object */
});

...optionally support context override as third arg to transport function, and pass appropriate object as third arg to callback

###Examples

var scriptNode = Y.io.script(url, function (err, response) {
    /* 'this' is the <script>? */
});

var xhrObj = Y.io.xhr(url, function (err, response) {
    /* 'this' is the XHR object?
*/ });

var func = Y.io.func(this.someMethod, function (err, resp) {
    /* 'this' is ??? */ },
    this); // override context?

Transports

• xhr
• xdr
• form (iframe?)
• script
• link
• flash?
• socket
• function/func
• array
• object -- see schema for usefulness
• more...

2. Transactions

---

Goals: create encapsulation of lifecycle wrapping transport, standardize API abstraction over transport layer, add events for consistent transaction lifecycle stages plus any stages that are appropriate per transport, support post-response processing

var transaction = Y.io(source, {
    transport: <transport name>,
    type: <post-processor name>,
    on: {
        // or start, end?
        send: callback(e) { /* 'this' and e.transaction are the transaction */ },
        response: callback(e) { /* same, plus e.response is the transport response */ },
    }
    after?
    context?
    args?
});

Alternate signatures

var transaction = Y.io(source, callback(e) { /* e.response */ } [, context?]);
var transaction = Y.io(source, { config, but without on: {...} }, callback(e) { /* e.response */ } [, context?]);
var transaction = Y.io(source, <type>, callback(e) { /* e.response */ });

Transport-specific configuration

Other configuration attributes/properties interpreted by the transport/type modules or generic feature modules/plugins, such as:

• method: ('get', 'post', 'put', 'delete', 'head')
• multipart: true || separatorString?
• sync: false
• headers: { 'content-type': 'application/json' }
• queuing: true
• polling: true || msInterval || { more config? }
• disabledFields: false
• native: true - for xdr transport
• data: ... - form data or maybe form Node or id
• timeout: ms
• schema - e.g.

schema: {
    resultListLocator: 'records.here',
    resultFields: [ ... ]
    output: ('array', 'arraylist', ModelList, MyArrayListSubclass, more?)
}

Other events interpreted by the transport

on: {
    success: callback(e) { ... },
    failure: callback(e) { ... }
}

Default transport

Y.io will default transport based on a test function on each transport, and/or set by the 'type' configuration.

CAVEAT: transport test-based defaulting could create confusion for JSONP vs XHR

var useAsDefault = Y.io.script.test(source, config); // => true/false/likelihood rating

var transaction Y.io('/service', function (e) {
    /* xhr transport passes test, is used */
});

// types can default transport
var transaction = Y.io('http://servi.ce/foo', { type: 'jsonp' }, function (e) {
    /* 'script' transport defaulted */
});

// types and transports can be incompatible
var boom = Y.io(source, { transport: 'link', type: 'jsonp' }, ...); // boom === null

3. Encapsulated configuration

---

Goals: Create transaction factories with configured defaults.

Use same/similar API signature to Y.io().

NAMING DISCUSSION: Y.DataSource, Y.Resource, Y.io.Resource, other?

var resourceA = new Y.DataSource(url/data, { ... });

var transaction = resourceA.send(extra/data, {
    /* config overrides, plus transaction level... */
    on: { ... }
}, contextOverride?);

// Transaction object points back to originating resource
transaction.resource; // => resourceA

// Post-creation subscription possible
transaction.on('success', fn);
transaction.after('end', fn);

// Transport-specific API decoration?
transaction.abort();

4. (bonus!) Widget extension API

---

Goals: normalize API for data layer configuration of Widgets (and others?)

Mirror the DataSource/IO API, but with 'source' property in configuration object taking the place of the leading url/source argument. Widget attribute should be 'data' (debatable).

var table = new Y.DataTable({
    data: {
        source: url,
        type: 'jsonp',
        schema: {
            resultListLocator: 'records',
            ...
        }
    },
    columns: [ ... ]
});

table.load( <sig args for dataSourceInstance.send( ... )> );

var chart = new Y.Chart({
    data: {
        source: url,
        type: 'jsonp',
        schema: {
            resultListLocator: 'records',
            ...
        }
    },
    type: 'pie'
});

chart.load( <sig args for dataSourceInstance.send( ... )> );

Would you pass the config for a transport to a specific transport static function as an argument? And the callback function will get called in response to any activity from a given transport? What about event subscriptions for that transport? How do you presubscribe to them before making the call?

One use case I am specifically thinking about is in uploader, where I use xhr to transport the file. I need to init the transport instance with some very specific initial settings; I need to listen to progress events from the transport instance; I need to listen to error events from the transport instance; I need to be able to abort the transport instance.

From this proposal, I understand that DataSource would potentially allow me to do all those things (yes?), but that seems like too heavy of a solution for those purposes.

And I imagine this would be a pretty common use case of any transport wrapper we produce.

A sidetrack that I also wanted to capture in a comment is a possibility of a server-side module to respond and process client-side data requests. For instance: xhr level 2 requests are tricky when used across domains, because they require a specific initial OPTIONS response. We could potentially create a server-side only YUI module that will facilitate responding to a variety of requests. This may be included in io with some sort of an additional io-server module. A very rough idea.

@lsmith, thanks for taking the time to think through all this stuff! I have some comments that I'll group in each of the three sections:

1. Transport Layer

I really like the callback-based, Node.js-like API. This seems like the way to go for sure!

I'm unsure what the context (this) inside the callback should be. I think we should consider whatever is standard in Node.js and trade that off with what is standard in YUI.

I feel that these APIs should be considered to be async, even the function and array transports. Do you think it makes sense to strictly enforce that by throwing things like array off thread via setTimeout()? Or is that crazy?

It seems like you are only thinking about these transports as only read-only APIs. When thinking about something like XHR, it should be able to be configured to POST, PUT, DELETE, etc. To clarify, the API could be defined like:

var nativeObject = Y.io.<transport>(source, callback (err, res) {
    // Do something...
});

Where source is an overloaded argument that each transport can dictate the "shorthand" type, but all transports should accept it as an object with a src property as well.

Going back to the XHR, a POST request could look like this:

var req = Y.io.xhr({
    src   : '/path/to/resource',
    method: 'POST',
    data  : Y.JSON.stringify({foo: 'bar'}),

    headers: {
        'Accept'      : 'application/json',
        'Content-Type': 'application/json'  
    }
}, function (err, res) {
    // Do something...
});

I think the Y.io.xhr function could also have some convenience methods hanging off of it to make it easier to get/send JSON data.

2. Transactions

Another goal for the transaction layer is to provide a standalone Y.Transaction (or some named thing) class which is based on the concepts of deferreds/promises.

I think it will be interesting to see if we end up wanting to build the transaction support on top of the current custom event API, or if we'll end up wanting to do something simpler…

To support your proposed API here, the user would also have to include the correct transport module in their use() statement, correct? Or were you thinking that they'd include io to get the transaction support, and the io module would depend on all the transport modules? <-- I don't like that.

I don't like the idea of overloading the transaction like you have it in the Transport-specific configuration section. I think the transport object should be a standardized on a single interface. They are an abstraction layer over all the nuances of the transports. In other words, I don't think Y.io() should necessarily be a transaction factory, in the sense that you're "configuring" a transaction. I think a standardized transaction object should just be the return value of the function. What you have smells more like it should be named: Y.io.createTransaction(source, options).

I think the pattern should be one that forces each transport to adhere to the common transaction interface, and the transport encapsulates the implementation of how this contract with the transaction is met.

3. Encapsulated Configuration

So yeah, everything I said in the section above about transactions is going to clash with what you have for this section :P

I think what you have here is definitely overloading the idea of transactions with this. We want to add another level of abstraction, but this time it's around a specific data resource. Y.DataSource/Y.Resource should be a fancy factory function that returns a DataSource/Resource instance. That instance object has API methods, that when called, return Y.Transaction objects.

I see the goals of this to be a smart factory that when called, it processes its arguments to determine the which transport to use. It then encapsulates the configuration for that transport and wraps it with a common "resource" API which will call into that transport and return transaction objects.

It makes sense to have DataSource/Resource subclasses that model a desirable API over a specific transport, but also work with a common DataSource/Resource API as well. This aligns well with the idea that Y.DataSource() is a factory which creates the appropriate DataSource/Resource subclass based on the inputs to the function.

Another option is to somehow expose the underlying transport's API, but wrap everything to use the configuration defaults and return transaction objects.

AutoComplete has some of this smart transport parsing, we should make this sort of thing available generically in the library and relieve AutoComplete of this duty.

var commentsResource, tx;

// Create the resource. The factory knows to create an XHR/REST one.
commentsResource = Y.DataSource('/comments/{id}', {
    headers: {
        'Accept'      : 'application/json',
        'Content-Type': 'application/json'
    }
});

// Get some data from the resource at: "/comments/1".
tx = commentsResource.get({
    params: {id: 1},

    on: {
        success: function (e) {
            console.log('Yay!');
        }
    }
});

// Register another event handler on the same tx.
tx.on('abort', function (e) {
    console.log('Oh noez!');
});

tx.abort(); // => "Oh noez!"

// Add a new comment by POSTing to "/comments/".
commentsResource.post({
    data: {
        author : 'ericf',
        comment: 'First!!!111'
    }
}).on('success', function (e) {
    var comment = e.res;
    console.log(e.res.id); // => 1
});

Some of these ideas for resource encapsulation are coming from my REST Resource Gallery module.

Some random thoughts:

• I agree with Eric, there should be only one Transaction class
• But Y.io<transport>() should return the Transaction instance which wraps the native XHR object
• Transaction should just keep a list of callbacks since it will be bundled with Get which is in the core YUI file. DataSource can be used when you need an EventTarget and more abstraction
• Transaction should have the same methods that are added to Y.io so Y.io.css(foo).script(bar) makes sense (with promises one would run after the other completes).
• Sockets should have a simple API that communicates clients and server. It doesn't get any better than socket.io.
• I kind of prefer plugins for IO in Widget

Deferred/promises proposal at lsmith/yui3#36

Feedback here pertaining to how it would fit the transaction layer, structural/code feedback on the pull request, please :)