waterfall data layer

collection.js

define([
  'underscore',
  'knockout'
], function(_, ko) {

  function Collection() {
  }

  Collection.prototype.init = function() {
    this._instances = ko.observableArray();
  };

  // to be overridden:
  // - Collection.prototype.url
  // - Collection.prototype.create
  // - Collection.prototype.update

  Collection.prototype._dedupe = function(item) {
    var existing = this.byId(item.id);
    if (!existing) {
      existing = this.create(item);
      this._instances.push(existing);
    } else {
      this.update(existing, item);
    }
    return existing;
  };

  Collection.prototype.byId = function(id) {
    return _.find(this._instances(), function(instance) {
      return instance.id() === id;
    });
  };

  Collection.prototype.fetch = function(params) {
    // the specific thing we're using for the ajax request should maybe be
    // abstracted
    return $.ajax(_.extend({
        url: this.url
      }, params))
      .then(function(result) {
        result.forEach(this._dedupe.bind(this));
        return result;
      });
  };

  Collection.prototype.fetchOne = function(id, params) {
    return $.ajax(_.extend({
        url: this._url + '/' + id
      }, params))
      .then(function(item) {
        return this._dedupe(item);
      });
  };

  // also need to figure out saving and deleting. two options:
  // - add them as methods to Collection.prototype (uglier)
  // - add them as methods to the items in resource._instances, but then we
  //   need a way of updating that list when an item is saved or deleted
  return Collection;
});

metadata-collection.js

define([
  'ko',
  'waterfall',
  'collection'
], function(ko, waterfall, Collection) {
  var instance;

  function MetadataCollection() {
    this.init();
  }
  
  MetadataCollection.prototype = new Collection();

  MetadataCollection.prototype.url = waterfall.metadata.url;

  // this is where things get weird. we probably want to return an instance of
  // something w/ a 'save' and 'delete' method, but those instances need some
  // way of notifying the Collection instance when they're deleted or saved for
  // the first time so the Collection can update its _instances OA accordingly
  //
  // so basically like a Record class
  MetadataCollection.prototype.create = function(item) {
    return {
      id: ko.observable(item.id),
      name: ko.observable(item.name),
      multiValue: ko.observable(item.multiValue),
      format: ko.observable(item.format)
    };
  };

  MetadataCollection.prototype.update = function(existing, item) {
    existing.id(item.id);
    existing.name(item.name);
    existing.multiValue(item.multiValue);
    existing.format(item.format);
  };

  // use an accessor function, since we'll usually just want a single shared
  // app-wide Collection of MetadataCollection, but we want to be able to reset
  // that instance for stuff like unit testing
  //
  // the "singleton pattern", or "object manager factory get instance function"
  function accessor() {
    return instance? instance : instance = new MetadataCollection();
  }

  accessor.MetadataCollection = MetadataCollection;

  accessor.reset = function() {
    instance = null;
  };

  return accessor;
});

Thanks for coding this up! Started looking at it a few days ago and got to spend some more time with it today.

Need to get used to the term "resource". I keep tripping over it in my mind. Here a resource is kind of like a Collection in Backbone. It's a group of "instances", which are kind of like Backbone models. Right?

This is a good start!

I think create should probably be called new if it's not going to add the record to _instances.

Here are my notes on data layer from our last 1:1:

• CursorModel, ModelCollection and the SDK may be unnecessary
• "Resources", a.k.a models, collections, client-side DB tables
• Resources handle server CRUD operations
• Resources use Knockout observable arrays (single point of truth)
• Some kind of "computed observable array" to filter resource observable arrays into collections that views can consume
• Pass shared observables into views. Views have state anyway so might as well share so they keep single point of truth and each other up to date.
• Resource reconciles server data with its observable array

@evanrmurphy my responses:

Need to get used to the term "resource"

i'm totally open to changing this to something else. ideas? Resource{Collection,Manager} seem to verbose, Manager seems terse enough, but maybe overly general? Collection seems terse enough and meaningful (also follows mongo convention), so maybe that's what we should call it, but then we'll need a name for the thing where we've got a limited view of it (i.e. if we have a Collection of every metadatum, but the /metadata page only fetches 20 of them, what do we call that subset? Query? CollectionView?)

I think create should probably be called new

i see the motivation here, since it's instantiating it and not actually performing a CRUD action, but the precedence here is Object.create and Supermodel.js, so i would vote to keep this as is. especially because we don't really have a method corresponding to each CRUD action anyway -- delete is a reserved word, so we can't use it, and the model.save() method will perform either a create or update.

CursorModel, ModelCollection and (maybe) the SDK are probably unnecessary

yea ModelCollection actually probably has the interface that we want, but you're right we probably would want to throw it out since it's built on CursorModel and SDK Collection.

Some kind of "computed observable array"

we did this once, and then copied it. we should just break it out, and then find a way to work it into the mapping. things like this are why i think the mapping may be unnecessary.

Views have state anyway so might as well share

this is where we get into that 2-level thing i wrote you a book about. some of a view's state is shared, some of it is specific to the view instance and can't be shared if we want to instantiate multiple copies of a view. think of a calendar picker, the shared state is an observable containing the selected date (or null), and the view-specific stuff is things like the range of acceptable dates. don't pay attention this just confuses the matter

come to think of it i'm really liking Model and Collection (backbone kinda nailed it). and then just a Query for the case where we want to view like the first page.

we did this once, and then copied it. we should just break it out, and then find a way to work it into the mapping. things like this are why i think the mapping may be unnecessary.

So here was my first attempt to write a computed observable array:

ko.computed(function() {
  function getName(user) { return user.name() }
  return _.sortBy(usersCollection(), getName)
})

It's just a computed that depends on the hypothetical observable array usersCollection.

What I gather from your links is that this simple approach will be too eager and can cause a jarring user experience in the views. Your way is better because it's more lazy, only updating the elements of the observable array that actually change rather than the whole thing every time. Is that right?

come to think of it i'm really liking Model and Collection (backbone kinda nailed it). and then just a Query for the case where we want to view like the first page.

I like "Model", "Collection" and "Query"! So here's a summary of our key components so far and how they relate:

• Model: Pretty much what you'd expect. A class defining general attributes and behaviors for users or stream entries or whatever. Model instances are observables corresponding to database records on the server. When a model instance is updated, parts of the app which depend on it are kept in sync through Knockout dependency tracking (and also simple shared reference?).
• Collection ("Resource" above but renamed): Master observable array of model instances. Handles server-side CRUD operations. There is one single-source-of-truth collection for each kind of model. So a users collection with all the user model instances the client cares about, a stream entries collection with all the stream entry model instances, etc.
• Query: Computed observable array, depending on a collection and returning a mapped/filtered/sorted/etc. version of it for things like pagination, ordering and getting subsets.
• When a model instance is created or deleted, its collection needs to be updated. The collection will update the server and any relevant queries will be automatically updated through Knockout dependency tracking (since a query is a computed observable array depending on a collection).

a few things about these COA's:

• yes, they need to be more complicated than simply a computed that returns an array because we want to bind them to UI elements, and be able to insert items into the underlying array w/o re-rendering the entire list. in order for this to happen, knockout needs to be binding to an actual OA
• the COA's i linked to can be thought of as _.map in that they just take some translation from the elements of the source array to the elements of the computed array, but they maintain the order. sorting is a different case, since the items remain the same, but the order changes. so if we want to support both, we'll have to re-think that implementation
• Querys can't just be COA's, they'll need to actually make an API call every time they want to get their contents. returning to the metadata example, if we've got a metadata collection and a Query of the first 20, then a separate view creates a new metadatum and saves it, the Query has no way of knowing if that new one belongs in it's set because that logic is implemented server-side (and we don't want to duplicate that logic). even though Query is some subset of its Collection's _item array, only the backend knows which subset. in order to maintain the decoupling here, we would need Querys to subscribe to updates of on the Collections, and then if they get some update like "new query created", they would need to make an API call if they wanted to keep themselves up to date. does that make sense?
• Query can be implemented independent of COA's. really this whole thing doesn't depend on COA's

also if we go w/ Model/Collection/Query, then we probably want Model.create (or at least Model.prototype.destroy), which implies some reference from Model instances back to their collection. this easily turns into spaghetti code, so we should be careful about this implementation.

Nice, I see you renamed "resource" to "collection" throughout the gist! 😄

Querys can't just be COA's, they'll need to actually make an API call every time they want to get their contents. returning to the metadata example, if we've got a metadata collection and a Query of the first 20, then a separate view creates a new metadatum and saves it, the Query has no way of knowing if that new one belongs in it's set because that logic is implemented server-side (and we don't want to duplicate that logic). even though Query is some subset of its Collection's _item array, only the backend knows which subset. in order to maintain the decoupling here, we would need Querys to subscribe to updates of on the Collections, and then if they get some update like "new query created", they would need to make an API call if they wanted to keep themselves up to date. does that make sense?

Hmm let's talk about this more. I actually think it would make sense to have logic like "first 20" on the client even if it's already on the server. I think all or most queries can be resolved entirely on the client simply using the data in the collections, and without having to make their own API calls. When API calls do need to be made, the collections can handle them and update the queries via Knockout dependency tracking.

Why would we want to rely on the server for what we could easily accomplish with _.first(MetadataCollection.accessor()._instances(), 20)? As long as we keep the collection up to date, then we have all the information we need to resolve that query with confidence. I think the same goes for queries to filter or order the collections. Are there other kinds of queries you're thinking of where we would need more help from the server?

So, I'm still thinking of queries as simply computed observable arrays that depend on collections. I think it would be simpler if we could let the collections handle all interactions with the API. Trying to understand the issue you're finding with this approach...

a few things about these COA's:

• yes, they need to be more complicated than simply a computed that returns an array because we want to bind them to UI elements, and be able to insert items into the underlying array w/o re-rendering the entire list. in order for this to happen, knockout needs to be binding to an actual OA
• the COA's i linked to can be thought of as _.map in that they just take some translation from the elements of the source array to the elements of the computed array, but they maintain the order. sorting is a different case, since the items remain the same, but the order changes. so if we want to support both, we'll have to re-think that implementation

Just took a stab at a generic COA implementation based on your filters and this discussion.

It accepts any compute function, so it should work for sorting as well as mapping translations. There's an example in the comments there for using it to sort the metadata collection alphabetically by name.

It uses a helper function mergeObservableArrays that takes the diff of two observable arrays and tries to merge them with minimal removes and inserts. We can work to optimize this algorithm over time.

The code probably doesn't work because, like any good developer, I didn't actually test it out. 😄 A working implementation shouldn't be far off though.

Why would we want to rely on the server for what we could easily accomplish with...

great question. 2 scenarios:

1. we have a view w/ a query of the 1st 20 items, then the user creates a new item and saves it. should this query include that new item? if so, which should it exclude (presumably the view is a table of 20 rows)? and how does this differ from one resource to the next? i.e. the 1st 20 of one resource may be determined by sorting the name field, while another is determined by sorting the id

2. the user vists /metadata/page/2, and then clicks a /metadata link. the state of the metadataCollection._items OA will be one of 2 things:

   1. a simple array of indices 0-19
   2. a sparse array w/ nothing in 0-19, and the metadata models in indices 20-39

   if it's the first case, then how does the Query instance for the /metadata page know that the stuff in metadataCollection._items corresponds to the second page?

   if it's the second case, then we've got to do a bunch of bookkeeping when we insert/remove models or change the sorting key.

i'm not saying these are impossible to solve, but they're non-trivial, hard to hammer all of the bugs out of, and hard to design in a way that's robust to system changes. so the question is what we get by trying to get this right, versus just making that API call every time we want a Query. i don't think the small perf benefits outweigh the cost of implementation, especially if this is something we can just implement later (i.e. our current design doesn't preclude this).

so a couple questions to get us on the same page:

• are we seeing the same cost of implementation? like when i think of keeping Query's in sync w/ server state i see that as a big task. do you?
• are we seeing the same benefit? the only benefit i see is the user will be able to go to a number of pages w/o waiting for the api call, which doesn't seem like much. am i missing other benefits tho?

i'll add comments to the COA gist.

Really productive talk today @aaronj1335! 👍 Some notes:

• Queries aren't computed observable arrays, but each one has an underlying (non-computed) observable array
• Collections and queries each have their own observable array, but they share model instances
• When a query calls the server, it gets updated with the appropriate model instances. The collection corresponding to that type of model also gets updated. If those model instances aren't in the collection yet, they're added. If they are in the collection, then they're updated with the new data from the server.
• Collections aren't bootstrapped at the initial page load. They just grow organically throughout the session as queries are made to the server.
• Computed array observables aren't needed for the initial implementation of this data layer
• Queries can be created, refreshed and deleted, but they can't be changed to different types of queries (queries with different parameters)

This stuff is feeling much more clear to me now. Planning to sketch out some code and/or detailed examples soon.

Now for the fun part: what should we name this little library we're developing? 😄 Knockout Data? Boxing Gloves? (Because you wouldn't want to knock someone out without boxing gloves!)

Started a branch for this called "data-layer" in WaterfallEngineering/frontend.