Skip to content

Instantly share code, notes, and snippets.

@juandopazo
Last active December 16, 2015 08:19
Show Gist options
  • Star 7 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save juandopazo/5405383 to your computer and use it in GitHub Desktop.
Save juandopazo/5405383 to your computer and use it in GitHub Desktop.
Storage module design and API for review

Storage

Motivation

There are 3 major types of storage is browsers up to date:

  • IndexedDB. The future.
  • localStorage. Limited in size and can be observed through the storage event.
  • WebSQL. Dropped by the W3C but it's still very present in the wild, with notable platforms like iOS and PhoneGap.

Browser support Green: IndexedDB; blue: WebSQL; yellow: localStorage

In the mobile world local cache and storage is very important for various reasons:

  • faster loading times
  • unreliable connections
  • offline use

My particular use case is an app to be used "in the field" where you probably don't have a connection and you need to do data entry. My solution is to save to a local storage and sync to the server when a connection is available. The 5 MB of localStorage may not be enough in my case, but I won't be able to tell until the app is used by real users.

Open questions

  • Is there enough interest in this considering the availability of localStorage?
  • Is it too early to use these technologies?

Architecture

The goal is to use conditional loading to load one of 3 or 4 sub-modules: indexeddb, websql, localstorage and a fallback to memory.

  • IndexedDB uses a key-based storage organized in "object stores".
  • WebSQL uses SQL tables. WebSQL can be used as a fallback for object stores by matching a table to a store and using tables with two columns: a key and a value column which saves the object serialized as JSON.
  • IndexedDB uses structured cloning which is similar to JSON.parse(JSON.stringify(obj)).

Open questions

  • Is falling back to localStorage a good idea considering the limited size and observability?
  • Is the whole concept OK or is it too hackish?
  • Should this include over-the-wire databases via some protocol?

API design

The API aims to find the lowest common denominator between all implementations. A database requires:

  • A name. Required by both IndexedDB and WebSQL
  • A version. Required by both IndexedDB and WebSQL
  • A structure, which means the name of each object store, tied to the version. Required by both IndexedDB and WebSQL
  • A size in bytes. Required by WebSQL

A Storage constructor receives all these parameters and creates an instance with properties with the names of each object store:

var storage = new Y.Storage({
  name: 'my database',
  version: 1,
  size: 5242880,
  stores: ['fooStore', 'barStore']
});

Assert.isObject(storage.fooStore); // true
Assert.isObject(storage.barStore); // true

A Storage instance also has a close method that closes the database connection/session.

A Store object is created for each requested store in a property of the same name. A store can perform the following actions:

  • get. Retrieve an object by key
  • put. Insert or update an object by key
  • remove. Remove an object by key
  • clear. Remove all objects from the store
  • count. Count the objects in the store

All methods return a promise.

Example:

storage.fooStore.put('some key', { foo: 'bar' }).then(...);

Open questions

  • Should Storage inherit from Base in order to use attributes for configuration and maybe fire close or error events? A versionchange event is very important.
@ItsAsbreuk
Copy link

Juan, I really like this!

Actually I need it (sort of) for a case of my own and started to create my own store, which doesn't come near to your approach.

Concerning your questions:

  • Iterest? YES! I found out troublecases using localstorage (some customers), but I didn't find out the real reason. So, we need a rocksolid storagesystem.
  • Too early: No
  • Store-methods return a Primose: Yes please. Even if you don't need a value, you might want to make the promise part of a chain
  • Inherit from Base: Think so. You don't create too many storageinstances seems to me. The attribute-events can be used when a user wants to change the memorysize, or add new stores.
  • I was used to work with storage.get('fooStore', 'somekey'). I like storage.fooStore.get('somekey') better. But I would hate to use storage.get('fooStore').get('somekey'), which conflicts with my suggestion about Base.

Just some notes to consider:

  1. It would be great if the storage could save and return simple types instead of just a String. IndexedDB seems to manage that, but localstorage doesn't (as far as I know). I managed that by adding a flag as the first character of the stored value and -after retreiving- removed the flag and parsing the right type to return (http://projects.itsasbreuk.nl/apidocs/classes/ITSAStorage.html).
  2. It would be awesome if we could store Y.Model (or descendents) instances without having to transform the attributes to an object ourself. Just store a Model and let the storage return a Modelinstance. You can actually store the data as an object under the hood. The difficulty lies with retreiving --> the user should specify what model-class it was. (Y.Model by default - when not specified). Just the same way as LazyModelList is using when Model-instances are revived. Only: LazyModelList holds the same class for all objects, where as the storage can hold different. You could then have something like: storage.fooStore.get('myWheatherModel', Y.WheaterModel).

Regards,
Marco.

@caridy
Copy link

caridy commented Apr 19, 2013

Juan, few notes:

  • we should sync up with @sdesai to revisit his work on Y.DB
  • our target should not be browser-level storage mechanism, but any storage mechanism, including over-the-wire, in which case Y.Storage can act like a shim. This obviously through "latency compensation" into the mix :)
  • We should check what Meteor's folks has done to shim multiple DB implementations.

In general I will argue that trying to get this to work with IndexedDB, localStorage and WebSQL is not enough because it is browser-level specific, while we are trying hard to narrow the line between client and server. I can imagine a Y.Model/ModelList/LazyModelList that represents data handled by a routine at the server side, and be completely runtime agnostic by relying on Y.Storage :)

@juandopazo
Copy link
Author

Thank you very much for the feedback guys!

Some answers:

@ItsAsbreuk

The attribute-events can be used when a user wants to change the memorysize, or add new stores

Actually you can't do that. In order to change the size and the structure, you need close the database and open a new one with a different version number. Abstracting over that seems hard and impractical.

It would be great if the storage could save and return simple types instead of just a String

Indeed, IndexedDB lets you do that and Storage uses JSON to store objects and arrays in WebSQL.

It would be awesome if we could store Y.Model (or descendents) instances without having to transform the attributes to an object ourself

Absolutely! WebSQL should already do it in my implementation since it passed the object through JSON.stringify which looks for toJSON. The IndexedDB side should look for toJSON.

@caridy

our target should not be browser-level storage mechanism, but any storage mechanism, including over-the-wire

I'd like to include anything local, including web app wrappers like PhoneGap or any other that I find. The reason why I didn't include over-the-wire implementations is that I want to leave the door open for different syncing/conflict resolution algorithms. I want to be able to have Models that sync to Storage and, when available, sync to the server via a REST-like API.

Basically my idea so far has been to handle this at the Model level.

@juandopazo
Copy link
Author

Updated the gist: just promises, no callbacks; all actions return promises. Also added the question to discuss if this should include over-the-wire databases.

@juandopazo
Copy link
Author

Ouch! light bulb

@caridy by over-the-wire do you mean using this in a Node environment to abstract over Mongo, MySQL or whatever? If that's the case, then I'm totally interested. Sadly my side project migrated from Node to Python, so I won't be using YUI on the server to test this in real life.

@caridy
Copy link

caridy commented Apr 19, 2013

@juandopazo, yes, I'm talking about Node. And that algorithm for conflict resolution is called "latency compensation" and there are a couple of interesting implementations. The one from Meteor works pretty well, we might be able to use that one. In theory, there is not much different between a REST-like API and IndexedDB.

Aside from that, I think we should use Y.DB instead of Y.Storage. I really liked the implementation from @sdesai for the CRT project.

@juandopazo
Copy link
Author

@caridy I guess I need a bit more research then. I'll look at Meteor and the literature on latency compensation.

I'd love to see @sdesai's implementation!

@caridy
Copy link

caridy commented Apr 20, 2013

@ItsAsbreuk
Copy link

Quite interesting "latency compensation".
Shouldn't YUI support this inside the core?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment