node-document next

TODO

::::: CURRENT/NEXT ::

- [FEATURE]: `Storage#get('post/*')` - get all documents of type 'post' (...using some kind of pagination if possible?)

      [x] global
      [x] redis
      [ ] fs
      [ ] memcached
      [ ] leveldb
      [ ] mongodb
      [ ] elasticsearch
      [ ] couchdb
      [ ] riak
      [ ] kyotocabinet
      [ ] amazons3

- [FEATURE]: Count records by type: `Storage#count(<TYPE>)`
      - Memory: `Object.keys`
      - FileSystem: http://nodejs.org/docs/v0.3.1/api/fs.html#fs.readdir
      - Memcache: https://gist.github.com/1365005 (HACK)
      - Redis: http://redis.io/commands/keys
      - MongoDB: http://docs.mongodb.org/manual/reference/method/db.collection.count
      - ElasticSearch: http://www.elasticsearch.org/guide/reference/api/count.html
      - AmazonS3: http://docs.amazonwebservices.com/AmazonS3/2006-03-01/API/RESTBucketGET.html (HACK - bad one)

      [x] global
      [x] redis
      [ ] fs
      [ ] memcached
      [ ] leveldb
      [ ] mongodb
      [ ] elasticsearch
      [ ] couchdb
      [ ] riak
      [ ] kyotocabinet
      [ ] amazons3

- [FEATURE]: Clear records by type: `Storage#clear(<TYPE>)`
      - Memory: `Object.keys`
      - FileSystem: http://nodejs.org/docs/v0.3.1/api/fs.html#fs.readdir
      - Memcache: https://gist.github.com/1365005 (HACK)
      - Redis: http://redis.io/commands/keys
      - MongoDB: http://docs.mongodb.org/manual/reference/method/db.collection.remove
      - ElasticSearch: http://www.elasticsearch.org/guide/reference/api/delete-by-query.html
      - AmazonS3:
            http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
            http://docs.amazonwebservices.com/AmazonS3/2006-03-01/API/RESTBucketGET.html (HACK - bad one)

      [x] global
      [x] redis
      [ ] fs
      [ ] memcached
      [ ] leveldb
      [ ] mongodb
      [ ] elasticsearch
      [ ] couchdb
      [ ] riak
      [ ] kyotocabinet
      [ ] amazons3

- [FEATURE]: `Storage#keys` - export all document keys.

      [ ] global
      [x] redis
      [ ] fs
      [ ] memcached
      [ ] leveldb
      [ ] mongodb
      [ ] elasticsearch
      [ ] couchdb
      [ ] riak
      [ ] kyotocabinet
      [ ] amazons3

- [FEATURE]: `Storage#types` - export all document types.

      [ ] global
      [ ] redis
      [ ] fs
      [ ] memcached
      [ ] leveldb
      [ ] mongodb
      [ ] elasticsearch
      [ ] couchdb
      [ ] riak
      [ ] kyotocabinet
      [ ] amazons3


::::: HIGH ::

- [FEATURE/API/PERFORMANCE]: Enhance `Collection` DB operations to not run as single operations, rather merge them and use bulk DB operation.
      1. Operations

           Post.set(1, data, callback) // Post.storage.set('Post', 1, data, callback)

           Article.set(1, data, callback) // Article.storage.set('Article', 1, data, callback)

      2. Convert to Bulk Operation
           Document([Post, Article]).set(1, data, callback)

- [FEATURE]: Support writing to multiple kinds of DBs at same time, e.g. mount array of storages.

      - Example:

          Post.use(redis)
          Post.use(memcached)

          (new Post()).save()

          Post.unuse(redis)
          Post.unuse(memcached)

          Post.use([redis, memcached])
          Post.unuse([redis, memcached])

- [FEATURE/API]: GET with option to get paginated content (limit, offset)

      - Example:

          storage.get('Post', {limit: 100})

          storage.get('Post', {limit: 100, page: 2})
          storage.get('Post', {limit: 100, index: 1})

          storage.get('Post', {limit: 100, page: 3})
          storage.get('Post', {limit: 100, index: 2})

          storage.get('Post', {limit: 100, index: 2})

- [ADAPTER/FEATURE]: Generic CLI for interacting with/between storages: `node-document-cli` - requires `Storage` to be ported to `Stream` first.

      - CLI

            $ node-document <COMMAND> <PATTERH> <OPTIONS> --storage redis://localhost:6379/db1

      - REPL:

            $ node-document --storage redis://localhost:6379/db1
            [redis://localhost:6379/db1] > GET hello
            [redis://localhost:6379/db1] > SET hello {a: 1}
            [redis://localhost:6379/db1] > DEL hello
            [redis://localhost:6379/db1] > STORAGE redis://localhost:6379/db2

- [ADAPTER/FEATURE]: Seeds (`node-document-seed`) - Simple seed generator (interface)

      - Adapters:
          - node-document-seed-factory - Seed data from fake data based on JSON schema
              - Lib: https://github.com/acatl/datafixture.js
          - node-document-seed-file - Seed data from existing files
              - CVS
              - JSON
              - YAML

- [ADAPTER/FEATURE]: Generic client-side HTTP GUI: `node-document-gui`
      - Example: https://github.com/tldrio/mongo-edit
      - Example: https://github.com/hij1nx/levelweb
      - Library: https://github.com/josdejong/jsoneditor

- [ENHANCEMENT/REVIEW/DISCUSS]: Event hooks both before and after (...and middle?):
      - API:

          Document.Model.prototype.emit('save:before') - before (first)
          Document.Model.prototype.emit('save:after') - after (last)

      - Decision: This should probably be a middleware?

- [FEATURE]: Review meta fields: `_updated_at`, `_created_at`. Adapter and hook into events, e.g. "before save", etc.? Middleware API?
      - Example:
          var TimestampsMiddleware = require('node-document-middleware-timestamps');

          Post.use(TimestampsMiddleware);
          post.use(TimestampsMiddleware);

          [Post, Article].use(TimestampsMiddleware);
          [post, article].use(TimestampsMiddleware);

      - Original *deprecated* idea - middleware could do this without bloating the API:

          // Will be saved as `_created_at` (on first write) - if meta prefix is '_'.
          Post.meta('created_at', false, function() {
            // Access to instance (doc) and klass (doc.klass) if needed.
            return new Date();
          });

          // Will be saved as `_fingerprint` (on every write) - if meta prefix is '_'.
          Post.meta('fingerprint', true, function() {
            return [this.klass.type, encodeURIComponent(this.url)].join('-');
          });

      - Decision: This should probably be a middleware?

- [FEATURE]: Support "expires" for documents (Storage)

      * Always set `_expires` property.

      - Adapters:
          - Memory: Invalidate on fetch
          - FileSystem: Invalidate on fetch
          - Redis: http://redis.io/commands/expire
          - MongoDB: http://docs.mongodb.org/manual/tutorial/expire-data
          - ElasticSearch: http://www.elasticsearch.org/guide/reference/mapping/ttl-field.html
          - Memcache: http://code.google.com/p/memcached/wiki/NewCommands#Standard_Protocol
          - AmazonS3: http://aws.typepad.com/aws/2011/12/amazon-s3-object-expiration.html (i.e. note really => invalidate on fetch)

      - Decision: This should probably be a middleware?

- [FEATURE]: Migrate from one DB to another via storage instances: `Storage#export(storage || Stream, ...)` + `Storage#import(storage || Stream, ...)`, and maybe Storage#sync(storage, ...
      - Inherit Stream?
            - http://maxogden.com/node-streams
            - http://blog.nodejs.org/2012/12/20/streams2/
      - Example:
            function copy (srcdb, dstdb, callback) {
              srcdb.readStream().pipe(dstdb.writeStream()).on('close', callback)
            }

- [API]: Subclassing for simpler use of `instanceof` keyword checking:

      EventEmitter
        Adapter
          Validator
          Differ
          Stream
            Storage


::::: LOW ::

- [FEATURE]: Connection pool for storage connections - not only per instance, per storage adapter
      - https://github.com/coopernurse/node-pool
      - Example:

        SomeStorage.pool = poolModule.Pool({...});
        SomeStorage.prototype.__defineGetter__('pool', function() { return this.klass.pool; });

        SomeStorage.aquire(callback);
        (new SomeStorage).aquire(callback);

      - Issue/Review: Needs to close connections manually to make this behave as intended. Make connection pools optional?

- [DOC]: In each storage README; list support table, connection-URL/key mapping table, connection URL format, etc.

- [REFACTOR]: Make `Differ#diff` and `Validator#validate` synchronous? - worst case drop support for `Amanda` or "fake it".

- [ISSUE]: Reserved attribute keywords (e.g. CouchDB); meta attribute field prefix should be overrideable by storage adapter ("mapper light").

- [FEATURE]: "Mappers" (`node-document-mapper`) - define how data should map between different databases. Major feature, will require a quite lot of work.

- [ENHANCEMENT]: Make `Document.storage` a property, to later support more advanced shortcuts like in the constructor:

      Post.storage = 'file:///app'

- [ADAPTER]: RethinkDB
      - http://rethinkdb.com/
      - http://rethinkdb.com/docs/install-drivers/javascript/

- [ADAPTER]: Dropbox API
      - https://www.dropbox.com/developers/datastore/docs/http
      - https://github.com/dpiccone/dropbox-datastore-node

- [ADAPTER]: Google Drive API
      - https://developers.google.com/drive/v2/reference/
      - https://github.com/niftylettuce/node-google-drive

- [ADAPTER]: Google Spreadsheet
      - https://github.com/samcday/node-google-spreadsheets

- [ADAPTER]: Google Fusiontables
      - https://www.google.com/fusiontables

- [ADAPTER]: OrientDB
      - https://github.com/gabipetrovay/node-orientdb
      - http://www.orientdb.org

- [ADAPTER]: ArangoDB
      - https://github.com/kaerus/arango-client
      - http://www.arangodb.org

- [ADAPTER]: AmazonSimpleDB
      - https://github.com/SaltwaterC/aws2js/wiki/SDB-Client
      - https://github.com/rjrodger/simpledb
      - http://aws.amazon.com/simpledb/

- [ADAPTER]: AmazonElastiCache
      - https://github.com/SaltwaterC/aws2js/wiki/ElastiCache-client
      - https://console.aws.amazon.com/elasticache/home?region=us-east-1

- [ADAPTER]: AmazonDynamoDB
      - https://github.com/SaltwaterC/aws2js/wiki/DynamoDB-Client
      - https://github.com/jed/dynamo
      - https://console.aws.amazon.com/dynamodb/home?region=us-east-1

- [ADAPTER]: Dropbox
      - https://github.com/sintaxi/node-dbox

- [ADAPTER]: RavenDB
      - https://github.com/tchype/node-ravendb

- [ADAPTER]: HBase
      - https://github.com/wdavidw/node-hbase

- [ADAPTER]: Cassandra
      - https://github.com/racker/node-cassandra-client
      - https://github.com/simplereach/helenus

- [ADAPTER]: Google Cloud Datastore
      - https://github.com/GoogleCloudPlatform/google-cloud-datastore

- [ADAPTER]: Google Storage
      - https://github.com/nodejitsu/pkgcloud#storage

- [ADAPTER]: Rackspace Storage
      - https://github.com/nodejitsu/pkgcloud#storage

- [ADAPTER]: Azure Storage
      - https://github.com/nodejitsu/pkgcloud#storage

- [REFACTOR]: Switch to lower-lever driver for `AmazonS3` storage adapter
      - https://github.com/SaltwaterC/aws2js/wiki/S3-Client

- [FEATURE]: "Middleware" API for storages, and such - e.g. `Post.use(storage_1).use(storage_2)` or `Post.storage.use(storage_1, storage_2)`.

- [REFACTOR]: var MyStorageAdapter = Storage.adapter('MyStorageAdapter', function() { /* ... */ })

- [ADAPTER/ENHANCEMENT]: Support SCP in `File` adapter, e.g. `fs://localhost:1234/tmp/{db}-{env}`

- [REFACTOR/HALTED]: Rewrite `Memcache` storage to driver that support SASL (auth) - tried MemJS but it contains serious race-condition bug. =S

- [REFACTOR]: Rewrite `ElasticSearch` storage using vanilla HTTP agent.
      - https://github.com/phillro/node-elasticsearch-client
      - https://github.com/fullscale/elastic.js

- [FEATURE/DISCUSS]: Support "versioning" for documents (Storage).
      * Always set `_version` property.

- [FEATURE/DISCUSS]: Wherever it makes sense; `Storage#index` + `Storage#unindex`

- [FEATURE/API]: Extend the Collection API with "Perform collection of operations". Possible enhancement: Merge it into bulk-request under the hood.
    - Example: Merge different operations on different document types into single bulk operation.

        [(new Post).save.build(...), (new Article).fetch.build(...), (new Post).fetch.build(...)].exec(function(err) {
          assert.typeOf (this, 'array');
          assert.instanceOf (this[0], Post);
          assert.instanceOf (this[1], Article);
          assert.instanceOf (this[2], Post);
        });

- [DOC]: Proper function documentation (JSDoc-style).
      - http://en.wikipedia.org/wiki/JSDoc
      - https://developers.google.com/closure/compiler/docs/js-for-compiler

- [FEATURE]: Support multiple endpoints - where applicable - for all storage adapters/clients.

- [REVIEW]: Support advanced URL convention (multiple hosts, options, etc.)
      - http://docs.mongodb.org/manual/reference/connection-string

- [ADAPTER]: SQLite3
      - https://github.com/developmentseed/node-sqlite3

- [ADAPTER]: PostgreSQL (HStore)
      - https://github.com/brianc/node-postgres
      - https://github.com/nonuby/node-postgres-hstore

- [ADAPTER]: MySQL
      - https://github.com/felixge/node-mysql
      - http://www.igvita.com/2010/03/01/schema-free-mysql-vs-nosql
      - https://github.com/igrigorik/em-proxy/blob/master/examples/schemaless-mysql/mysql_interceptor.rb


::::: MAYBE ::

- [FEATURE]: `#monitor` - shortcut for listening on DB operations (like Redis MONITOR).

- [ADAPTER/EXPERIMENT]: `Serializer` adapters support (...for storages where it makes sense)
      - JSON - DONE
      - BSON - DONE
      - MsgPack - DONE

- [ADAPTER/EXPERIMENT]: `Compressor` adapter support (...for storages where it makes sense)
      - Deflate: DONE
      - LZF - DONE
      - Snappy - DONE
      - ZIP/TAR - https://github.com/ctalkington/node-archiver

- [FEATURE]: Detect and handle lost connections in a reliable and automatic way. Fallback to other storage?

- [ADAPTER]: CouchBase
      - https://github.com/couchbase/couchnode
      - https://github.com/membase/manifest
      - https://github.com/clojurewerkz/spyglass/blob/master/.travis.yml

- [ADAPTER]: BerkelyDB
      - https://github.com/mcavage/node-bdb
            - OS X: './build/default/*' => './build/Release/*' (fork it)
            - https://npmjs.org/package/bindings - to fix the above problem
            - NOTE: Got it working standalone, but fails when installed via `npm install`

- [ADAPTER]: HyperDex
      - https://github.com/rescrv/HyperDex/tree/master/hyperclient/nodejs

- [ADAPTER]: REST (storage adapter)
      - https://github.com/danwrong/restler

- [REVIEW]: Get/Set/Del nested attribute values in some convenient way.