Building realtime application with DynamoDB (and potentially any database with an operation log)

phineas-intro.md

Building realtime applications is hard.

A scalable solution to this problem involves many cumbersome steps:

• Hooking into replication logs of the database servers, or writing custom data invalidating logic for realtime UI components.
• Adding messaging infrastructure (e.g. RabbitMQ) to your project.
• Writing sophisticated routing logic to avoid broadcasting every message to every web server.
• Reimplementing database functionality in the backend if your app requires realtime computation (e.g. realtime leaderboards).
  
  All this requires enormous commitment of time and engineering resources.

- RethinkDB: Advancing the realtime web

but at the same time it's increasingly mainstream. Most of the apps these days ranging from Quora, to Uber, to Tinder do realtime.

However, not every company has Quora's engineering resources to build their own realtime framework. That's why we have been working on phineas, with that aim of providing realtime functionality for your existing databases, as a service.

How is it different from X?

Firebase

Firebase is great, and is easy to get started up with. However, it provides fairly limited querying abilities. Also, Sometimes it's not possible model your data as a json store. Compared to that databases like DynamoDB provide more features, without any maintenance overhead.

Meteor

If you are looking for and end to end solution, Meteor is great. But it's a full blown framework. Phineas, however, aims to work with existing ecosystem of frontend tools and just provides simple Javascript API.

Building Apps with Phineas

To build an application with Phineas, you need to create a phineas project to work with your DynamoDB table. You can do that using the CLI. Then you can publish a set of DynamoDB queries, and subscribe to those queries on the client using the client side sdk. Once subscribed, you get the updates for any database changes that affect the result of that query. (At this point queries can be written only in Javascript.)

Let's say you have a messaging application. The messages are stored in a dynamodb table called 'Messages' Each item in table has a channel field, and createdAt field (and other fields for content). channel is the hash key, and and createdAt is ths sortKey. You can fetch a conversation by getting all the messages in a channel.

Example:

queries.js

/**
 * exports a query by the name of 'conversation' that, provided a channel, gets messages in that channel.
 You can subscibe to this query on the client.
 */
exports.conversation = function (channel) {
  var params = {
      "TableName": this.tableName,
      "KeyConditionExpression": "channel = :channel",
      "ExpressionAttributeValues": {
          ":channel": {"S": channel}
      }
  };

  return this.dynamodb.query(params);
}

app.js

const Phineas = require('phineas-sdk')

// appID and secret available from web UI 
const app = Phineas.inititalize({
  appID: <app_id>
  secret: <secret>
})

// 'Messages' is the name of DynamoDB table
const Messages = app.table('Messages')

const channel = 'alice#bob'  // get the conversation between alice and bob
const queryToSubscribeTo = 'conversation'
const subscription = Messages.subscribe(queryToSubscribeTo, channel, function (err, data) {
  // first time data load
  if(err) {
    throw err
  } 

  console.log(data)
})

// updates
subscription
  .on('INSERT', function (event) {
    console.log('new item added', event.newItem)
  })
  .on('MODIFY', function (e) {
    console.log('item modified', e.oldItem, e.newItem)
  })
  .on('REMOVE', function (e) {
    console.log('item deleted', e.oldItem)
  })
  

Note that this does not include any code to write to the database. You can use the Javascript client, or your traditional server to do that. As long as the data changes, you'll get the update on clients.

A demo of how to create a project to work with your DynamoDB table and realtime functionality is available here: Screencast

---

The project is still in very initial stages of development, and is definitely not ready for production. The site could use a lot of improvements. However, do give it a try and let us know your thoughts on the same!

---

_{Links: cli | Javascript SDK | Web Console}