MarketDB

MDb.md

Proposed Architecture

Market data provider bridge

• Interface to the Market data vendor (RMDS, Bloomberg, etc.)
• Convert their data model into our own
  • Data fields
  • Image vs Refresh messages
  • Market by order, by depth
• Outputs tick by tick in unified data model

Tasks

• Specs the data model
• Map between input data model into new one
• Implement bridge
• Test

DB Store

• Take input from various data bridges
• Merge multiple sources
• Store raw data stream
• Create metadata and compress raw data
• Write to disk
• Push real time data out to message bus

Tasks

• Design data workflow
• Design raw data storage
• Analyze query pattern to make optimal storage and compression strategy
• Investigate best storage system that provides fault tolerance
• Convert from internal data model to output data model
• Implement writers
• Test

Aggregation Engine

• Read tick by tick data and produce min/bar, daily/bar
• Other synthetic metrics can be added: moving average, technical analysis indicators, etc.

Tasks

• Run as map/reduce task
• Spec tasks
• Implement job
• Setup cluster
• Test

Query Engine

• Subscriptions
• Non RT: Read from storage after job completes
• RT: Read from market data bus

Tasks

• Interpret and route requests based on their type
• Gather results and return with identical API
• Write TCP interface
• Write C++ interface
• Write Python on top of C++ interface

Monitoring

• Client to Query Engine
• Client to Cluster monitor

Tasks

• Dashboard (HTML5)
• Identify useful metrics
• Display
• Provide query/response panels

Project plan

Resources

• Hardware
  • 3 mid-size machines
  • Market data connectivity

Post

• Additional support for non market data (orders, etc.)
• Additional market data providers
• Elastic deployment