Qexec Queue and Server Flavors

queueoverview.txt

Flavors of queues in Qexec system topology:

|| Queue type: ||              Receives new tasks from:             ||              Routes tasks to:        || 
--------------------------------------------------------------------------------------------------------------
||   LQueue    || http PUT requests (via WSGI app)                  || RQueue (via ZMQ socket)              ||
||   RQueue    || polls an LQueue for new tasks (via ZMQ socket)    || Arbiter instances (via gevent queues)||
||   MQueue    || http PUT requests (via WSGI app)                  || Arbiter instances (via gevent queues)||

Flavors of servers:

|| Server type:|| Arbiter || Deluge || RQueue || LQueue || MQueue || 
--------------------------------------------------------------------
||   Worker    ||   yes   ||   no   ||   yes  ||   no   ||   no   || 
||   Router    ||   no    ||   yes  ||   no   ||   yes  ||   no   ||
||  Monolith   ||   yes   ||   yes  ||   no   ||   no   ||   yes  ||

MQueue formerly known as TaskQueue.  Rationale for breaking apart into RQueue and LQueue is to facilitate future
horizontal scaling. 

Qexec servers can be configured as either Routers, Workers, or Monoliths (currently the system only deploys
monoliths, but will eventually transition to only routers and workers).  

Routers each contain an LQueue, which receives new tasks via http requests mediated by a WSGI app.  Routers
also contain a Deluge and a Websocket to process and return data from ziplines spawned by delegated tasks.
Routers do not contain an Arbiter.

Workers each contain an RQueue, which receives new tasks from an LQueue over a ZeroMQ socket The RQueue-LQueue 
connection is over a ZeroMQ REQ/REP socket.  RQueue requests new tasks, LQueue replies.Bind is on the LQueue socket.
Workers can contain multiple Arbiters running in separate Greenlets (lightweight python-managed threads).
The worker's RQueue evenly distributes new tasks to its arbiter instances via Gevent queues.  Workers do not
contain a Deluge (results streaming from zipline get routed to the Deluge instance of the Router server that
processed the initial task).

Monoliths contain a single MQueue, as well as a Deluge and one or more Arbiters.  The MQueue receives new tasks 
over http in the same manner as an LQueue, and it distributes tasks to its Arbiter(s) in the same manner as an
RQueue.  A monolith performs essentially the same work as a Router/Worker system.