============================
Espra Protoplex Architecture
============================
-----
Nodes
-----
Espra Protoplex is specifically designed in order to be deployed in the context
of EC2, S3 and App Engine.
There are 7 types of nodes which will be running on top of EC2:
* Proxy
* Fileserver
* App
* Mail
* Live
* Seed
* Admin
These will be complemented by 2 App Engine applications:
* Espra
* EspraLog
Node Structure
==============
On startup all nodes establish a connection to the Seed node.
::
+----------------+
| Internet Horde |
+----------------+
| +-------------+
| +-----------+ | Other Nodes |
± | Seed Node | +-------------+
| +-----------+ |
| | \ |
+-------------+ | \ |
| Public Port | | +----------------------------------+
+-------------+ | | Meta Port (Internal Access Only) |
\ | +----------------------------------+
\ | /
\ | /
+===========\========|=====/=================================+
| \ | / |
| +----------------------+ |
| | Node: Parent Process | |
| +----------------------+ |
| | |
| | |
| +-----------------------+-----------------------+ |
| | | | |
| +---------------+ | +---------------+ |
| | Child Process | +---------------+ | Child Process | |
| +---------------+ | Child Process | +---------------+ |
| +---------------+ |
| |
+============================================================+
Proxy Nodes
===========
:Protocols: HTTP, HTTPS
:ELB: Yes
:LocalPorts: 8080, 8443
:RemotePorts: 80, 443
Proxy nodes are intelligent proxies to the Live nodes. They:
* Parse enough of the request according to predefined handlers.
* Query the Seed node to find out which particular Live node they should be
relaying the request to.
* Stop any further processing and simply proxy to and from the target Live node.
In order to facilitate high throughput, Proxy nodes will use the multi-process
single-threaded coroutines-based HTTP server.
Depending on whether they are in the ``us-east`` or ``eu-west`` region, the
Proxy nodes will respectively respond to requests on either
``us-1.live.espra.com`` or ``eu-1.live.espra.com``.
For scalability, the Proxy nodes will sit behind Auto Scaling enabled Elastic
Load Balancers (ELB) in both regions.
Fileserver Nodes
================
:Protocols: HTTP, HTTPS
:ELB: Yes
:LocalPorts: 8080, 8443
:RemotePorts: 80, 443
Fileserver nodes handle static files in a variety of different ways. Initially,
four specific handlers would be specified:
1. The ``AppFilesHandler`` will serve the main Espra app related assets from
memory, e.g. javascript, css, images, etc. These in-memory caches would be
invalidated when a new app build is pushed out by the Seed node.
2. The ``S3FilesHandler`` will look for the requested file in a local disk cache
before querying S3 for the source file if it's not found. If found, the
source file will be cached locally and (if appropriate) uncompressed before
being served as a response. If not found, the handler will register for an
update from the Live nodes and store the file key in an in-memory cache so as
to minimise unnecessary S3 requests.
3. The ``VhostFilesHandler`` will query the Main Datastore for the storage
reference for the file key and Host combination and then use the
S3FilesHandler to do the actual serving. Any found or not found storage
references will be saved in local caches and invalidated from registrations
from the Live nodes.
4. The ``UploadHandler`` will first validate the upload token sent with a POST
request. And if it's valid according to the Main Datastore, it will start
saving the uploaded file locally. As the upload progresses, a combined
(sha256+whirlpool) hash will be created and the Live nodes notified so that
upload progress can be relayed back to the uploader. Once the upload is
complete, the handler will in turn compress the file if it's compressible,
before updating the Main Datastore and uploading it to S3.
All responses will be aggressively cached with HTTP headers with a minimum
expiration set to at least 1 month. And the nodes will be using the same HTTP
server as the Proxy nodes and similarly sit behind an Auto Scaling ELB.
However, since latency isn't too critical an issue with file serving, the
Fileserver nodes and the S3 storage will only exist in the ``us-east`` region
and will respond to requests on ``*.espfile.com`` or appropriately CNAME'd
hosts.
App Nodes
=========
:Protocols: HTTP, HTTPS
:ELB: Yes
:LocalPorts: 8080, 8443
:RemotePorts: 80, 443
The App nodes are more CPU-bound and therefore will use a slightly different
server to the other nodes: a multi-process multi-threaded HTTP server.
Main Datastore
==============
:Protocols: HTTP, HTTPS
:RemotePorts: 80, 443
The Main Datastore application will be running on the ``espra`` App Engine
application. It will be accessed only via SSL and with a token and will have two minimal handlers:
This is where all the structured data will be stored and we rely on
App Engine to provide a query-able and
* Provide access to App Engine's Remote API for access by the various nodes.
Mail
Image
Datastore
Taskqueue
-------------
Remote Access
-------------
--------------
Load Balancing
--------------
Multi process
Queue
SSL
Update
Accounting/Quota
Planned Maintenance
GAE
* (Buildbot)
* Urlfetch
* Memcache
Thus will get load balanced by the kernel.
Worker
Queue
cpus -- core
I/O bound
Understudy
node roles
Failure and crashes
DNS
===
DNS for the various Espra domains will be delegated to the DNS services provided
by Linode and Slicehost.
This should be sufficient protection for now in case of extreme failure at
either provider. Both providers also offer relatively decent APIs which can be
used to update the zone records.
Support Services
================
A number of support services will be running on the Linode and Slicehost VPS
servers:
* Since the ``espra.com`` zone apex cannot be CNAME'd onto the ELB, it will
instead round-robin to Apache instances at the various VPS servers. Apache
will then redirect the request to the ``www.espra.com`` host on EC2.
* Off-site monitoring apps will test the responsiveness of the various node
services on EC2 as well as the App Engine applications. The data will be
logged locally and if any service stops responding, a priority SMS will be
sent to the Admins.
) is private.
