document management diagrams

basic_deployment_

Basic document management deployment.

eapp and isms both use inforouter for document management facilities; the front-ends for these are scaled 
independent of the UI front-ends. The only bottleneck and single point of failure is the metadata db, 
which by Inforouter's definition must be a single database.

The inforouter fs cluster is still slightly undefined at the moment. I'm hoping we can use any standard filesystem 
providers. If we were to use s3, we may need an arbitration subsystem to act as a gatekeeper and translate inforouter calls
into HTTP REST calls.

      ______________                       ____________________
     /              \                     /                    \
     | API Client   |                     | Document           |
     | (eapp, isms) |                     | Administrators (?) |
     |              |                     |                    |
     \______________/                     \____________________/
              ^                                  ^            
              |                                  |            
   ___________|__________________________________|__________________________
  /           |                                  |                          \ 
  |         HTTPS                              HTTPS              Documents |     
  |           |                                  |         .region-id.docs. |
  |         __v__                              __v__                        |
  |        /     \                            /     \                       |
  |        | ELB |                            | ELB |                       |
  |        \_____/                            \_____/                       |
  |           ^                                  ^                          |
  |  _________|______________________    ________|_______________________   |
  | /         |                      \  /        |                       \  |
  | |       HTTP         API cluster |  |       HTTP       Admin cluster |  |
  | |         |                .api. |  |        |               .admin. |  |
  | |         |                      |  |        |                       |  |
  | |    +----+---------+            |  |    +---+----------+            |  |
  | |    |              |            |  |    |              |            |  |
  | |  __v_________   __v__________  |  |  __v_________   __v__________  |  |
  | | /            \ /             \ |  | /            \ /             \ |  |
  | | |      1-api | |      n-api  | |  | |    1-admin | |    n-admin  | |  |
  | | | 1.iis.web. | |  n.iis.web. | |  | | 1.iis.web. | |  n.iis.web. | |  |
  | | \____________/ \_____________/ |  | \____________/ \_____________/ |  |
  | |       ^   ^           ^   ^    |  |       ^   ^       ^       ^    |  |
  | |       |   +-----------|---+---------------|---+-------|-------+    |  |
  | |       |               |        |  |       |           |       |    |  |
  | \_______|_______________|________/  \_______|___________|_______|____/  |
  |         |               |                   |           |       |       |
  |         +-------+-------+- mysql or sqlsrv -+-----------+    gluster    |
  |                 |                                             iscsi     |
  |         ________v_____________                                http      |
  |        /                      \                               samba     |
  |        |          Metadata db |              ___________________v____   |
  |        |  metadata.sqlsrv.db. |             /                        \  |
  |        \______________________/             |    File server cluster |  |   
  |                                             |                  n.fs. |  | 
  |                                             \________________________/  |
  \_________________________________________________________________________/

mixed_environment_deployment.txt

Mixed environment deployment.

Everything from the aspect of isms is still purely inforouter, but there is now only one primary API client. The 
other API client, the eapp, uses its own document store (which I've sketched out here) and documents are
imported into isms. Any inforouter-specific status information must then be exported back to the student portal
and its corresponding document updated.

The eapp fs cluster could be a hardware SAN, a software-based SAN using gluster or dfs, a straight set of NFS servers or
a hosted service akin to S3.

The inforouter fs cluster is still slightly undefined at the moment. I'm hoping we can use any standard filesystem 
providers. If we were to use s3, we may need an arbitration subsystem to act as a gatekeeper and translate inforouter calls
into HTTP REST calls.

 _____________________________________________                        _______________                      ____________________
/                                             \                      /               \                    /                    \
|                                         AWS |                      | API Client    |                    | Document           |
|                             .region-id.aws. |      +---------------> isms, import, |                    | Administrators (?) |
|                                             |      |               | export        |                    |                    |
|     _______                                 |      |               \_______________/                    \____________________/
|    /       \                                |      |                        ^                                  ^            
| +--> web-1 <--+                             |      |                        |                                  |            
| |  \_______/  |                             |      |             ___________|__________________________________|__________________________
| |   _______   |- gluster, dfs, nfs, http    |    HTTPS          /           |                                  |                          \
| |  /       \  |          |                  |      |            |         HTTPS                              HTTPS              Documents |
| +--> web-n <--+    ______v_______           |      |            |           |                                  |         .region-id.docs. |
| |  \_______/      /              \          |      |            |         __v__                              __v__                        |
| |                 |  fs cluster  |          |      |            |        /     \                            /     \                       |
| |                 \______________/          |      |            |        | ELB |                            | ELB |                       |
| |                             ^             |      |            |        \_____/                            \_____/                       |
| +- redis, cassandra, hbase    |             |      |            |           ^                                  ^                          |
|              |               _v___________  |      |            |  _________|______________________    ________|_______________________   |
|      ________v_________     /             \ |      |            | /         |                      \  /        |                       \  |
|     /                  \    |   imp/exp   | |      |            | |       HTTP         API cluster |  |       HTTP       Admin cluster |  |
|     | metadata cluster <---->   worker    <--------+            | |         |                .api. |  |        |               .admin. |  |
|     \__________________/    |   cluster   | |                   | |         |                      |  |        |                       |  |
|                             \_____________/ |                   | |    +----+---------+            |  |    +---+----------+            |  |
\_____________________________________________/                   | |    |              |            |  |    |              |            |  |
                                                                  | |  __v_________   __v__________  |  |  __v_________   __v__________  |  |
                                                                  | | /            \ /             \ |  | /            \ /             \ |  |
                                                                  | | |      1-api | |      n-api  | |  | |    1-admin | |    n-admin  | |  |
                                                                  | | | 1.iis.web. | |  n.iis.web. | |  | | 1.iis.web. | |  n.iis.web. | |  |
                                                                  | | \____________/ \_____________/ |  | \____________/ \_____________/ |  |
                                                                  | |       ^   ^           ^   ^    |  |       ^   ^       ^       ^    |  |
                                                                  | |       |   +-----------|---+---------------|---+-------|-------+    |  |
                                                                  | |       |               |        |  |       |           |       |    |  |
                                                                  | \_______|_______________|________/  \_______|___________|_______|____/  |
                                                                  |         |               |                   |           |       |       |
                                                                  |         +-------+-------+- mysql or sqlsrv -+-----------+    gluster    |
                                                                  |                 |                                             iscsi     |
                                                                  |         ________v_____________                                http      |
                                                                  |        /                      \                               samba     |
                                                                  |        |          Metadata db |                          _______v____   |
                                                                  |        |  metadata.sqlsrv.db. |                         /            \  |
                                                                  |        \______________________/                         | fs cluster |  |
                                                                  |                                                         |      n.fs. |  |
                                                                  |                                                         \____________/  |
                                                                  \_________________________________________________________________________/

mixed_environment_with_s3.txt

Mixed environment deployment using s3 as a backend.

In this environment, the storage backend is shared by both environments, and the only specific element to each environment
is the metadata layer describing the files. In this scenario the api clients remain the same, but rather than shipping actual
files, the shipment is either a reference creation, status update, or removal notification. Due to this smaller data size, changes to
to documents could be shipped in real time as opposed to batching them with app shipments.

The only question as to how this will work is whether inforouter will accept using a remote service as for its "warehouse" file server
cluster. If it doesn't, we may need to write an arbitration subsystem (sketched out below) to facilitate this -- to translate inforouter
API calls into Amazon S3 REST API calls.

 _____________________________________________                        _______________                      ____________________
/                                             \                      /               \                    /                    \
|                                         AWS |                      | API Client    |                    | Document           |
|                             .region-id.aws. |      +---------------> isms, import, |                    | Administrators (?) |
|                                             |      |               | export        |                    |                    |
|     _______                                 |      |               \_______________/                    \____________________/
|    /       \               _____________    |      |                        ^                                  ^            
| +--> web-1 <-+            /             \   |      |                        |                                  |            
| |  \_______/ |            |   imp/exp   |   |      |             ___________|__________________________________|__________________________
| |   _______  |       +---->   worker    <------ HTTPS           /           |                                  |                          \
| |  /       \ |       |    |   cluster   |   |                   |         HTTPS                              HTTPS              Documents |
| +--> web-n <-+       |    \_____________/   |                   |           |                                  |         .region-id.docs. |
| |  \_______/ |       |                      |                   |         __v__                              __v__                        |
| |          redis     |                      |                   |        /     \                            /     \                       |
| |        cassandra   |                      |                   |        | ELB |                            | ELB |                       |
| |          hbase     |                      |                   |        \_____/                            \_____/                       |
| |            |       |                      |                   |           ^                                  ^                          |
| |            |       |                      |                   |  _________|______________________    ________|_______________________   |
| |    ________v_______v_                     |                   | /         |                      \  /        |                       \  |
| |   /                  \                    |                   | |       HTTP         API cluster |  |       HTTP       Admin cluster |  |
| |   | metadata cluster |                    |                   | |         |                .api. |  |        |               .admin. |  |
| |   \__________________/                    |                   | |         |                      |  |        |                       |  |
| |                                           |                   | |    +----+---------+            |  |    +---+----------+            |  |
\_|___________________________________________/                   | |    |              |            |  |    |              |            |  |
  |                                                               | |  __v_________   __v__________  |  |  __v_________   __v__________  |  |
  |                                                               | | /            \ /             \ |  | /            \ /             \ |  |
HTTPS                                                             | | |      1-api | |      n-api  | |  | |    1-admin | |    n-admin  | |  |
  |                                                               | | | 1.iis.web. | |  n.iis.web. | |  | | 1.iis.web. | |  n.iis.web. | |  |
  |             ____________________                              | | \____________/ \_____________/ |  | \____________/ \_____________/ |  |
  |            /                    \                             | |       ^   ^           ^   ^    |  |       ^   ^       ^       ^    |  |
  +------------>     Amazon S3      |                             | |       |---+-----------|---+---------------|---+-------|-------+    |  |
               \____________________/                             | |       |               |        |  |       |           |       |    |  |
                       ^                                          | \_______|_______________|________/  \_______|___________|_______|____/  |
                       |                                          |         |               |                   |           |       |       |
                       |                                          |         +-------+-------+- mysql or sqlsrv -+-----------+       |       |
                       |                                          |                 |                                             HTTP      |
                       |                                          |         ________v_____________                                  |       |
                       |                                          |        /                      \                                 |       |
                       |                                          |        |          Metadata db |                         ________v___    |
                       |                                          |        |  metadata.sqlsrv.db. |                        /            \   |
                       |                                          |        \______________________/                        |  IF -> S3  |   |
                       +------------------------ HTTPS --------------------------------------------------------------------> API bridge |   |
                                                                  |                                                        \____________/   |
                                                                  \_________________________________________________________________________/