sid597/Athens RTC

## Athens RTC
- Architecture :
    - Current self-hosted Athens app architecture :
        - Athens is built using re-frame + datascript ( as the frontend in memory db). The general architecture can be understood by
          learning re-frame, in addition to that there is a parser and file system. Parser is used to parse the block text and
          filesystem is there to make the Athens pages durable by saving them on disk.
    - How to make the self-hosted Athens app RTC
        - Approach 1 : Instead of using the local datascript db we map all the clients to a remote db which takes up the role of local
                       datascript db.
            - Overview :
                - First some terminology :
                    - Transaction : The data which is sent to be stored in the db, in re-frame this refers to the effects handling.
                    - Client : Basically a user running a self-hosted Athens app
                    - To process something : Do something with it.
                - Any transaction made by some client is sent to the remote db.
                - Remote db processes this transaction  and then broadcasts this to all the other connected clients.
                - Clients receive transactions from the remote db and process it.
            - Pros :
                - Easier to implement than Approach 2. Can implement fast for the MVP
                - Would get the job done for current Athens users (the community itself and the enterprise customer)
            - Cons :
                - Need to be connected to internet to make progress
                - Cannot scale horizontally
                -
            - What happens when :
                - A new client is added to the remote db?
            - Implementation :
                - How can a client have stale data and how does the system update it?
                    - Client can have stale data if it is connected to the db server but due to network issues is not able to receive
                      broadcasted transactions. The staleness can be removed by adding heartbeat mechanism to the client side.
                - What is heartbeat mechanism ?
                    - It is a mechanism which can be used to check which clients are still connected and to bring up stale clients on
                      track.
                    - It is possible to send heartbeat from db to all clients and vice-versa, which approach should we take?
                        - Db to clients : Should not take this
                            - Would not be efficient for large no. of clients
                        - Clients to Db
                            - After some fixed time client sends a heartbeat to db which also contains the information of the log index
                              of the last transaction which was received from db by this client.
                - Remote db maintains:
                    - A log (vector) which keeps all the transactions received from various clients
                        - To be persisted so that in case of machine failure state  can be rebuilt by replaying this log. Another way
                          to make this persistent could be to use a db that saves data on disk.
                    - A map of all the clients and the log index of the last transaction that was sent to that client
                        - Don't need this to be persisted because all the clients would be just like a new client is added to the
                          remote db.
                    - A map of all the clients and the socket they are connected to.
                    - A map to keep track of last time heartbeat from a client was received.
                - Clients maintain :
                    - Db's log index of  the last received transaction.

                        - Why? checkout ((0ad9dae10))
                - Cogs of the db server:
                    - Frontend : This part receives the transaction and the heartbeat. And based on the type takes decision.
                        - If transaction received :
                            - Compares this with the last transaction received and if both relate to same block reject the new
                              transaction and if not then add this to the log. Also update the map that keeps track of last heartbeat
                              received time
                        - If heartbeat :
                            - update the map that keeps track of last heartbeat received time  and map that keeps track of last
                              transaction that was sent to that client
                    - Backend : Processes the transaction, I don't know what processing is to be done
                    - Broadcaster : After a transaction is processed broadcast the end result to all the clients
                        - For each client broadcaster first checks if that client is still connected, if not it is removed and
                          broadcaster moves on to next client
                        - If the client is connected then, all the transactions since the last transaction sent to that server are
                          fetched and then send to the client along with the latest db's log index
                        - Note :  The map that keeps track of last transaction that was sent to that client is only updated when
                          heartbeat from that client is received
                - How does client send a transaction to remote db?
                    - Clients knows the address to which it can connect, after establishing a connection all the transactions can be
                      sent to that socket.
                - What happens when a client receives a broadcast from db?
                    - Check if the transactions received in the broadcast are new or are they already present in the client.
                      This can be done by checking the db's current log index (which is included in the broadcast tx) and the
                      corresponding value the client has. If both are same then client rejects this transaction.
                    - If not then client can add all the broadcast transactions to its queue from where re-frame would take over.
                    Also update the Db's log index of  the last received transaction.
                - Is a client blocked till a response for its last transaction is not received?
                    - Yes
                - Types of failures that can occur in this system?
                    - Network failures between clients and db
                    - Db server can fail
            - Questions related to implementation :
                - What happens in case same block is edited by multiple clients?
                - Why reject all but one transaction to the same block? Why not just append all the blocks in some order?
        - Approach 2 : Write to local db, this local db is synced to a remote db, any conflicts are automatically resolved.
            Don't know how to implement this
	- Architecture :
	- Current self-hosted Athens app architecture :
	- Athens is built using re-frame + datascript ( as the frontend in memory db). The general architecture can be understood by
	learning re-frame, in addition to that there is a parser and file system. Parser is used to parse the block text and
	filesystem is there to make the Athens pages durable by saving them on disk.
	- How to make the self-hosted Athens app RTC
	- Approach 1 : Instead of using the local datascript db we map all the clients to a remote db which takes up the role of local
	datascript db.
	- Overview :
	- First some terminology :
	- Transaction : The data which is sent to be stored in the db, in re-frame this refers to the effects handling.
	- Client : Basically a user running a self-hosted Athens app
	- To process something : Do something with it.
	- Any transaction made by some client is sent to the remote db.
	- Remote db processes this transaction and then broadcasts this to all the other connected clients.
	- Clients receive transactions from the remote db and process it.
	- Pros :
	- Easier to implement than Approach 2. Can implement fast for the MVP
	- Would get the job done for current Athens users (the community itself and the enterprise customer)
	- Cons :
	- Need to be connected to internet to make progress
	- Cannot scale horizontally
	-
	- What happens when :
	- A new client is added to the remote db?
	- Implementation :
	- How can a client have stale data and how does the system update it?
	- Client can have stale data if it is connected to the db server but due to network issues is not able to receive
	broadcasted transactions. The staleness can be removed by adding heartbeat mechanism to the client side.
	- What is heartbeat mechanism ?
	- It is a mechanism which can be used to check which clients are still connected and to bring up stale clients on
	track.
	- It is possible to send heartbeat from db to all clients and vice-versa, which approach should we take?
	- Db to clients : Should not take this
	- Would not be efficient for large no. of clients
	- Clients to Db
	- After some fixed time client sends a heartbeat to db which also contains the information of the log index
	of the last transaction which was received from db by this client.
	- Remote db maintains:
	- A log (vector) which keeps all the transactions received from various clients
	- To be persisted so that in case of machine failure state can be rebuilt by replaying this log. Another way
	to make this persistent could be to use a db that saves data on disk.
	- A map of all the clients and the log index of the last transaction that was sent to that client
	- Don't need this to be persisted because all the clients would be just like a new client is added to the
	remote db.
	- A map of all the clients and the socket they are connected to.
	- A map to keep track of last time heartbeat from a client was received.
	- Clients maintain :
	- Db's log index of the last received transaction.

	- Why? checkout ((0ad9dae10))
	- Cogs of the db server:
	- Frontend : This part receives the transaction and the heartbeat. And based on the type takes decision.
	- If transaction received :
	- Compares this with the last transaction received and if both relate to same block reject the new
	transaction and if not then add this to the log. Also update the map that keeps track of last heartbeat
	received time
	- If heartbeat :
	- update the map that keeps track of last heartbeat received time and map that keeps track of last
	transaction that was sent to that client
	- Backend : Processes the transaction, I don't know what processing is to be done
	- Broadcaster : After a transaction is processed broadcast the end result to all the clients
	- For each client broadcaster first checks if that client is still connected, if not it is removed and
	broadcaster moves on to next client
	- If the client is connected then, all the transactions since the last transaction sent to that server are
	fetched and then send to the client along with the latest db's log index
	- Note : The map that keeps track of last transaction that was sent to that client is only updated when
	heartbeat from that client is received
	- How does client send a transaction to remote db?
	- Clients knows the address to which it can connect, after establishing a connection all the transactions can be
	sent to that socket.
	- What happens when a client receives a broadcast from db?
	- Check if the transactions received in the broadcast are new or are they already present in the client.
	This can be done by checking the db's current log index (which is included in the broadcast tx) and the
	corresponding value the client has. If both are same then client rejects this transaction.
	- If not then client can add all the broadcast transactions to its queue from where re-frame would take over.
	Also update the Db's log index of the last received transaction.
	- Is a client blocked till a response for its last transaction is not received?
	- Yes
	- Types of failures that can occur in this system?
	- Network failures between clients and db
	- Db server can fail
	- Questions related to implementation :
	- What happens in case same block is edited by multiple clients?
	- Why reject all but one transaction to the same block? Why not just append all the blocks in some order?
	- Approach 2 : Write to local db, this local db is synced to a remote db, any conflicts are automatically resolved.
	Don't know how to implement this