petmongrels/gist:211cd482ad908265a0fd80c9a4f2a9c6

## gistfile1.txt
Increasing device memory from 2GB to 3 GB has no effect on the performance.

The time taken for a page can be divided into following sections.
1. Database query
2. Server converting the jdbc result-set to JSON
3. Client receiving the data from server
4. Client deserialising the JSON on getting the response
5. Client saving the data onto the offline database


Network speed - 7 mbps, 3G/4G

Page Size = 100
---------------
- 1+2+3 takes 4-5 seconds
- 1+2+3 takes 6 seconds (reducing network from 7mbps to 200kbps)
- 4 negligible < 10 ms
- 5 200 ms (performance doesn't degrade with increase in database size)
Total time taken 3 hours (Joy's results, my results are agreeing with his)

Page Size = 500
----------------
- 1+2+3 takes around 5 seconds
- 1+2+3 takes 9 seconds (on reducing network from 7mbps to 200kbps)
- 4 negligible < 10 ms
- 5 5 SECONDS!!

Total time taken 37 min (Joy's results)


Page Size = 1000
----------------
- 1+2+3 takes around 5 seconds
- 1+2+3 takes 10-11 seconds (on reducing network from 7mbps to 200kbps)
- 4 negligible < 10 ms
- 5 10-11 SECONDS!!


Conclusions
-----------
- 1,2 take more or less constant time
- 3 takes more or less constant time with good network
- 3 on bad network, has visible negative impact, but it is likely scaling linearly
- 5 scales poorly, non-linearly and significantly with increase in page size. save of 100 records = 200 ms. save of 1000 records should have been 2 seconds but is 10-11 seconds.


What can we do?
--------------
- Improve 5 and increase page size to 1000
We save the entire page in a single transaction. We can break this down into smaller number of transactions. Say to 100 per txn. We can tune it once we have the ability to save a page in multiple txns.

- Parallelize 1+2+3 and 4+5
While the client is busy saving the previous request we can fire the next request. In the time the response comes back from the server the client would be busy saving the previous one and having done most of it.


How much improvement we can see with this
-----------------------------------------
Calculation - For 500 page size.

Based on improving 5
- Assuming 50-50 split between server and client, 37 minutes becomes, 19-19 minutes for each.
- Each page processing on client can be improved from 5 seconds to 1 second.
- Hence 19 minutes on client can be reduced to 4 minutes, bringing total time down to 23 minutes. Purchase here may be more when increase the page size to 1000.

Based on parallelising
- Theoretically we can get done in 19 minutes for 500 page size.
- GOing to page size 1000 wouldn't help the overall time because server kind of takes constant time. And if that is the case, improving 5 is not of much value for good network, but may be useful in slower network.

More numbers
------------
- When I ran with page size 500, without any changes the sync took 70 minutes. So the improvements made by above changes would be lesser.
- It seems the sync for entire district could take upto 7 hours currently, 100 page size. This could go upto 28 hours if the entire district is registered, which is the plan. The above optimisations could only reduce it to 4-5 hours. Still not good enough for such use cases.
	Increasing device memory from 2GB to 3 GB has no effect on the performance.

	The time taken for a page can be divided into following sections.
	1. Database query
	2. Server converting the jdbc result-set to JSON
	3. Client receiving the data from server
	4. Client deserialising the JSON on getting the response
	5. Client saving the data onto the offline database


	Network speed - 7 mbps, 3G/4G

	Page Size = 100
	---------------
	- 1+2+3 takes 4-5 seconds
	- 1+2+3 takes 6 seconds (reducing network from 7mbps to 200kbps)
	- 4 negligible < 10 ms
	- 5 200 ms (performance doesn't degrade with increase in database size)
	Total time taken 3 hours (Joy's results, my results are agreeing with his)

	Page Size = 500
	----------------
	- 1+2+3 takes around 5 seconds
	- 1+2+3 takes 9 seconds (on reducing network from 7mbps to 200kbps)
	- 4 negligible < 10 ms
	- 5 5 SECONDS!!

	Total time taken 37 min (Joy's results)


	Page Size = 1000
	----------------
	- 1+2+3 takes around 5 seconds
	- 1+2+3 takes 10-11 seconds (on reducing network from 7mbps to 200kbps)
	- 4 negligible < 10 ms
	- 5 10-11 SECONDS!!


	Conclusions
	-----------
	- 1,2 take more or less constant time
	- 3 takes more or less constant time with good network
	- 3 on bad network, has visible negative impact, but it is likely scaling linearly
	- 5 scales poorly, non-linearly and significantly with increase in page size. save of 100 records = 200 ms. save of 1000 records should have been 2 seconds but is 10-11 seconds.


	What can we do?
	--------------
	- Improve 5 and increase page size to 1000
	We save the entire page in a single transaction. We can break this down into smaller number of transactions. Say to 100 per txn. We can tune it once we have the ability to save a page in multiple txns.

	- Parallelize 1+2+3 and 4+5
	While the client is busy saving the previous request we can fire the next request. In the time the response comes back from the server the client would be busy saving the previous one and having done most of it.



	How much improvement we can see with this
	-----------------------------------------
	Calculation - For 500 page size.

	Based on improving 5
	- Assuming 50-50 split between server and client, 37 minutes becomes, 19-19 minutes for each.
	- Each page processing on client can be improved from 5 seconds to 1 second.
	- Hence 19 minutes on client can be reduced to 4 minutes, bringing total time down to 23 minutes. Purchase here may be more when increase the page size to 1000.

	Based on parallelising
	- Theoretically we can get done in 19 minutes for 500 page size.
	- GOing to page size 1000 wouldn't help the overall time because server kind of takes constant time. And if that is the case, improving 5 is not of much value for good network, but may be useful in slower network.

	More numbers
	------------
	- When I ran with page size 500, without any changes the sync took 70 minutes. So the improvements made by above changes would be lesser.
	- It seems the sync for entire district could take upto 7 hours currently, 100 page size. This could go upto 28 hours if the entire district is registered, which is the plan. The above optimisations could only reduce it to 4-5 hours. Still not good enough for such use cases.