This doc intends to discuss preformance improvements for AeroGear SimplePush Server. Basically we are trying to figure out where we should focus to begin with.
This will start by picking an action that is performed by a client and profile for CPU perfomance, memory utilization and thread usage. After making improvements in that area, meansurement will be done again to see what the "hotspot" is after this.
This branch contains tests that can be run against an external server. Please see the "Integration Testing" section for details how to run the tests.
The SimplePush Server used in the Netty based implementation deployed to AS 7.1.1.Final. VisualVM is used to gather performance meansurements. Before executing the tests CPU profiling was enabled a java package filter (org.jboss.aerogear). After the tests completed a snaphost was taken in visualvm.
The following tests was run to gather measurements for sending Hello messages. This is the handshake message in the SimplePush specificiation and is performed once per connection to the server.
mvn exec:java -Dexec.args="ws://localhost:7777/simplepush/websocket 600 100" -Phello
The first parameter is the server to connect to. The second is the number of threads to use. Each thread will create a connection and then execute one hello message. The last 100 is to give the JVM a warmup and will send 100 hello messages using a single thread.
Output from running test:
Starting warmup of 100 operations
Warmup done.
Total: 13574 ms for 600 operations
Average: 22 ms per operations
Hotspots:
The hotspots shown below are filtered by package name org.jboss.aerogear to only focus on our code.
Backtrace of first hotspot:
The method getChannelIds uses the JPA to connecto the underlying database to retrieve the channels belonging to the useragent id passed in. This is all going to take time and while I'm pretty sure we might be able to improve the code that is there it might only take us to a certain point after which we won't be able to do anything more.
We have discussed using a rusing a non-relational database to hopefully improve performance and scalability of the server. This mailing list discussion brought up the idea that we might just look into a non-relational database instead of spending/wasting time trying to improve the current solution.
The current version of the server can also only be run separately as one instance, it will not know about any other instances that exist and they would be completely separate. This will not work if we want to scale out and allow mulitple server handle requests.
This test spins up a number of threads and then fires a number of notification requests.
mvn exec:java -Dexec.args="ws://localhost:7777/simplepush/websocket 10 100 100" -Pnotify
Starting warmup of 100 operations
Warmup done.
Total: 2713 ms for 1000 operations using 10 threads
Average: 271 ms (271332100 ns) per operations
Hotspots:
The hotspots shown below are filtered by package name org.jboss.aerogear to only focus on our code.
Backtrace:
- Use a non-relation database
- Investigate thread usage to support more requests
Using Redis as a data store
This was done to figure out what would be required and find issues around using a non-relational database for storage. The redis branch was used for experimenting. The branch uses Redis 2.6.16 and running the test requires a running Redis server listening to localhost on port 6379.
The same test as was run before but this time using Redis.
mvn exec:java -Dexec.args="ws://localhost:7777/simplepush/websocket 600 100" -Phello
The first parameter is the server to connect to. The second is the number of threads to use. Each thread will create a connection and then execute one hello message. The last 100 is to give the JVM a warmup and will send 100 hello messages using a single thread.
Output from running test:
Starting warmup of 100 operations
Warmup done.
Total: 10004 ms for 600 operations
Average: 16 ms per operations
Hotspots:
The hotspots shown below are filtered by package name org.jboss.aerogear to only focus on our code.
Backtrace of first hotspot:
This test spins up a number of threads and then fires a number of notification requests.
mvn exec:java -Dexec.args="ws://localhost:7777/simplepush/websocket 10 100 100" -Pnotify
Starting warmup of 100 operations
Warmup done.
Total: 2167 ms for 1000 operations using 10 threads
Average: 2 ms per operations
Hotspots:
The hotspots shown below are filtered by package name org.jboss.aerogear to only focus on our code.
One thing to notice when comparing the hotspots from the previous version that uses JPA is that the decryption is no longer a hotspot. The reason for this is that the we only encrypt the 'endpointToken' and store it in Redis. Instead of decrypting it as we did before to get at the uaid and channelId we use the endpointToken to look up that information instead. This would probably have improved the JPA version as well but the encryption/decryption as added quite late and there was not time to consider this at that point.
The above solution with Redis is far from perfect and was my first time using Redis. But I think it look promising and worth persuing. The choice of which non-relational database is probably a question that need to be answered before we spend any time on an implementation.