During each release, we execute various automated performance test scenarios and publish the results.
| Test Scenarios | Description |
|---|---|
| Auth Code Grant Redirect With Consent | Obtain an access token using the OAuth 2.0 authorization code grant type. |
| SAML2 SSO Redirect Binding | Obtain a SAML 2 assertion response using redirect binding. |
Our test client is Apache JMeter. We test each scenario for a fixed duration of time and split the test results into warm-up and measurement parts and use the measurement part to compute the performance metrics. For this particular instance, the duration of each test is 15 minutes and the warm-up period is 5 minutes.
We run the performance tests under different numbers of concurrent users and heap sizes to gain a better understanding on how the server reacts to different loads.
The main performance metrics:
- Throughput: The number of requests that the WSO2 Identity Server processes during a specific time interval (e.g. per second).
- Response Time: The end-to-end latency for a given operation of the WSO2 Identity Server. The complete distribution of response times was recorded.
In addition to the above metrics, we measure the load average and several memory-related metrics.
The following are the test parameters.
| Test Parameter | Description | Values |
|---|---|---|
| Scenario Name | The name of the test scenario. | Refer to the above table. |
| Heap Size | The amount of memory allocated to the application | 2G |
| Concurrent Users | The number of users accessing the application at the same time. | 50, 100, 150, 300, 500 |
| Tenants | The number of tenants active druing the test. | |
| IS Instance Type | The AWS instance type used to run the Identity Server. | c5.xlarge |
The following are the measurements collected from each performance test conducted for a given combination of test parameters.
| Measurement | Description |
|---|---|
| Error % | Percentage of requests with errors |
| Average Response Time (ms) | The average response time of a set of results |
| Standard Deviation of Response Time (ms) | The Standard Deviation of the response time. |
| 99th Percentile of Response Time (ms) | 99% of the requests took no more than this time. The remaining samples took at least as long as this |
| Throughput (Requests/sec) | The throughput measured in requests per second. |
| Average Memory Footprint After Full GC (M) | The average memory consumed by the application after a full garbage collection event. |
The following is the summary of performance test results collected for the measurement period.
| Scenario Name | Concurrent Users | Tenants | Label | Error % | Throughput (Requests/sec) | Average Response Time (ms) | Standard Deviation of Response Time (ms) | 99th Percentile of Response Time (ms) | WSO2 Identity Server GC Throughput (%) |
|---|---|---|---|---|---|---|---|---|---|
| Auth Code Grant Redirect With Consent | 50 | 1 | Authorize Request | 0 | 133.22 | 79.58 | 46.49 | 217 | 98.24 |
| Auth Code Grant Redirect With Consent | 50 | 1 | Common Auth Login HTTP Request | 0 | 133.2 | 57.37 | 28.25 | 140 | 98.24 |
| Auth Code Grant Redirect With Consent | 50 | 1 | Get access token | 0 | 133.22 | 183.25 | 50.36 | 323 | 98.24 |
| Auth Code Grant Redirect With Consent | 50 | 1 | Send request to authorize end poiont | 0 | 133.18 | 54.27 | 34.91 | 158 | 98.24 |
| Auth Code Grant Redirect With Consent | 100 | 1 | Authorize Request | 0 | 142.02 | 137.06 | 58 | 295 | 97.99 |
| Auth Code Grant Redirect With Consent | 100 | 1 | Common Auth Login HTTP Request | 0 | 141.96 | 118.11 | 49.31 | 269 | 97.99 |
| Auth Code Grant Redirect With Consent | 100 | 1 | Get access token | 0 | 142 | 355.8 | 85.87 | 579 | 97.99 |
| Auth Code Grant Redirect With Consent | 100 | 1 | Send request to authorize end poiont | 0 | 142 | 92.55 | 47.27 | 243 | 97.99 |
| Auth Code Grant Redirect With Consent | 150 | 1 | Authorize Request | 0 | 145.6 | 182.45 | 68.79 | 393 | 97.85 |
| Auth Code Grant Redirect With Consent | 150 | 1 | Common Auth Login HTTP Request | 0 | 145.68 | 178.02 | 70.65 | 387 | 97.85 |
| Auth Code Grant Redirect With Consent | 150 | 1 | Get access token | 0 | 145.54 | 529.76 | 119.63 | 815 | 97.85 |
| Auth Code Grant Redirect With Consent | 150 | 1 | Send request to authorize end poiont | 0 | 145.68 | 139.42 | 65.95 | 367 | 97.85 |
| Auth Code Grant Redirect With Consent | 300 | 1 | Authorize Request | 0 | 136.57 | 396.64 | 179.57 | 963 | 97.75 |
| Auth Code Grant Redirect With Consent | 300 | 1 | Common Auth Login HTTP Request | 0 | 136.5 | 428.29 | 235.41 | 1047 | 97.75 |
| Auth Code Grant Redirect With Consent | 300 | 1 | Get access token | 0 | 136.48 | 1062.94 | 312.13 | 1831 | 97.75 |
| Auth Code Grant Redirect With Consent | 300 | 1 | Send request to authorize end poiont | 0 | 136.65 | 307.54 | 203.02 | 955 | 97.75 |
| Auth Code Grant Redirect With Consent | 500 | 1 | Authorize Request | 0 | 136.33 | 718.36 | 330.32 | 1767 | 97.32 |
| Auth Code Grant Redirect With Consent | 500 | 1 | Common Auth Login HTTP Request | 0 | 136.45 | 675.91 | 347.12 | 1631 | 97.32 |
| Auth Code Grant Redirect With Consent | 500 | 1 | Get access token | 0 | 136.2 | 1683.04 | 561.99 | 3151 | 97.32 |
| Auth Code Grant Redirect With Consent | 500 | 1 | Send request to authorize end poiont | 0 | 136.34 | 588.08 | 379.86 | 1799 | 97.32 |
| SAML2 SSO Redirect Binding | 50 | 1 | Identity Provider Login | 0 | 184.59 | 211.62 | 66.45 | 395 | 97.13 |
| SAML2 SSO Redirect Binding | 50 | 1 | Initial SAML Request | 0 | 184.63 | 57.96 | 37.1 | 172 | 97.13 |
| SAML2 SSO Redirect Binding | 100 | 1 | Identity Provider Login | 0 | 187.28 | 406.03 | 131.98 | 775 | 96.98 |
| SAML2 SSO Redirect Binding | 100 | 1 | Initial SAML Request | 0 | 187.22 | 126.37 | 93.11 | 431 | 96.98 |
| SAML2 SSO Redirect Binding | 150 | 1 | Identity Provider Login | 0 | 189.68 | 598.57 | 209.21 | 1159 | 96.84 |
| SAML2 SSO Redirect Binding | 150 | 1 | Initial SAML Request | 0 | 189.77 | 190.66 | 157.91 | 727 | 96.84 |
| SAML2 SSO Redirect Binding | 300 | 1 | Identity Provider Login | 0 | 179.11 | 1300.06 | 462.5 | 2575 | 96.68 |
| SAML2 SSO Redirect Binding | 300 | 1 | Initial SAML Request | 0 | 179.05 | 372.66 | 354.47 | 1663 | 96.68 |
| SAML2 SSO Redirect Binding | 500 | 1 | Identity Provider Login | 0 | 185.28 | 2082.26 | 730.22 | 4015 | 96.18 |
| SAML2 SSO Redirect Binding | 500 | 1 | Initial SAML Request | 0 | 185.36 | 609.86 | 580.19 | 2703 | 96.18 |