Skip to content

Instantly share code, notes, and snippets.

@omindu
Created November 6, 2019 04:53
Show Gist options
  • Save omindu/404d56644f9cccc148f9cc20f2de303e to your computer and use it in GitHub Desktop.
Save omindu/404d56644f9cccc148f9cc20f2de303e to your computer and use it in GitHub Desktop.

IAM Performance Test Results

During each release, we execute various automated performance test scenarios and publish the results.

Test Scenarios Description
SAML2 SSO Redirect Binding Multi Tenant Obtain a SAML 2 assertion response using redirect binding.
Auth Code Grant Redirect Multi Tenant Obtain an access token using the OAuth 2.0 authorization code grant type.

Our test client is Apache JMeter. We test each scenario for a fixed duration of time and split the test results into warm-up and measurement parts and use the measurement part to compute the performance metrics. For this particular instance, the duration of each test is 15 minutes and the warm-up period is 5 minutes.

We run the performance tests under different numbers of concurrent users and heap sizes to gain a better understanding on how the server reacts to different loads.

The main performance metrics:

  1. Throughput: The number of requests that the WSO2 Identity Server processes during a specific time interval (e.g. per second).
  2. Response Time: The end-to-end latency for a given operation of the WSO2 Identity Server. The complete distribution of response times was recorded.

In addition to the above metrics, we measure the load average and several memory-related metrics.

The following are the test parameters.

Test Parameter Description Values
Scenario Name The name of the test scenario. Refer to the above table.
Heap Size The amount of memory allocated to the application 2G
Concurrent Users The number of users accessing the application at the same time. 50, 100, 150, 300, 500
Tenants The number of tenants active druing the test.
IS Instance Type The AWS instance type used to run the Identity Server. c5.xlarge

The following are the measurements collected from each performance test conducted for a given combination of test parameters.

Measurement Description
Error % Percentage of requests with errors
Average Response Time (ms) The average response time of a set of results
Standard Deviation of Response Time (ms) The Standard Deviation of the response time.
99th Percentile of Response Time (ms) 99% of the requests took no more than this time. The remaining samples took at least as long as this
Throughput (Requests/sec) The throughput measured in requests per second.
Average Memory Footprint After Full GC (M) The average memory consumed by the application after a full garbage collection event.

The following is the summary of performance test results collected for the measurement period.

Scenario Name Concurrent Users Tenants Label Error % Throughput (Requests/sec) Average Response Time (ms) Standard Deviation of Response Time (ms) 99th Percentile of Response Time (ms) WSO2 Identity Server GC Throughput (%)
SAML2 SSO Redirect Binding Multi Tenant 50 5 Identity Provider Login 0 118.38 370.43 74.54 563 97.86
SAML2 SSO Redirect Binding Multi Tenant 50 5 Initial SAML Request 0 118.4 50.87 36.25 173 97.86
SAML2 SSO Redirect Binding Multi Tenant 50 10 Identity Provider Login 0 114.13 387.79 74.56 583 97.9
SAML2 SSO Redirect Binding Multi Tenant 50 10 Initial SAML Request 0 114.16 49.27 35.78 168 97.9
SAML2 SSO Redirect Binding Multi Tenant 50 25 Identity Provider Login 0 112.83 392.73 77.97 595 97.88
SAML2 SSO Redirect Binding Multi Tenant 50 25 Initial SAML Request 0 112.87 49.35 37.03 176 97.88
SAML2 SSO Redirect Binding Multi Tenant 50 50 Identity Provider Login 0 117.52 376.13 77.17 579 97.83
SAML2 SSO Redirect Binding Multi Tenant 50 50 Initial SAML Request 0 117.53 48.29 36.08 169 97.83
SAML2 SSO Redirect Binding Multi Tenant 50 100 Identity Provider Login 0 115.55 382.57 77.61 583 97.73
SAML2 SSO Redirect Binding Multi Tenant 50 100 Initial SAML Request 0 115.58 48.98 35.41 170 97.73
SAML2 SSO Redirect Binding Multi Tenant 100 5 Identity Provider Login 0 119.18 727.55 160.46 1143 97.75
SAML2 SSO Redirect Binding Multi Tenant 100 5 Initial SAML Request 0 119.26 110.08 91.16 411 97.75
SAML2 SSO Redirect Binding Multi Tenant 100 10 Identity Provider Login 0 117.8 736.61 167.25 1183 97.8
SAML2 SSO Redirect Binding Multi Tenant 100 10 Initial SAML Request 0 117.83 110.75 94.8 429 97.8
SAML2 SSO Redirect Binding Multi Tenant 100 25 Identity Provider Login 0 116.81 742.07 168.36 1183 97.68
SAML2 SSO Redirect Binding Multi Tenant 100 25 Initial SAML Request 0 116.87 112.67 94.15 425 97.68
SAML2 SSO Redirect Binding Multi Tenant 100 50 Identity Provider Login 0 115.89 751.32 167.19 1183 97.68
SAML2 SSO Redirect Binding Multi Tenant 100 50 Initial SAML Request 0 115.96 110.01 92.54 413 97.68
SAML2 SSO Redirect Binding Multi Tenant 100 100 Identity Provider Login 0 112.54 776.31 174.19 1231 97.74
SAML2 SSO Redirect Binding Multi Tenant 100 100 Initial SAML Request 0 112.61 110.62 90.71 409 97.74
SAML2 SSO Redirect Binding Multi Tenant 150 5 Identity Provider Login 0 117.65 1088.36 265.76 1775 97.65
SAML2 SSO Redirect Binding Multi Tenant 150 5 Initial SAML Request 0 117.68 184.23 165.85 743 97.65
SAML2 SSO Redirect Binding Multi Tenant 150 10 Identity Provider Login 0 116.97 1096.27 260.76 1775 97.7
SAML2 SSO Redirect Binding Multi Tenant 150 10 Initial SAML Request 0 117.11 183.14 166.64 735 97.7
SAML2 SSO Redirect Binding Multi Tenant 150 25 Identity Provider Login 0 110.95 1169.38 265.12 1831 97.68
SAML2 SSO Redirect Binding Multi Tenant 150 25 Initial SAML Request 0 111.01 179.79 159.34 711 97.68
SAML2 SSO Redirect Binding Multi Tenant 150 50 Identity Provider Login 0 109.31 1187.83 270.95 1871 97.7
SAML2 SSO Redirect Binding Multi Tenant 150 50 Initial SAML Request 0 109.34 182.06 164.11 743 97.7
SAML2 SSO Redirect Binding Multi Tenant 150 100 Identity Provider Login 0 112.63 1142.44 276.27 1855 97.6
SAML2 SSO Redirect Binding Multi Tenant 150 100 Initial SAML Request 0 112.63 186.8 167.19 719 97.6
SAML2 SSO Redirect Binding Multi Tenant 300 5 Identity Provider Login 0 115.25 2209.02 626.17 3839 97.45
SAML2 SSO Redirect Binding Multi Tenant 300 5 Initial SAML Request 0 115.29 388.19 380.31 1703 97.45
SAML2 SSO Redirect Binding Multi Tenant 300 10 Identity Provider Login 0 114.42 2212.96 614.85 3791 97.43
SAML2 SSO Redirect Binding Multi Tenant 300 10 Initial SAML Request 0 114.51 403.47 394.44 1759 97.43
SAML2 SSO Redirect Binding Multi Tenant 300 25 Identity Provider Login 0 115.17 2213.59 617.33 3855 97.41
SAML2 SSO Redirect Binding Multi Tenant 300 25 Initial SAML Request 0 115.25 383.69 372.07 1631 97.41
SAML2 SSO Redirect Binding Multi Tenant 300 50 Identity Provider Login 0 112.87 2286.91 614.21 3903 97.44
SAML2 SSO Redirect Binding Multi Tenant 300 50 Initial SAML Request 0 112.92 362.56 353.44 1599 97.44
SAML2 SSO Redirect Binding Multi Tenant 300 100 Identity Provider Login 0.11 110.74 2326.46 623.53 3935 97.36
SAML2 SSO Redirect Binding Multi Tenant 300 100 Initial SAML Request 0 110.71 375.76 357.56 1567 97.36
SAML2 SSO Redirect Binding Multi Tenant 500 5 Identity Provider Login 0 109.59 3975.11 1083.64 6847 97.16
SAML2 SSO Redirect Binding Multi Tenant 500 5 Initial SAML Request 0 109.62 568.84 575.17 2607 97.16
SAML2 SSO Redirect Binding Multi Tenant 500 10 Identity Provider Login 0 110.34 3970.98 1087.59 6879 97.14
SAML2 SSO Redirect Binding Multi Tenant 500 10 Initial SAML Request 0 110.3 545.06 555.33 2575 97.14
SAML2 SSO Redirect Binding Multi Tenant 500 25 Identity Provider Login 0 107.61 4074.46 1072.19 7007 97.19
SAML2 SSO Redirect Binding Multi Tenant 500 25 Initial SAML Request 0 107.49 553.87 550.09 2607 97.19
SAML2 SSO Redirect Binding Multi Tenant 500 50 Identity Provider Login 0 111.88 3832.89 1049.7 6623 97.03
SAML2 SSO Redirect Binding Multi Tenant 500 50 Initial SAML Request 0 112.03 613.26 616.93 2895 97.03
SAML2 SSO Redirect Binding Multi Tenant 500 100 Identity Provider Login 0 110.26 3927.27 1087.99 6847 97
SAML2 SSO Redirect Binding Multi Tenant 500 100 Initial SAML Request 0 110.25 592.85 587.95 2831 97
Auth Code Grant Redirect Multi Tenant 50 5 Authorize Request 0 32.49 93.71 26.76 164 99.18
Auth Code Grant Redirect Multi Tenant 50 5 Common Auth Login HTTP Request 0 32.48 58.71 18.64 108 99.18
Auth Code Grant Redirect Multi Tenant 50 5 Get access token 0 32.45 1324.13 365.98 2271 99.18
Auth Code Grant Redirect Multi Tenant 50 5 Send request to authorize end poiont 0 32.48 62.2 20.73 117 99.18
Auth Code Grant Redirect Multi Tenant 50 10 Authorize Request 0 32.68 92.27 27.33 163 99.15
Auth Code Grant Redirect Multi Tenant 50 10 Common Auth Login HTTP Request 0 32.67 58.89 19.03 108 99.15
Auth Code Grant Redirect Multi Tenant 50 10 Get access token 0 32.63 1318.01 365.94 2255 99.15
Auth Code Grant Redirect Multi Tenant 50 10 Send request to authorize end poiont 0 32.66 61.19 20.6 115 99.15
Auth Code Grant Redirect Multi Tenant 50 25 Authorize Request 0 32.79 91.26 26.48 159 99.09
Auth Code Grant Redirect Multi Tenant 50 25 Common Auth Login HTTP Request 0 32.79 57.98 20.24 107 99.09
Auth Code Grant Redirect Multi Tenant 50 25 Get access token 0 32.74 1314.61 364.5 2255 99.09
Auth Code Grant Redirect Multi Tenant 50 25 Send request to authorize end poiont 0 32.79 60.74 20.7 114 99.09
Auth Code Grant Redirect Multi Tenant 50 50 Authorize Request 0 32.96 92.52 26.76 162 99.12
Auth Code Grant Redirect Multi Tenant 50 50 Common Auth Login HTTP Request 0 32.97 58.52 18.45 106 99.12
Auth Code Grant Redirect Multi Tenant 50 50 Get access token 0 32.92 1303.97 364.42 2255 99.12
Auth Code Grant Redirect Multi Tenant 50 50 Send request to authorize end poiont 0 32.96 60.94 19.77 111 99.12
Auth Code Grant Redirect Multi Tenant 50 100 Authorize Request 0 32.48 93.73 27.14 166 99.09
Auth Code Grant Redirect Multi Tenant 50 100 Common Auth Login HTTP Request 0 32.47 58.91 18.28 107 99.09
Auth Code Grant Redirect Multi Tenant 50 100 Get access token 0 32.43 1325.01 361.3 2271 99.09
Auth Code Grant Redirect Multi Tenant 50 100 Send request to authorize end poiont 0 32.46 61.41 20.2 113 99.09
Auth Code Grant Redirect Multi Tenant 100 5 Authorize Request 0 32.5 97.51 31.69 191 99.15
Auth Code Grant Redirect Multi Tenant 100 5 Common Auth Login HTTP Request 0 32.51 63.27 22.85 130 99.15
Auth Code Grant Redirect Multi Tenant 100 5 Get access token 0 32.39 2850.21 904.09 5375 99.15
Auth Code Grant Redirect Multi Tenant 100 5 Send request to authorize end poiont 0 32.51 66.05 23.65 130 99.15
Auth Code Grant Redirect Multi Tenant 100 10 Authorize Request 0 32.61 96.96 30.37 173 99.13
Auth Code Grant Redirect Multi Tenant 100 10 Common Auth Login HTTP Request 0 32.64 62.91 21.57 121 99.13
Auth Code Grant Redirect Multi Tenant 100 10 Get access token 0 32.53 2838.78 909.62 5503 99.13
Auth Code Grant Redirect Multi Tenant 100 10 Send request to authorize end poiont 0 32.64 65.78 23.3 129 99.13
Auth Code Grant Redirect Multi Tenant 100 25 Authorize Request 0 32.75 96.53 31.04 179 99.11
Auth Code Grant Redirect Multi Tenant 100 25 Common Auth Login HTTP Request 0 32.76 62.64 21.72 121 99.11
Auth Code Grant Redirect Multi Tenant 100 25 Get access token 0 32.65 2829.18 911.3 5471 99.11
Auth Code Grant Redirect Multi Tenant 100 25 Send request to authorize end poiont 0 32.76 64.89 23.09 127 99.11
Auth Code Grant Redirect Multi Tenant 100 50 Authorize Request 0 32.93 97.65 30.13 177 99.12
Auth Code Grant Redirect Multi Tenant 100 50 Common Auth Login HTTP Request 0 32.94 62.98 21.57 123 99.12
Auth Code Grant Redirect Multi Tenant 100 50 Get access token 0 32.83 2811.87 897.7 5343 99.12
Auth Code Grant Redirect Multi Tenant 100 50 Send request to authorize end poiont 0 32.91 65.39 22.81 128 99.12
Auth Code Grant Redirect Multi Tenant 100 100 Authorize Request 0 32.76 98.49 31.21 181 99.08
Auth Code Grant Redirect Multi Tenant 100 100 Common Auth Login HTTP Request 0 32.77 63.92 21.59 121 99.08
Auth Code Grant Redirect Multi Tenant 100 100 Get access token 0 32.68 2822.9 926.19 5503 99.08
Auth Code Grant Redirect Multi Tenant 100 100 Send request to authorize end poiont 0 32.77 66.53 24.05 132 99.08
Auth Code Grant Redirect Multi Tenant 150 5 Authorize Request 0 32.53 104.85 38.33 222 99.07
Auth Code Grant Redirect Multi Tenant 150 5 Common Auth Login HTTP Request 0 32.49 68.63 27.14 155 99.07
Auth Code Grant Redirect Multi Tenant 150 5 Get access token 0 32.36 4373.54 1550.15 8767 99.07
Auth Code Grant Redirect Multi Tenant 150 5 Send request to authorize end poiont 0 32.49 71.16 27.93 154 99.07
Auth Code Grant Redirect Multi Tenant 150 10 Authorize Request 0 32.64 103.1 36.78 211 99.08
Auth Code Grant Redirect Multi Tenant 150 10 Common Auth Login HTTP Request 0 32.64 68.36 26.69 151 99.08
Auth Code Grant Redirect Multi Tenant 150 10 Get access token 0 32.47 4353.34 1599.4 9087 99.08
Auth Code Grant Redirect Multi Tenant 150 10 Send request to authorize end poiont 0 32.64 70.86 27.97 151 99.08
Auth Code Grant Redirect Multi Tenant 150 25 Authorize Request 0 32.9 102.4 37.66 210 99.07
Auth Code Grant Redirect Multi Tenant 150 25 Common Auth Login HTTP Request 0 32.9 68.86 27.09 159 99.07
Auth Code Grant Redirect Multi Tenant 150 25 Get access token 0 32.7 4322.6 1591.65 9087 99.07
Auth Code Grant Redirect Multi Tenant 150 25 Send request to authorize end poiont 0 32.9 70.61 27.64 151 99.07
Auth Code Grant Redirect Multi Tenant 150 50 Authorize Request 0 32.91 102.14 35.58 218 99.07
Auth Code Grant Redirect Multi Tenant 150 50 Common Auth Login HTTP Request 0 32.91 69.11 33.61 168 99.07
Auth Code Grant Redirect Multi Tenant 150 50 Get access token 0 32.74 4319.46 1571.83 8959 99.07
Auth Code Grant Redirect Multi Tenant 150 50 Send request to authorize end poiont 0 32.91 69.66 27.62 154 99.07
Auth Code Grant Redirect Multi Tenant 150 100 Authorize Request 0 32.77 101.81 34.32 196 99.06
Auth Code Grant Redirect Multi Tenant 150 100 Common Auth Login HTTP Request 0 32.79 69.37 26.52 151 99.06
Auth Code Grant Redirect Multi Tenant 150 100 Get access token 0 32.63 4330.76 1546.27 8959 99.06
Auth Code Grant Redirect Multi Tenant 150 100 Send request to authorize end poiont 0 32.79 71.05 27.32 151 99.06
Auth Code Grant Redirect Multi Tenant 300 5 Authorize Request 0 33.19 134.73 61.06 351 98.88
Auth Code Grant Redirect Multi Tenant 300 5 Common Auth Login HTTP Request 0 33.19 101.13 59.85 321 98.88
Auth Code Grant Redirect Multi Tenant 300 5 Get access token 0 32.82 8722.32 3472.99 18815 98.88
Auth Code Grant Redirect Multi Tenant 300 5 Send request to authorize end poiont 0 33.19 98.11 50.6 275 98.88
Auth Code Grant Redirect Multi Tenant 300 10 Authorize Request 0 33.05 131.77 65.02 385 98.92
Auth Code Grant Redirect Multi Tenant 300 10 Common Auth Login HTTP Request 0 33.05 97.23 57.24 309 98.92
Auth Code Grant Redirect Multi Tenant 300 10 Get access token 0 32.74 8770.45 3616.42 19455 98.92
Auth Code Grant Redirect Multi Tenant 300 10 Send request to authorize end poiont 0 33.05 95.27 51.71 287 98.92
Auth Code Grant Redirect Multi Tenant 300 25 Authorize Request 0 32.94 129.35 58.49 341 98.95
Auth Code Grant Redirect Multi Tenant 300 25 Common Auth Login HTTP Request 0 32.95 94.12 51.17 287 98.95
Auth Code Grant Redirect Multi Tenant 300 25 Get access token 0 32.59 8795.89 3439.26 18687 98.95
Auth Code Grant Redirect Multi Tenant 300 25 Send request to authorize end poiont 0 32.95 92.34 45.37 252 98.95
Auth Code Grant Redirect Multi Tenant 300 50 Authorize Request 0 33.07 131.38 62.9 349 98.98
Auth Code Grant Redirect Multi Tenant 300 50 Common Auth Login HTTP Request 0 33.06 97.98 56.12 299 98.98
Auth Code Grant Redirect Multi Tenant 300 50 Get access token 0 32.76 8741.11 3501.71 18687 98.98
Auth Code Grant Redirect Multi Tenant 300 50 Send request to authorize end poiont 0 33.06 94 47.97 265 98.98
Auth Code Grant Redirect Multi Tenant 300 100 Authorize Request 0 33.1 132.87 70.65 375 98.9
Auth Code Grant Redirect Multi Tenant 300 100 Common Auth Login HTTP Request 0 33.12 99.28 54.89 289 98.9
Auth Code Grant Redirect Multi Tenant 300 100 Get access token 0 32.78 8735.29 3440.36 18431 98.9
Auth Code Grant Redirect Multi Tenant 300 100 Send request to authorize end poiont 0 33.12 96.78 51.5 279 98.9
Auth Code Grant Redirect Multi Tenant 500 5 Authorize Request 0 32.87 206.6 119.2 599 98.8
Auth Code Grant Redirect Multi Tenant 500 5 Common Auth Login HTTP Request 0 32.69 156.22 103.33 483 98.8
Auth Code Grant Redirect Multi Tenant 500 5 Get access token 0 32.37 14790.2 5960.82 31359 98.8
Auth Code Grant Redirect Multi Tenant 500 5 Send request to authorize end poiont 0 32.8 147.27 87.38 447 98.8
Auth Code Grant Redirect Multi Tenant 500 10 Authorize Request 0 33.06 229.21 139.99 679 98.81
Auth Code Grant Redirect Multi Tenant 500 10 Common Auth Login HTTP Request 0 33.07 195.21 157.08 747 98.81
Auth Code Grant Redirect Multi Tenant 500 10 Get access token 0 32.46 14622.59 5853.05 31103 98.81
Auth Code Grant Redirect Multi Tenant 500 10 Send request to authorize end poiont 0 33.05 165.41 106.98 523 98.81
Auth Code Grant Redirect Multi Tenant 500 25 Authorize Request 0 33.04 213.39 122.4 659 98.84
Auth Code Grant Redirect Multi Tenant 500 25 Common Auth Login HTTP Request 0 33.04 168.71 121.93 607 98.84
Auth Code Grant Redirect Multi Tenant 500 25 Get access token 0 32.44 14622.2 5927.95 31103 98.84
Auth Code Grant Redirect Multi Tenant 500 25 Send request to authorize end poiont 0 33.16 154.08 97.91 497 98.84
Auth Code Grant Redirect Multi Tenant 500 50 Authorize Request 0 33.42 225.27 146.55 631 98.82
Auth Code Grant Redirect Multi Tenant 500 50 Common Auth Login HTTP Request 0 33.34 199.93 152.91 707 98.82
Auth Code Grant Redirect Multi Tenant 500 50 Get access token 0 32.96 14432.49 5817.8 31103 98.82
Auth Code Grant Redirect Multi Tenant 500 50 Send request to authorize end poiont 0 33.19 170.41 111.31 535 98.82
Auth Code Grant Redirect Multi Tenant 500 100 Authorize Request 0 33.14 202.03 101.53 523 98.77
Auth Code Grant Redirect Multi Tenant 500 100 Common Auth Login HTTP Request 0 33.15 162.59 106.19 509 98.77
Auth Code Grant Redirect Multi Tenant 500 100 Get access token 0 32.55 14575.29 5904.53 31359 98.77
Auth Code Grant Redirect Multi Tenant 500 100 Send request to authorize end poiont 0 33.15 149.81 94.46 451 98.77
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment