This test goal was to determine ELB is to scale out / scale-up at what time when a load is applied to the ELB. But results of the tests it was completely different from the behavior of ELB that I was assumed in advance. As there is a possibility that there was a problem in the test method, the test I do again. This test result is I keep a record for interesting.
Jan 12, 2015
- ap-northeast-1
- VPC
- EC2(c3.8xlarge)
- Internet-facing ELB(attach to 2 Availability-Zones)
[ec2-user@ip-10-14-1-132 ~]$ ping 10.14.1.47
PING 10.14.1.47 (10.14.1.47) 56(84) bytes of data.
64 bytes from 10.14.1.47: icmp_seq=1 ttl=64 time=0.336 ms
64 bytes from 10.14.1.47: icmp_seq=2 ttl=64 time=0.313 ms
64 bytes from 10.14.1.47: icmp_seq=3 ttl=64 time=0.240 ms
64 bytes from 10.14.1.47: icmp_seq=4 ttl=64 time=0.262 ms
64 bytes from 10.14.1.47: icmp_seq=5 ttl=64 time=0.323 ms
[ec2-user@ip-10-14-1-132 ~]$ ping 10.14.0.46
PING 10.14.0.46 (10.14.0.46) 56(84) bytes of data.
64 bytes from 10.14.0.46: icmp_seq=1 ttl=64 time=2.09 ms
64 bytes from 10.14.0.46: icmp_seq=2 ttl=64 time=2.14 ms
64 bytes from 10.14.0.46: icmp_seq=3 ttl=64 time=2.13 ms
64 bytes from 10.14.0.46: icmp_seq=4 ttl=64 time=2.08 ms
64 bytes from 10.14.0.46: icmp_seq=5 ttl=64 time=2.16 ms
I was determine same Availability-Zone ELB's end-point ip address by "dig" and healthcheck access. I thought that ELB may scale-out/scale-up by total load condition.(It is my mistake...)
$ cat wrk2elb.sh
#!/bin/bash
taskset 1 /path/to/wrk -t1 -c100 -d60 http://single-ip-addr-of-elb/1
- 54.199.198.200 -> 10.14.1.47 (ap-northeast-1c)
- 54.238.165.224 -> 10.14.0.46 (ap-northeast-1a)
$ dig ELBDNSNAME +short
54.199.198.200
54.238.165.224
[ec2-user@ip-10-14-1-159 ~]$ ./wrk2elb.sh
Running 3m test @ http://54.199.198.200/1
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 18.43ms 48.00ms 542.06ms 99.15%
Req/Sec 7.38k 1.33k 10.17k 78.68%
1308827 requests in 3.00m, 280.84MB read
Requests/sec: 7271.25
Transfer/sec: 1.56MB
[ec2-user@ip-10-14-1-159 ~]$ ./wrk2elb.sh
Running 3m test @ http://54.199.198.200/1
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 15.64ms 23.34ms 317.92ms 99.21%
Req/Sec 7.49k 1.31k 11.15k 76.40%
1327794 requests in 3.00m, 284.91MB read
Requests/sec: 7376.59
Transfer/sec: 1.58MB
- Internal ELB instance 2 -> 4
- Internet DNS record 2 -> 2
- 10.14.1.47
- 10.14.0.46
- 10.14.1.221
- 10.14.0.145
$ dig ELBDNSNAME +short
54.64.133.218
54.64.172.150
- 54.64.172.150 -> 10.14.1.221 (ap-northeast-1c)
- 54.64.133.218 -> 10.14.0.145 (ap-northeast-1a)
[ec2-user@ip-10-14-1-159 ~]$ ./wrk2elb.sh
Running 3m test @ http://54.64.172.150/1
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 20.98ms 13.32ms 227.71ms 92.25%
Req/Sec 5.25k 1.27k 9.25k 74.12%
935016 requests in 3.00m, 200.63MB read
Requests/sec: 5194.38
Transfer/sec: 1.11MB
[ec2-user@ip-10-14-1-159 ~]$ ./wrk2elb.sh
Running 3m test @ http://54.64.172.150/1
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 19.80ms 9.59ms 116.32ms 91.50%
Req/Sec 5.46k 1.23k 8.71k 76.67%
971446 requests in 3.00m, 208.45MB read
Requests/sec: 5396.92
Transfer/sec: 1.16MB
- Internal ELB instance 4 -> 6
- Internet DNS record 2 -> 2
- 10.14.1.47
- 10.14.0.46
- 10.14.1.221
- 10.14.0.145
- 10.14.1.238
- 10.14.0.57
$ dig ELBDNSNAME +short
54.238.128.213
54.238.153.39
- 54.238.153.39 -> 10.14.1.238 (ap-northeast-1c)
- 54.238.128.213 -> 10.14.0.57 (ap-northeast-1a)
[ec2-user@ip-10-14-1-159 ~]$ ./wrk2elb.sh
Running 3m test @ http://54.238.153.39/1
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 16.11ms 3.96ms 71.16ms 79.88%
Req/Sec 6.42k 0.94k 8.68k 71.66%
1140660 requests in 3.00m, 244.76MB read
Requests/sec: 6337.00
Transfer/sec: 1.36MB
[ec2-user@ip-10-14-1-159 ~]$ ./wrk2elb.sh
Running 3m test @ http://54.238.153.39/1
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 16.11ms 4.00ms 63.85ms 80.38%
Req/Sec 6.43k 0.95k 9.30k 73.10%
1142916 requests in 3.00m, 245.24MB read
Requests/sec: 6349.53
Transfer/sec: 1.36MB
- Internal ELB instance 6 -> 8
- Internet DNS record 2 -> 2
- 10.14.1.47
- 10.14.0.46
- 10.14.1.221
- 10.14.0.145
- 10.14.1.238
- 10.14.0.57
- 10.14.1.176
- 10.14.0.168
$ dig ELBDNSNAME +short
54.65.186.86
54.65.5.225
- 54.65.5.225 -> 10.14.1.176 (ap-northeast-1c)
- 54.65.186.86 -> 10.14.0.168 (ap-northeast-1a)
ELB scale-out / scale-up is based on the HTTP request from the client. However, the results of this test, it indicates that the Internet-facing capacity of ELB is not increased with the client access. Each ELB instance were removed from ELB's DNS set, and new 2 instance's A-record were added to ELB's DNS set. that seems like ELB cluster was rotated.
This behavior is contrary to my intuition. I expected ELB scale-out increased A-record set, or scale-up is rotated A-record. But all ELB instance were same capacity, and each old ELB instance was removed from ELB. I think that caused by load traffic inequality across Availability-Zone.
Note:
DNS name of ELB was not contained in request host header of this test. I will trying that in next time.
Todo:
- Simultaneous and equal to load test for all of Availability-Zone
- Load access to be contained a host header that contains dns name of ELB.
- Trying load test based on network bandwidth.
2matz