PRB-0010422.md

> jq '.Records[].sourceIPAddress' event_history_2020-05-18_19.json > source
rremer@rremer-ltm7:~/Documents/PRB-0010422/
> jq '.Records[].errorCode' event_history_2020-05-18_19.json > code
rremer@rremer-ltm7:~/Documents/PRB-0010422/
> jq '.Records[].eventTime' event_history_2020-05-18_19.json > time
rremer@rremer-ltm7:~/Documents/PRB-0010422/
> paste time source code > event_history_2020-05-18_19.tsv

describeInstance calls

5/18-5/19

No incident

32830/day

requests > autoscaling.amazonaws.com

1606 "34.213.99.195"
1752 "54.148.134.157"
1818 "54.244.131.238"
27102 "96.43.156.9"

503s > 1

5 "96.43.156.9"

5/25-5/26

Peak incident

44393/day

requests > autoscaling.amazonaws.com

1185 "54.148.134.157"
1189 "54.244.131.238"
1262 "34.213.99.195"
40372 "96.43.156.9"

### 503s > 1
   8 "34.199.87.33"
  13 "34.197.4.161"
  29 "34.199.94.171"
 768 "54.148.134.157"
 807 "54.244.131.238"
 840 "34.213.99.195"
20146 "96.43.156.9"

5/27-5/28

Incident ongoing

43300/day

requests > autoscaling.amazonaws.com

1084 "52.38.201.66"
1130 "54.213.200.20"
1186 "18.236.6.66"
1444 "54.148.134.157"
1510 "54.244.131.238"
1613 "34.213.99.195"
34214 "96.43.156.9"

503s > 1

   2 "104.155.129.50"
   2 "104.197.21.145"
   2 "104.198.160.64"
   2 "130.211.230.41"
   2 "34.66.15.229"
   2 "34.66.19.59"
   2 "34.67.110.86"
   2 "34.69.225.215"
   2 "34.70.187.191"
   2 "34.70.99.38"
   2 "34.71.213.89"
   2 "34.72.135.92"
   2 "35.192.104.128"
   2 "35.192.33.186"
   2 "35.193.213.181"
   2 "35.193.86.110"
   2 "35.194.48.249"
   2 "35.194.51.226"
   2 "35.194.55.159"
   2 "35.202.186.233"
   2 "35.223.146.91"
   2 "35.223.192.155"
   2 "35.224.141.129"
   2 "trustedadvisor.amazonaws.com"
   3 "104.155.170.81"
   3 "34.66.195.37"
   3 "34.67.108.22"
   3 "35.223.198.52"
   4 "34.71.18.107"
   5 "204.14.239.82"
  46 "34.199.94.171"
  50 "34.199.87.33"
  95 "34.197.4.161"
 609 "54.213.200.20"
 660 "52.38.201.66"
 692 "18.236.6.66"
 872 "54.148.134.157"
 954 "54.244.131.238"
 977 "34.213.99.195"
22055 "96.43.156.9"

6/01-6/02

No incident (patches applied)

29643/day

requests > autoscaling.amazonaws.com

 235 "54.213.200.20"
 247 "18.236.6.66"
 254 "52.38.201.66"
 339 "10.215.130.109"
 472 "10.215.128.6"
 552 "10.215.129.9"
 927 "10.215.128.164"
 954 "10.215.128.145"
 988 "10.215.129.113"
1033 "10.215.129.200"
1057 "10.215.130.37"
1063 "10.215.129.156"
1087 "10.215.130.27"
1094 "10.215.129.72"
1102 "10.215.128.63"
1106 "10.215.129.239"
1109 "10.215.128.41"
1126 "10.215.130.243"
1130 "10.215.128.197"
1192 "10.215.130.183"
1224 "10.215.130.199"
1506 "10.215.128.25"
1586 "10.215.130.144"
1663 "10.215.130.103"
1687 "10.215.129.150"
1893 "10.215.128.166"
2272 "10.215.129.36"

503s > 1

2 "10.215.130.144"
4 "13.110.54.42"

And in case it wasn't clear, the difference between 5/28 and 6/1 in source IPs is that we had 22 pods going through the crd public proxy (96.43.156.9), and now they go to the vpc endpoint for ec2 api, showing up with their private ips.