blah.markdown

Good news

1. Our stopgap hack of "PR hook -> delay -> start build if push hook not received" seems to be working. It was needed >1.8k times since deployed.

   http://localhost:9000/search?interval=day&width=1241&relative=86400&page=7&sortOrder=desc&q=%22missing%20corresponding%20push%20hook%22&rangetype=relative&fields=message%2Csource&sortField=timestamp

Bad news

1. In that period only ~1k builds didn't require the hack.

   http://localhost:9000/search?rangetype=relative&fields=message%2Csource&width=1241&highlightMessage=&relative=86400&q=%22already+triggered+by+corresponding+push+hook%22

2. This hack relies on the existence of a PR branch. We don't know how many builds are missing that don't belong to a PR.

3. We've somehow lost nginx logging (therefore 413 visibility) in Graylog since "2016-10-25 21:00:09.000"

   https://circleci.slack.com/archives/ops-discuss/p1477621548004238

   @bear is working to get that back

Interesting

1. Another data point that points to a change in our system that resulted in us rejecting large payloads is amount of force pushes we receive (thanks @frank). The assumption is force pushes are larger payloads with more commits.

   http://localhost:9000/search?interval=day&width=1241&from=2016-09-18T18%3A44%3A37.000Z&page=1&sortOrder=desc&q=%22%5C%22x-github-event%5C%22%20%5C%22push%5C%22%22%20AND%20%22%5C%22forced%5C%22%3Atrue%22&rangetype=absolute&fields=message%2Csource&to=2016-10-28T04%3A00%3A00.000Z&sortField=timestamp

   We started receiving 80% less force pushes on Sep 22.