Database looks fine. Ends with this:
{"log":"LOG: could not send data to client: Broken pipe\n","stream":"stderr","time":"2020-04-01T21:09:16.86734608Z"}
{"log":"FATAL: connection to client lost\n","stream":"stderr","time":"2020-04-01T21:09:16.867349505Z"}
{"log":"FATAL: canceling authentication due to timeout\n","stream":"stderr","time":"2020-04-01T21:09:16.867352182Z"}
{"log":"FATAL: canceling authentication due to timeout\n","stream":"stderr","time":"2020-04-01T21:09:16.867354895Z"}
{"log":"FATAL: canceling authentication due to timeout\n","stream":"stderr","time":"2020-04-01T21:09:16.867357632Z"}
Which is probably just the restart.
Sidekiq looks normal.
Streaming server looks fine. Started hanging around `2020-04-01T21:11:47.425933469Z
ElasticSearch is running low on disk space lol.
Rails server seems fine. Started hanging around 2020-04-01T21:12:37.386801784Z UTC.
Redis server is normal.
nginx shows no entries in /var/log/nginx/error.log. Looking in nginx_error.log I found thousands of these entries:
2020/04/01 15:00:28 [warn] 30738#30738: *9190046 an upstream response is buffered to a temporary file /var/lib/nginx/proxy/0/14/0000600140 while reading upstream, client: 35.226.223.64, server: mastodon.technology, request: "GET /@fakefred HTTP/1.0", upstream: "http://127.0.0.1:3000/@fakefred", host: "mastodon.technology"
2020/04/01 15:00:32 [warn] 30738#30738: *9190125 an upstream response is buffered to a temporary file /var/lib/nginx/proxy/6/14/0000600146 while reading upstream, client: 35.226.223.64, server: mastodon.technology, request: "GET /@bugaevc HTTP/1.0", upstream: "http://127.0.0.1:3000/@bugaevc", host: "mastodon.technology"
2020/04/01 15:00:36 [warn] 30738#30738: *9190213 an upstream response is buffered to a temporary file /var/lib/nginx/proxy/4/15/0000600154 while reading upstream, client: 35.226.223.64, server: mastodon.technology, request: "GET /@IzzyOnDroid HTTP/1.0", upstream: "http://127.0.0.1:3000/@IzzyOnDroid", host: "mastodon.technology"
2020/04/01 15:01:27 [warn] 30740#30740: *9191070 a client request body is buffered to a temporary file /var/lib/nginx/body/0000600226, client: 164.132.90.112, server: mastodon.technology, request: "POST /inbox HTTP/1.1", host: "mastodon.technology"
2020/04/01 15:01:33 [warn] 30739#30739: *9191154 a client request body is buffered to a temporary file /var/lib/nginx/body/0000600229, client: 178.63.240.148, server: mastodon.technology, request: "POST /inbox HTTP/1.1", host: "mastodon.technology"
2020/04/01 15:01:34 [warn] 30739#30739: *9191172 a client request body is buffered to a temporary file /var/lib/nginx/body/0000600230, client: 99.149.215.66, server: mastodon.technology, request: "POST /inbox HTTP/1.1", host: "mastodon.technology"
They start out infrequent but then pile up. Must be an nginx config issue.