Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

Database looks fine. Ends with this:

{"log":"LOG:  could not send data to client: Broken pipe\n","stream":"stderr","time":"2020-04-01T21:09:16.86734608Z"}
{"log":"FATAL:  connection to client lost\n","stream":"stderr","time":"2020-04-01T21:09:16.867349505Z"}
{"log":"FATAL:  canceling authentication due to timeout\n","stream":"stderr","time":"2020-04-01T21:09:16.867352182Z"}
{"log":"FATAL:  canceling authentication due to timeout\n","stream":"stderr","time":"2020-04-01T21:09:16.867354895Z"}
{"log":"FATAL:  canceling authentication due to timeout\n","stream":"stderr","time":"2020-04-01T21:09:16.867357632Z"}

Which is probably just the restart.


Sidekiq looks normal.


Streaming server looks fine. Started hanging around `2020-04-01T21:11:47.425933469Z


ElasticSearch is running low on disk space lol.


Rails server seems fine. Started hanging around 2020-04-01T21:12:37.386801784Z UTC.


Redis server is normal.


nginx shows no entries in /var/log/nginx/error.log. Looking in nginx_error.log I found thousands of these entries:

2020/04/01 15:00:28 [warn] 30738#30738: *9190046 an upstream response is buffered to a temporary file /var/lib/nginx/proxy/0/14/0000600140 while reading upstream, client: 35.226.223.64, server: mastodon.technology, request: "GET /@fakefred HTTP/1.0", upstream: "http://127.0.0.1:3000/@fakefred", host: "mastodon.technology"
2020/04/01 15:00:32 [warn] 30738#30738: *9190125 an upstream response is buffered to a temporary file /var/lib/nginx/proxy/6/14/0000600146 while reading upstream, client: 35.226.223.64, server: mastodon.technology, request: "GET /@bugaevc HTTP/1.0", upstream: "http://127.0.0.1:3000/@bugaevc", host: "mastodon.technology"
2020/04/01 15:00:36 [warn] 30738#30738: *9190213 an upstream response is buffered to a temporary file /var/lib/nginx/proxy/4/15/0000600154 while reading upstream, client: 35.226.223.64, server: mastodon.technology, request: "GET /@IzzyOnDroid HTTP/1.0", upstream: "http://127.0.0.1:3000/@IzzyOnDroid", host: "mastodon.technology"
2020/04/01 15:01:27 [warn] 30740#30740: *9191070 a client request body is buffered to a temporary file /var/lib/nginx/body/0000600226, client: 164.132.90.112, server: mastodon.technology, request: "POST /inbox HTTP/1.1", host: "mastodon.technology"
2020/04/01 15:01:33 [warn] 30739#30739: *9191154 a client request body is buffered to a temporary file /var/lib/nginx/body/0000600229, client: 178.63.240.148, server: mastodon.technology, request: "POST /inbox HTTP/1.1", host: "mastodon.technology"
2020/04/01 15:01:34 [warn] 30739#30739: *9191172 a client request body is buffered to a temporary file /var/lib/nginx/body/0000600230, client: 99.149.215.66, server: mastodon.technology, request: "POST /inbox HTTP/1.1", host: "mastodon.technology"

They start out infrequent but then pile up. Must be an nginx config issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.