Skip to content

Instantly share code, notes, and snippets.

@jhaubrich

jhaubrich/errors.md

Last active October 16, 2020 23:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jhaubrich/80da5778d7abdae0fc98b1030544d7ca to your computer and use it in GitHub Desktop.
Save jhaubrich/80da5778d7abdae0fc98b1030544d7ca to your computer and use it in GitHub Desktop.
Download ZIP
Raw
errors.md

after staging update

2020-10-17 00:05:56,870 [ERROR]: 500 for http://staging.legend-es.com/api/v0/form/save-sensor-data with RESPONSE TEXT 'class java.lang.NullPointerException is not understandable' HEADERS {'Server': 'nginx/1.10.3 (Ubuntu)', 'Date': 'Fri, 16 Oct 2020 23:05:54 GMT', 'Content-Length': '58', 'Connection': 'keep-alive'} REQUEST HEADERS {'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'Basic bGVnZW5kOnRoaW5rbGVnZW5k', 'Content-Length': '3926', 'Content-Type': 'multipart/form-data; boundary=5a6ccd89e2efeaf7643a49a1c382e367'} in /home/pi/legend-pi/hub.py:203

inexplicable

2020-10-16 22:43:41,236 [ERROR]: 500 for http://live.legend-es.com/api/v0/form/save-sensor-data with RESPONSE TEXT 'class java.lang.IllegalArgumentException is not understandable' HEADERS {'Server': 'nginx/1.10.3 (Ubuntu)', 'Date': 'Fri, 16 Oct 2020 21:43:40 GMT', 'Content-Length': '62', 'Connection': 'keep-alive'} REQUEST HEADERS {'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'Basic bGVnZW5kOnRoaW5rbGVnZW5k', 'Content-Length': '3926', 'Content-Type': 'multipart/form-data; boundary=59642bca18d978ca563e420618098091'} in /home/pi/legend-pi/hub.py:203

hub.py stalled out for 1hr with good network

After full queue followed by full recovery, I disconnected the antennae again. For about 30min no errors wer being produced and the queue continued to grow. I disconnected the antenna and watched it. After about an hour it started erroring out at a rate of 1000's of messages per sec. I reconnnected antenna and it continues to error out with an upload speed of 10Mbps.

Hub was restarted by supervisor(?) 30min after it started freaking out.

Hub recovered and resumed backfilling from queue!

I'm going to add a sleep(.5) if the response isn't 200

pi@LAB:/home/pi $ date #removed antenna
Tue 06 Oct 2020 06:49:24 PM BST
pi@LAB:/home/pi $ date # replaced antenna
Tue 06 Oct 2020 07:47:17 PM BST
pi@LAB:/home/pi $ date
Tue 06 Oct 2020 08:00:47 PM BST
pi@LAB:/home/pi $ date # full
Tue 06 Oct 2020 08:05:59 PM BST
pi@LAB:/home/pi $ date # last connection error at 19:44
Tue 06 Oct 2020 08:19:44 PM BST
pi@LAB:/home/pi $ date # 1000's of timeout msg/sec - reattached antenna
Tue 06 Oct 2020 08:40:06 PM BST

2020-10-06 20:41:45,030 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,030 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,030 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,031 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,031 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,031 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,032 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,032 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,032 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,032 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,032 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,033 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,033 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,033 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,033 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,033 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,034 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,034 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,034 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,034 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,034 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,035 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,035 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,035 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,035 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,035 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,036 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,036 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,036 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,036 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,047 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,047 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,047 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,048 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,048 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250
2020-10-06 20:41:45,048 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Read timed out. (read timeout=None) in /home/pi/legend-pi/hub.py:250

hub.py is unresponsive, will not recover, and produces no errors

restarted and unplugged antennae

intended timeout is taking more than 2minutes, and there is a strange split in the file_request_queue

Seeing retry timeout around 5min under normal operation. I reduced it to 10sec and the minimum retry to .1sec. I think the larger gaps and inconsistant timestamps on fire_request_queue are due to intermitent connectivity.

2020-10-06 14:02:38,498 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:04:12,151 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:09:10,995 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:10:37,383 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:11:04,435 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Max retries exceeded with url: /api/v0/form/save-sensor-data (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb60ac190>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:12:19,237 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Max retries exceeded with url: /api/v0/form/save-sensor-data (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb2cf36d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:36:20,887 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:45:51,102 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:46:16,680 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Max retries exceeded with url: /api/v0/form/save-sensor-data (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb2c513b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:48:28,034 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:51:26,304 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:52:56,700 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:54:17,905 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Max retries exceeded with url: /api/v0/form/save-sensor-data (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb36cdc10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:55:45,327 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 14:59:32,407 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 15:05:04,121 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 15:06:53,826 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 15:08:39,094 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-06 15:26:12,975 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240

pi@LAB:/home/pi/var/ramdisk/file_request_queue $ ls -lh
total 1.4G
-rw------- 1 pi pi  80 Oct  6 15:35 info
-rw-r--r-- 1 pi pi 37M Oct  5 23:05 q00037
-rw-r--r-- 1 pi pi 37M Oct  5 23:07 q00038
-rw-r--r-- 1 pi pi 37M Oct  5 23:09 q00039
-rw-r--r-- 1 pi pi 37M Oct  5 23:10 q00040
-rw-r--r-- 1 pi pi 26M Oct  5 23:12 q00041
-rw-r--r-- 1 pi pi 26M Oct  5 23:14 q00042
-rw-r--r-- 1 pi pi 25M Oct  5 23:15 q00043
-rw-r--r-- 1 pi pi 25M Oct  5 23:17 q00044
-rw-r--r-- 1 pi pi 26M Oct  5 23:19 q00045
-rw-r--r-- 1 pi pi 26M Oct  5 23:20 q00046
-rw-r--r-- 1 pi pi 26M Oct  5 23:22 q00047
-rw-r--r-- 1 pi pi 27M Oct  5 23:24 q00048
-rw-r--r-- 1 pi pi 26M Oct  5 23:25 q00049
-rw-r--r-- 1 pi pi 25M Oct  5 23:27 q00050
-rw-r--r-- 1 pi pi 25M Oct  5 23:29 q00051
-rw-r--r-- 1 pi pi 25M Oct  5 23:30 q00052
-rw-r--r-- 1 pi pi 26M Oct  5 23:32 q00053
-rw-r--r-- 1 pi pi 32M Oct  5 23:34 q00054
-rw-r--r-- 1 pi pi 37M Oct  5 23:35 q00055
-rw-r--r-- 1 pi pi 37M Oct  5 23:37 q00056
-rw-r--r-- 1 pi pi 37M Oct  5 23:39 q00057
-rw-r--r-- 1 pi pi 37M Oct  5 23:40 q00058
-rw-r--r-- 1 pi pi 37M Oct  5 23:42 q00059
-rw-r--r-- 1 pi pi 37M Oct  5 23:44 q00060
-rw-r--r-- 1 pi pi 37M Oct  5 23:45 q00061
-rw-r--r-- 1 pi pi 37M Oct  5 23:47 q00062
-rw-r--r-- 1 pi pi 37M Oct  5 23:49 q00063
-rw-r--r-- 1 pi pi 37M Oct  5 23:50 q00064
-rw-r--r-- 1 pi pi 37M Oct  5 23:52 q00065
-rw-r--r-- 1 pi pi 37M Oct  5 23:54 q00066
-rw-r--r-- 1 pi pi 37M Oct  5 23:55 q00067
-rw-r--r-- 1 pi pi 37M Oct  5 23:57 q00068
-rw-r--r-- 1 pi pi 37M Oct  6 06:32 q00069
-rw-r--r-- 1 pi pi 37M Oct  6 08:44 q00070
-rw-r--r-- 1 pi pi 37M Oct  6 09:37 q00071
-rw-r--r-- 1 pi pi 37M Oct  6 10:24 q00072
-rw-r--r-- 1 pi pi 37M Oct  6 11:10 q00073
-rw-r--r-- 1 pi pi 37M Oct  6 11:59 q00074
-rw-r--r-- 1 pi pi 37M Oct  6 13:00 q00075
-rw-r--r-- 1 pi pi 36M Oct  6 14:40 q00076
-rw-r--r-- 1 pi pi 34M Oct  6 15:35 q00077

errors seen after returning service after 9 days degraded

2020-10-05 19:52:33,778 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Max retries exceeded with url: /api/v0/form/save-sensor-data (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb2b19dd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) in /home/pi/legend-pi/hub.py:240
2020-10-05 19:54:28,379 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Max retries exceeded with url: /api/v0/form/save-sensor-data (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb2d9d130>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) in /home/pi/legend-pi/hub.py:240
2020-10-05 19:57:46,576 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-05 20:05:36,179 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-05 20:06:54,679 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-05 20:08:42,600 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240
2020-10-05 20:10:07,783 [ERROR]: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) in /home/pi/legend-pi/hub.py:240

First 413 occured

2020-09-23 23:19:45,728 [ERROR]: HTTPConnectionPool(host='live.legend-es.com', port=80): Max retries exceeded with url: /api/v0/form/save-sensor-data (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xacfe03b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) in /home/pi/legend-pi/hub.py:238
2020-09-23 23:46:43,896 [ERROR]: 413 for http://live.legend-es.com/api/v0/form/save-sensor-data with RESPONSE TEXT '<html>
<head><title>413 Request Entity Too Large</title></head>
<body bgcolor=white>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>
' HEADERS {'Server': 'nginx/1.10.3 (Ubuntu)', 'Date': 'Wed, 23 Sep 2020 22:46:43 GMT', 'Content-Type': 'text/html', 'Content-Length': '208', 'Connection': 'close'} REQUEST HEADERS {'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'Basic bGVnZW5kOnRoaW5rbGVnZW5k', 'Content-Length': '1086712', 'Content-Type': 'multipart/form-data; boundary=69eef1626dc31f0daa19aec1ac753fb6'} in /home/pi/legend-pi/hub.py:201
2020-09-23 23:47:01,778 [ERROR]: 413 for http://live.legend-es.com/api/v0/form/save-sensor-data with RESPONSE TEXT '<html>
<head><title>413 Request Entity Too Large</title></head>
<body bgcolor=white>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>
' HEADERS {'Server': 'nginx/1.10.3 (Ubuntu)', 'Date': 'Wed, 23 Sep 2020 22:46:46 GMT', 'Content-Type': 'text/html', 'Content-Length': '208', 'Connection': 'close'} REQUEST HEADERS {'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'Basic bGVnZW5kOnRoaW5rbGVnZW5k', 'Content-Length': '1086712', 'Content-Type': 'multipart/form-data; boundary=90beb613fc0bc8482eb413aaa1dab4f1'} in /home/pi/legend-pi/hub.py:201

When I first noticed

2020-09-24 14:43:45,894 [ERROR]: 413 for http://live.legend-es.com/api/v0/form/save-sensor-data with RESPONSE TEXT '

<title>413 Request Entity Too Large</title>

413 Request Entity Too Large

nginx/1.10.3 (Ubuntu) ' HEADERS {'Server': 'nginx/1.10.3 (Ubuntu)', 'Date': 'Thu, 24 Sep 2020 13:43:35 GMT', 'Content-Type': 'text/html', 'Content-Length': '208', 'Connection': 'close'} REQUEST HEADERS {'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'Basic bGVnZW5kOnRoaW5rbGVnZW5k', 'Content-Length': '3605973', 'Content-Type': 'multipart/form-data; boundary=0bc9e5d05a5c849dba26755d8d581039'} in /home/pi/legend-pi/hub.py:201
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment