Skip to content

Instantly share code, notes, and snippets.

@wolfman1956

wolfman1956/logs_couchdb-haproxy_intermittent_timeouts.txt

Created April 27, 2020 18:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save wolfman1956/8aa3d76f7898cb4e882a002adea3e3a2 to your computer and use it in GitHub Desktop.
Save wolfman1956/8aa3d76f7898cb4e882a002adea3e3a2 to your computer and use it in GitHub Desktop.
Download ZIP
Kazoo startup causes various timeout (intermittent)
Raw
logs_couchdb-haproxy_intermittent_timeouts.txt
last 20 lines from haproxy.log, with the string "e9cc159ebcb93062e51c5f38604ec77f":
# grep e9cc159ebcb93062e51c5f38604ec77f haproxy.log |tail -20
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.150] CouchDB-data CouchDB-data/ourservername01.kazoo.dsritelecom.ca 1/0/1/7/9 200 5418 - - ---- 3/3/1/1/0 0/0 "GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.159] CouchDB-data CouchDB-data/ourservername02.kazoo.dsritelecom.ca 2/0/0/8/10 200 42148 - - ---- 3/3/1/2/0 0/0 "GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx4.png HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.170] CouchDB-data CouchDB-data/ourservername03.kazoo.dsritelecom.ca 1/0/0/8/9 200 303 - - ---- 3/3/1/1/0 0/0 "HEAD /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.180] CouchDB-data CouchDB-data/ourservername01.kazoo.dsritelecom.ca 1/0/1/101/103 200 352 - - ---- 3/3/1/1/0 0/0 "DELETE /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx4.png?rev=6521-0dafa254eb0675d5e599490cc1044b81 HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.284] CouchDB-data CouchDB-data/ourservername02.kazoo.dsritelecom.ca 1/0/0/10/11 200 5277 - - ---- 3/3/1/2/0 0/0 "GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.295] CouchDB-data CouchDB-data/ourservername03.kazoo.dsritelecom.ca 1/0/0/8/11 200 29138 - - ---- 3/3/1/1/0 0/0 "GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx5.png HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.307] CouchDB-data CouchDB-data/ourservername01.kazoo.dsritelecom.ca 1/0/0/7/8 200 303 - - ---- 3/3/1/1/0 0/0 "HEAD /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.316] CouchDB-data CouchDB-data/ourservername02.kazoo.dsritelecom.ca 1/0/1/103/105 200 352 - - ---- 3/3/1/2/0 0/0 "DELETE /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx5.png?rev=6522-ba66413c0f8db98dda4a9c83223644bc HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.421] CouchDB-data CouchDB-data/ourservername03.kazoo.dsritelecom.ca 1/0/1/11/13 200 5119 - - ---- 3/3/1/1/0 0/0 "GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.434] CouchDB-data CouchDB-data/ourservername01.kazoo.dsritelecom.ca 2/0/0/119/121 201 484 - - ---- 2/2/0/1/0 0/0 "PUT /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/SmartPBX_app.png?rev=6523-67f5f9c327d2ac4aa2dc6995909f2a26 HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.556] CouchDB-data CouchDB-data/ourservername02.kazoo.dsritelecom.ca 2/0/0/9/11 200 5280 - - ---- 2/2/0/1/0 0/0 "GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.567] CouchDB-data CouchDB-data/ourservername03.kazoo.dsritelecom.ca 2/0/0/126/128 201 481 - - ---- 2/2/0/1/0 0/0 "PUT /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx1.png?rev=6524-2cbdef3ae81dac5f0fd011825bf0dc38 HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.696] CouchDB-data CouchDB-data/ourservername01.kazoo.dsritelecom.ca 1/0/0/8/9 200 5421 - - ---- 2/2/0/1/0 0/0 "GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.705] CouchDB-data CouchDB-data/ourservername02.kazoo.dsritelecom.ca 2/0/0/119/121 201 481 - - ---- 2/2/0/1/0 0/0 "PUT /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx2.png?rev=6525-8160fb8ea5ad3fb5aab45c1ca99d4f06 HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.827] CouchDB-data CouchDB-data/ourservername03.kazoo.dsritelecom.ca 1/0/1/8/10 200 5562 - - ---- 2/2/0/1/0 0/0 "GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.837] CouchDB-data CouchDB-data/ourservername01.kazoo.dsritelecom.ca 1/0/1/124/126 201 481 - - ---- 2/2/0/1/0 0/0 "PUT /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx3.png?rev=6526-157dbdfbccfc97898303427b727a5cc3 HTTP/1.1"
Apr 27 15:39:53 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.963] CouchDB-data CouchDB-data/ourservername02.kazoo.dsritelecom.ca 2/0/0/9/11 200 5703 - - ---- 2/2/0/1/0 0/0 "GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f HTTP/1.1"
Apr 27 15:39:54 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:53.974] CouchDB-data CouchDB-data/ourservername03.kazoo.dsritelecom.ca 1/0/1/118/120 201 481 - - ---- 2/2/0/1/0 0/0 "PUT /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx4.png?rev=6527-829158e93a3f925315bb0bee4c9816c6 HTTP/1.1"
Apr 27 15:39:54 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:54.094] CouchDB-data CouchDB-data/ourservername01.kazoo.dsritelecom.ca 1/0/1/9/11 200 5844 - - ---- 2/2/0/1/0 0/0 "GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f HTTP/1.1"
Apr 27 15:39:54 ourservername01 haproxy[1269]: 127.0.0.1:58570 [27/Apr/2020:15:39:54.106] CouchDB-data CouchDB-data/ourservername02.kazoo.dsritelecom.ca 2/0/0/134/136 201 481 - - ---- 2/2/0/1/0 0/0 "PUT /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx5.png?rev=6528-87534a2e362d32c7642307179dc4592f HTTP/1.1"
==================================================================================
grep e9cc159ebcb93062e51c5f38604ec77f couchdb.log
[notice] 2020-04-27T15:39:52.569290Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.8259.442> e8263aecb2 127.0.0.1:15984 192.168.11.24 admin GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/SmartPBX_app.png 200 ok 8
[notice] 2020-04-27T15:39:52.718731Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.9644.442> f9f53b31fe 127.0.0.1:15984 192.168.11.24 admin GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f 200 ok 7
[notice] 2020-04-27T15:39:52.861441Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.11643.442> 836e4ce362 127.0.0.1:15984
192.168.11.24 admin DELETE /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx1.png?rev=6518-8c0261da7da89f4799fc5e876f53604a 200 ok 118
[notice] 2020-04-27T15:39:52.891667Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.5382.442> 61c663a643 127.0.0.1:15984 192.168.11.24 admin HEAD /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f 200 ok 7
[notice] 2020-04-27T15:39:53.030955Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.23181.441> 9dbde9f0cd 127.0.0.1:15984
192.168.11.24 admin GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx3.png 200
ok 7
[notice] 2020-04-27T15:39:53.160369Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.12355.442> 2f13673762 127.0.0.1:15984
192.168.11.24 admin GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f 200 ok 7
[notice] 2020-04-27T15:39:53.284973Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.15481.441> d21f7cf8e9 127.0.0.1:15984
192.168.11.24 admin DELETE /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx4.png?rev=6521-0dafa254eb0675d5e599490cc1044b81 200 ok 101
[notice] 2020-04-27T15:39:53.316722Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.6240.442> c2c59c2221 127.0.0.1:15984 192.168.11.24 admin HEAD /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f 200 ok 7
[notice] 2020-04-27T15:39:53.556500Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.30331.441> 8757b8059e 127.0.0.1:15984
192.168.11.24 admin PUT /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/SmartPBX_app.png?rev=6523-67f5f9c327d2ac4aa2dc6995909f2a26 201 ok 115
[notice] 2020-04-27T15:39:53.706118Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.13296.442> d92afd1d8a 127.0.0.1:15984
192.168.11.24 admin GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f 200 ok 7
[notice] 2020-04-27T15:39:53.964271Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.6201.442> 243db66969 127.0.0.1:15984 192.168.11.24 admin PUT /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f/smartpbx3.png?rev=6526-157dbdfbccfc97898303427b727a5cc3 201 ok 122
[notice] 2020-04-27T15:39:54.106972Z couchdb@ourservername01.kazoo.dsritelecom.ca <0.31392.441> 29f279fbe9 127.0.0.1:15984
192.168.11.24 admin GET /account%2F09%2F44%2F1c0eaf176ba59570fea2f287d23b/e9cc159ebcb93062e51c5f38604ec77f 200 ok 9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment