logs for swift.log, proxy.log, account.log

I wouldn't worry too much about the timestamps, since in the logs the same or very similar messages seem to repeat/cycle at very regular intervals. So I went ahead and posted a few snippets:

Also, I scrambled some of the 'txn' labels before posting them here (similar to how I used <node_name> and <public_ip_address_of_single_node_setup> to mask these values here), but I don't think the real txn labels would be what's causing the problem. If these are needed though, let me know.

swift.log:

Aug 31 22:19:50 <node_name> swift: Handoff requested (1) (txn: tx2atjcoeldan14f1d8e0cb-0054039f86)
Aug 31 22:19:52 <node_name> swift: Handoff requested (1) (txn: tx20nadleocjtd450a96586-0054039f88)
Aug 31 22:19:56 <node_name> swift: Handoff requested (1) (txn: txf7cjeolnatr247bb9a283-0054039f8c)
Aug 31 22:23:36 <node_name> swift: Handoff requested (1) (txn: txd8lqjxaujwo3466c82f69-005403a068)
Aug 31 22:23:38 <node_name> swift: Handoff requested (1) (txn: tx84wudzjtroa44245a0543-005403a06a)
Aug 31 22:23:42 <node_name> swift: Handoff requested (1) (txn: tx88owjzkrwnz34effa3e8a-005403a06e)
Aug 31 22:28:19 <node_name> swift: Handoff requested (1) (txn: txd5fffwuznmp44d2f98209-005403a183)
Aug 31 22:28:21 <node_name> swift: Handoff requested (1) (txn: tx5fwevjkotyia4083a9f82-005403a185)
Aug 31 22:28:25 <node_name> swift: Handoff requested (1) (txn: tx23ghbuwilrse40ae935e1-005403a189)
(continues along this pattern, over and over...)

proxy-server.log:

Aug 31 22:20:42 <node_name> proxy-server: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck proxy_logging dlo memcache tempauth proxy_logging proxy".
Aug 31 22:20:42 <node_name> proxy-server: Adding required filter dlo to pipeline at position 2
Aug 31 22:20:42 <node_name> proxy-server: Adding required filter gatekeeper to pipeline at position 0
Aug 31 22:20:42 <node_name> proxy-server: Adding required filter catch_errors to pipeline at position 0
Aug 31 22:20:42 <node_name> proxy-server: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck proxy_logging dlo memcache tempauth proxy_logging proxy".
Aug 31 22:21:01 <node_name> proxy-server: <public_ip_address_of_single_node_setup> <public_ip_address_of_single_node_setup> 31/Aug/2014/22/21/01 GET /auth/v1.0 HTTP/1.0 200 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.15.3%20zlib/1.2.3%20libidn/1.18%20libssh2/1.4.2 - - - - tx9ac72e7e99d145e9959c2-0054039fcd - 0.0017 - - 1409523661.214817047 1409523661.216514111
Aug 31 22:23:07 <node_name> proxy-server: Handoff requested (1) (txn: tx4bjnoqrwl52d450d83e6e-005403a04b) (client_ip: <same_as_the_node_running_the_cluster_services>)
Aug 31 22:23:07 <node_name> proxy-server: <public_ip_address_of_single_node_setup> <public_ip_address_of_single_node_setup> 31/Aug/2014/22/23/07 GET /v1/AUTH_system HTTP/1.0 400 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.15.3%20zlib/1.2.3%20libidn/1.18%20libssh2/1.4.2 AUTH_tk18e9045fa6594ee58f0e733c388e02f9 - 41 - tx4b3fe766842d450d83e6e-005403a04b - 0.0096 - - 1409523787.908746004 1409523787.918334961


account-server.log:

Aug 31 22:23:36 <node_name> account-replicator: Beginning replication run
Aug 31 22:23:36 <node_name> account-replicator: Replication run OVER
Aug 31 22:23:36 <node_name> account-replicator: Attempted to replicate 0 dbs in 0.00235 seconds (0.00000/s)
Aug 31 22:23:36 <node_name> account-replicator: Removed 0 dbs
Aug 31 22:23:36 <node_name> account-replicator: 0 successes, 0 failures
Aug 31 22:23:36 <node_name> account-replicator: no_change:0 ts_repl:0 diff:0 rsync:0 diff_capped:0 hashmatch:0 empty:0
Aug 31 22:23:38 <node_name> account-server: <public_ip_address_of_single_node_setup> - - [31/Aug/2014:22:23:38 +0000] "HEAD /node3/sdd1/947/.expiring_objects" 400 47 "tx842969681c844245a0543-005403a06a" "HEAD http://localhost/v1/.expiring_objects" "proxy-server 27107" 0.0002 ""
Aug 31 22:23:38 <node_name> account-server: <public_ip_address_of_single_node_setup> - - [31/Aug/2014:22:23:38 +0000] "HEAD /node3/sde1/947/.expiring_objects" 400 47 "tx842969681c844245a0543-005403a06a" "HEAD http://localhost/v1/.expiring_objects" "proxy-server 27107" 0.0002 ""
Aug 31 22:23:38 <node_name> account-server: <public_ip_address_of_single_node_setup> - - [31/Aug/2014:22:23:38 +0000] "HEAD /node3/sdf1/947/.expiring_objects" 400 47 "tx842969681c844245a0543-005403a06a" "HEAD http://localhost/v1/.expiring_objects" "proxy-server 27107" 0.0001 ""
Aug 31 22:23:38 <node_name> account-server: <public_ip_address_of_single_node_setup> - - [31/Aug/2014:22:23:38 +0000] "HEAD /node3/sdc1/947/.expiring_objects" 400 47 "tx842969681c844245a0543-005403a06a" "HEAD http://localhost/v1/.expiring_objects" "proxy-server 27107" 0.0002 ""
Aug 31 22:23:42 <node_name> account-server: <public_ip_address_of_single_node_setup> - - [31/Aug/2014:22:23:42 +0000] "HEAD /node3/sde1/947/.expiring_objects" 400 47 "tx88b3a49f30a34effa3e8a-005403a06e" "HEAD http://localhost/v1/.expiring_objects" "proxy-server 27107" 0.0002 ""
Aug 31 22:23:42 <node_name> account-server: <public_ip_address_of_single_node_setup> - - [31/Aug/2014:22:23:42 +0000] "HEAD /node3/sdf1/947/.expiring_objects" 400 47 "tx88b3a49f30a34effa3e8a-005403a06e" "HEAD http://localhost/v1/.expiring_objects" "proxy-server 27107" 0.0001 ""
Aug 31 22:23:42 <node_name> account-server: <public_ip_address_of_single_node_setup> - - [31/Aug/2014:22:23:42 +0000] "HEAD /node3/sdd1/947/.expiring_objects" 400 47 "tx88b3a49f30a34effa3e8a-005403a06e" "HEAD http://localhost/v1/.expiring_objects" "proxy-server 27107" 0.0002 ""
Aug 31 22:23:42 <node_name> account-server: <public_ip_address_of_single_node_setup> - - [31/Aug/2014:22:23:42 +0000] "HEAD /node3/sdc1/947/.expiring_objects" 400 47 "tx88b3a49f30a34effa3e8a-005403a06e" "HEAD http://localhost/v1/.expiring_objects" "proxy-server 27107" 0.0001 ""