Skip to content

Instantly share code, notes, and snippets.

@charz

charz/0. swift settings.md Secret

Last active August 29, 2015 14:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save charz/5e53c3f735f736bc6ced to your computer and use it in GitHub Desktop.
Save charz/5e53c3f735f736bc6ced to your computer and use it in GitHub Desktop.
Download ZIP
missing .durable files
Raw
0. swift settings.md

swift version

Swift Package Version:           2.3.0.3-1~trusty
(https://github.com/swiftstack/swift/releases/tag/ss-release-2.3.0.3)

ring setting

[storage-policy:1]
name = ec42
default = no
deprecated = no

policy_type = erasure_coding

ec_type = isa_l_rs_vand
ec_num_data_fragments = 4
ec_num_parity_fragments = 2
ec_object_segment_size = 1048576

reconstructor, but I stop it during the tests

[object-reconstructor]
run_pause = 30
concurrency = 1
stats_interval = 300
http_timeout = 60
node_timeout = 10
lockup_timeout = 1800
reclaim_age = 604800
ring_check_interval = 15
disk_chunk_size = 1048576
Raw
1. ssbench result.md

1. Stop swift-object-reconstructor

2. Run ssbench(two benchmark nodes, VMs) with 1MB PUT (fixed object size and keep objects) against a two nodes cluster (VMs)

## Scenario file
{
  "name": "1MB-PUT test scenario",
  "sizes": [{
    "name": "1MB-PUT",
    "size_min": 1048576,
    "size_max": 1048576
  }],
  "initial_files": {
    "1MB-PUT": 100
  },
  "operation_count": 10000,
  "crud_profile": [1, 0, 0, 0],
  "user_count": 100,
  "container_base": "ssbench_1MB-PUT_0803_142640",
  "container_count": 100,
  "container_concurrency": 2
}

## Trigger ssbench-master and workers
$ ssbench-master run-scenario \
  -f /opt/benchmark/fullness/1MB-PUT_0803_142640.sc \
  -S http://192.168.200.21/v1/AUTH_demo \
  -S http://192.168.200.23/v1/AUTH_demo \
  -T AUTH_tk9d40839522a346f38f2e82d2272ef63b \
  --keep-objects \
  -u 30 -r 10 --worker 2 

## Benchmark result
[
  {
    "retries": 0, 
    "max_latency": 11.817417000000001, 
    "errors": 0, 
    "bk_errors": 2, 
    "scenario": "1MB-PUT_0803_142009", 
    "reqs_sec": 9.1761824429547314, 
    "duration": 15.147911548633289, 
    "avg_latency": 5.1827930000000002, 
    "start": "2015-08-03 14:21:05 UTC", 
    "objects": 139, 
    "concurrency": 60, 
    "policy": "ec42"
  }, 
  {
    "retries": 0, 
    "max_latency": 13.380112, 
    "errors": 0, 
    "bk_errors": 18, 
    "scenario": "1MB-PUT_0803_142130", 
    "reqs_sec": 7.1466693136990944, 
    "duration": 15.25186002255111, 
    "avg_latency": 7.0867690000000003, 
    "start": "2015-08-03 14:22:33 UTC", 
    "objects": 109, 
    "concurrency": 60, 
    "policy": "ec42"
  }, 
  {
    "retries": 0, 
    "max_latency": 15.825599, 
    "errors": 0, 
    "bk_errors": 217, 
    "scenario": "1MB-PUT_0803_142258", 
    "reqs_sec": 6.1170025969697761, 
    "duration": 16.020918488500719, 
    "avg_latency": 8.3536415000000002, 
    "start": "2015-08-03 14:24:18 UTC", 
    "objects": 98, 
    "concurrency": 60, 
    "policy": "ec42"
  }, 
  {
    "retries": 10, 
    "max_latency": 18.575337000000001, 
    "errors": 0, 
    "bk_errors": 49, 
    "scenario": "1MB-PUT_0803_142445", 
    "reqs_sec": 4.3360116451447235, 
    "duration": 19.603268384940638, 
    "avg_latency": 10.525769499999999, 
    "start": "2015-08-03 14:26:02 UTC", 
    "objects": 85, 
    "concurrency": 60, 
    "policy": "ec42"
  }, 
  {
    "retries": 26, 
    "max_latency": 24.187833000000001, 
    "errors": 0, 
    "bk_errors": 92, 
    "scenario": "1MB-PUT_0803_142640", 
    "reqs_sec": 3.3335178513575205, 
    "duration": 23.698688149466051, 
    "avg_latency": 12.516536500000001, 
    "start": "2015-08-03 14:28:24 UTC", 
    "objects": 79, 
    "concurrency": 60, 
    "policy": "ec42"
  }
]

3. after benchmark

vagrant@dev21:~$ swift stat
                    Account: AUTH_demo
                 Containers: 500
                    Objects: 1006
                      Bytes: 1054867456
Containers in policy "ec42": 500
   Objects in policy "ec42": 1006
     Bytes in policy "ec42": 1054867456
                X-Timestamp: 1438610647.50616
                 X-Trans-Id: tx8d1a76b28557489ab7d8f-0055bf8f91
               Content-Type: text/plain; charset=utf-8
              Accept-Ranges: bytes
Raw
2. check data and durable.md 
[vagrant@dev23 node]$ date;find ./ -name *#*.data |wc -l ; find ./ -name *.durable |wc -l;date
Mon Aug  3 14:03:58 UTC 2015
0
0
Mon Aug  3 14:03:59 UTC 2015

[vagrant@dev23 node]$ date;find ./ -name *#*.data |wc -l ; find ./ -name *.durable |wc -l;date
Mon Aug  3 14:12:42 UTC 2015
3035
3023
Mon Aug  3 14:12:42 UTC 2015

vagrant@dev21:/srv/node$ date;find ./ -name *#*.data |wc -l ; find ./ -name *.durable |wc -l;date
Mon Aug  3 14:20:21 UTC 2015
0
0
Mon Aug  3 14:20:21 UTC 2015

vagrant@dev21:/srv/node$ date;find ./ -name *#*.data |wc -l ; find ./ -name *.durable |wc -l;date
Mon Aug  3 14:29:25 UTC 2015
3051
3018
Mon Aug  3 14:29:26 UTC 2015
Raw
3. dig into one disk on dev21.md 
$ cd /srv/node/d0/objects-1
/srv/node/d0/objects-1 $ tree |less
├── 279
│   └── 092
│       └── 45ea1ea9fd44ff8cb86677d61a7c4092
│           ├── 1438610853.90021#3.data
│           ├── 1438610853.90021.durable
│           └── 1438611820.58758#3.data

├── 293
│   ├── 8c7
│   │   └── 494a8bd2e50c14706a6a340845eba8c7
│   │       ├── 1438611819.14361#5.data
│   │       └── 1438611823.58215#5.data


├── 32
│   └── 84c
│       └── 082bded1708c18a59497c443d718484c
│           ├── 1438612106.68606#5.data
│           ├── 1438612114.57488#5.data
│           └── 1438612114.57488.durable


├── 32
│   └── 84c
│       └── 082bded1708c18a59497c443d718484c
│           ├── 1438612106.68606#5.data
│           ├── 1438612114.57488#5.data
│           └── 1438612114.57488.durable


├── 39
│   ├── 204
│   │   └── 09cff93db64af47aa171e03880e96204
│   │       ├── 1438610837.56836#0.data
│   │       ├── 1438611811.20652#0.data
│   │       └── 1438611811.20652.durable


├── 397
│   ├── 400
│   │   └── 63627865d5c41f0ca35472df4e576400
│   │       ├── 1438611115.78335#4.data
│   │       ├── 1438611115.78335.durable
│   │       └── 1438612105.92040#4.data

├── 44
│   ├── 217
│   │   └── 0b2f6b0f9b3da2e57032fe8ab110c217
│   │       ├── 1438611118.43732#1.data
│   │       ├── 1438611118.43732.durable
│   │       └── 1438612105.68049#1.data
Raw
4.1. check 1st objects.md
  • object
├── 279
│   └── 092
│       └── 45ea1ea9fd44ff8cb86677d61a7c4092
│           ├── 1438610853.90021#3.data
│           ├── 1438610853.90021.durable
│           └── 1438611820.58758#3.data
  • swift-object-info (1438610853.90021#3.data)
vagrant@dev21:/srv/node/d0/objects-1$ sudo -i swift-object-info /srv/node/d0/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092/1438610853.90021#3.data
Path: /AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049
  Account: AUTH_demo
  Container: ssbench_1MB-PUT_0803_142258_000048_ec42
  Object: 1MB-PUT_000049
  Object hash: 45ea1ea9fd44ff8cb86677d61a7c4092
Content-Type: application/octet-stream
Timestamp: 2015-08-03T14:07:33.900210 (1438610853.90021)
System Metadata:
  X-Object-Sysmeta-Ec-Content-Length: 1048576
  X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
  X-Object-Sysmeta-Ec-Frag-Index: 3
  X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
  X-Object-Sysmeta-Ec-Segment-Size: 1048576
User Metadata:
  No metadata found
Other Metadata:
  No metadata found
ETag: 254c278788b0651a1e59c650c5227632 (valid)
Content-Length: 262224 (valid)
Partition	279
Hash     	45ea1ea9fd44ff8cb86677d61a7c4092

Server:Port Device	192.168.200.21:6000 d2
Server:Port Device	192.168.200.23:6000 d7
Server:Port Device	192.168.200.23:6000 d4
Server:Port Device	192.168.200.21:6000 d0
Server:Port Device	192.168.200.23:6000 d6
Server:Port Device	192.168.200.21:6000 d3
Server:Port Device	192.168.200.23:6000 d5	 [Handoff]
Server:Port Device	192.168.200.21:6000 d1	 [Handoff]


curl -I -XHEAD "http://192.168.200.21:6000/d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d7/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.21:6000/d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.21:6000/d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d5/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]
curl -I -XHEAD "http://192.168.200.21:6000/d1/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]


Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d2/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d7/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d4/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d0/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d6/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d3/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d5/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092" # [Handoff]
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d1/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092" # [Handoff]

note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node.
  • swift-object-info (1438611820.58758#3.data)
vagrant@dev21:/srv/node/d0/objects-1$ sudo -i swift-object-info /srv/node/d0/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092/1438611820.58758#3.data 
Path: /AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049
  Account: AUTH_demo
  Container: ssbench_1MB-PUT_0803_142258_000048_ec42
  Object: 1MB-PUT_000049
  Object hash: 45ea1ea9fd44ff8cb86677d61a7c4092
Content-Type: application/octet-stream
Timestamp: 2015-08-03T14:23:40.587580 (1438611820.58758)
System Metadata:
  X-Object-Sysmeta-Ec-Content-Length: 1048576
  X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
  X-Object-Sysmeta-Ec-Frag-Index: 3
  X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
  X-Object-Sysmeta-Ec-Segment-Size: 1048576
User Metadata:
  No metadata found
Other Metadata:
  No metadata found
ETag: 254c278788b0651a1e59c650c5227632 (valid)
Content-Length: 262224 (valid)
Partition	279
Hash     	45ea1ea9fd44ff8cb86677d61a7c4092

Server:Port Device	192.168.200.21:6000 d2
Server:Port Device	192.168.200.23:6000 d7
Server:Port Device	192.168.200.23:6000 d4
Server:Port Device	192.168.200.21:6000 d0
Server:Port Device	192.168.200.23:6000 d6
Server:Port Device	192.168.200.21:6000 d3
Server:Port Device	192.168.200.23:6000 d5	 [Handoff]
Server:Port Device	192.168.200.21:6000 d1	 [Handoff]


curl -I -XHEAD "http://192.168.200.21:6000/d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d7/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.21:6000/d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.21:6000/d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d5/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]
curl -I -XHEAD "http://192.168.200.21:6000/d1/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]


Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d2/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d7/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d4/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d0/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d6/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d3/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d5/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092" # [Handoff]
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d1/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092" # [Handoff]

note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node.

list all objects from disks

vagrant@dev21:/srv/node/d0/objects-1$ ls -lah /srv/node*/d?/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092
/srv/node/d0/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092:
total 532K
drwxr-xr-x 2 swift swift   97 Aug  3 14:24 .
drwxr-xr-x 3 swift swift   45 Aug  3 14:23 ..
-rw------- 1 swift swift 257K Aug  3 14:24 1438610853.90021#3.data
-rw-r--r-- 1 swift swift    0 Aug  3 14:24 1438610853.90021.durable
-rw------- 1 swift swift 257K Aug  3 14:23 1438611820.58758#3.data

/srv/node/d2/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092:
total 532K
drwxr-xr-x 2 swift swift   97 Aug  3 14:24 .
drwxr-xr-x 3 swift swift   45 Aug  3 14:23 ..
-rw------- 1 swift swift 257K Aug  3 14:24 1438610853.90021#0.data
-rw-r--r-- 1 swift swift    0 Aug  3 14:24 1438610853.90021.durable
-rw------- 1 swift swift 257K Aug  3 14:23 1438611820.58758#0.data

/srv/node/d3/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092:
total 532K
drwxr-xr-x 2 swift swift   97 Aug  3 14:24 .
drwxr-xr-x 3 swift swift   45 Aug  3 14:23 ..
-rw------- 1 swift swift 257K Aug  3 14:24 1438610853.90021#5.data
-rw-r--r-- 1 swift swift    0 Aug  3 14:24 1438610853.90021.durable
-rw------- 1 swift swift 257K Aug  3 14:23 1438611820.58758#5.data

[vagrant@dev23 node]$ ls -lah /srv/node*/d?/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092
/srv/node/d4/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092:
total 528K
drwxr-xr-x. 2 swift swift   97 Aug  3 14:07 .
drwxr-xr-x. 3 swift swift   45 Aug  3 14:07 ..
-rw-------. 1 swift swift 257K Aug  3 14:07 1438610853.90021#2.data
-rw-r--r--. 1 swift swift    0 Aug  3 14:07 1438610853.90021.durable
-rw-------. 1 swift swift 257K Aug  3 14:07 1438611820.58758#2.data

/srv/node/d6/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092:
total 528K
drwxr-xr-x. 2 swift swift   97 Aug  3 14:07 .
drwxr-xr-x. 3 swift swift   45 Aug  3 14:07 ..
-rw-------. 1 swift swift 257K Aug  3 14:07 1438610853.90021#4.data
-rw-r--r--. 1 swift swift    0 Aug  3 14:07 1438610853.90021.durable
-rw-------. 1 swift swift 257K Aug  3 14:07 1438611820.58758#4.data

/srv/node/d7/objects-1/279/092/45ea1ea9fd44ff8cb86677d61a7c4092:
total 528K
drwxr-xr-x. 2 swift swift   97 Aug  3 14:07 .
drwxr-xr-x. 3 swift swift   45 Aug  3 14:07 ..
-rw-------. 1 swift swift 257K Aug  3 14:07 1438610853.90021#1.data
-rw-r--r--. 1 swift swift    0 Aug  3 14:07 1438610853.90021.durable
-rw-------. 1 swift swift 257K Aug  3 14:07 1438611820.58758#1.data

Retrieve all frag frmo object server (6 success, no failure)

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.21:6000/d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Frag-Index: 0
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:07:34 GMT
Etag: "dcde2e7caf1e874d80ad3c929b4cb1f7"
X-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 15:41:13 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.23:6000/d7/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Frag-Index: 1
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:07:34 GMT
Etag: "bfbd583afd01407bdf2e6dafd38fe4f8"
X-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 15:22:10 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.23:6000/d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Frag-Index: 2
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:07:34 GMT
Etag: "026634889036bde8a6b08036a4d3957d"
X-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 15:22:10 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.21:6000/d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Frag-Index: 3
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:07:34 GMT
Etag: "254c278788b0651a1e59c650c5227632"
X-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 15:41:13 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.23:6000/d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Frag-Index: 4
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:07:34 GMT
Etag: "491a7cae7d8f834d66ccba3d22543354"
X-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 15:22:11 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.21:6000/d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Frag-Index: 5
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:07:34 GMT
Etag: "ab4a36c2c0bd13a8f9eb01c1abefb1bc"
X-Timestamp: 1438610853.90021
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 15:41:14 GMT

No frag in handoff

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.23:6000/d5/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Mon, 03 Aug 2015 15:23:27 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.21:6000/d1/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Mon, 03 Aug 2015 15:42:35 GMT

Find 7 related transaction ids (dev21)

vagrant@dev21:~$ sudo grep "txbc4f2ef5f4ab48378fa68-0055bf7593" /var/log/swift/all.log
Aug  3 14:23:38 localhost container-server: 192.168.200.23 - - [03/Aug/2015:14:23:38 +0000] "HEAD /d0/14962/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42" 204 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42" "txbc4f2ef5f4ab48378fa68-0055bf7593" "proxy-server 3968" 0.0216 "-" 31519 1
Aug  3 14:23:38 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:38 +0000] "HEAD /d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txbc4f2ef5f4ab48378fa68-0055bf7593" "proxy-server 3968" 0.0673 "-" 31511 1
Aug  3 14:23:39 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:39 +0000] "HEAD /d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txbc4f2ef5f4ab48378fa68-0055bf7593" "proxy-server 3968" 0.2403 "-" 31515 1
Aug  3 14:23:40 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:40 +0000] "HEAD /d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txbc4f2ef5f4ab48378fa68-0055bf7593" "proxy-server 3968" 0.1112 "-" 31511 1
Aug  3 14:23:40 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:40 +0000] "HEAD /d1/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txbc4f2ef5f4ab48378fa68-0055bf7593" "proxy-server 3968" 0.0388 "-" 31511 1
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ sudo grep "txadc9b713bc2746cebd4d0-0055bf796c" /var/log/swift/all.log
Aug  3 14:23:40 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:40 +0000] "HEAD /d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txadc9b713bc2746cebd4d0-0055bf796c" "proxy-server 31531" 0.0002 "-" 31522 1
Aug  3 14:23:41 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:41 +0000] "HEAD /d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txadc9b713bc2746cebd4d0-0055bf796c" "proxy-server 31531" 0.0243 "-" 31511 1
Aug  3 14:23:45 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:42 +0000] "HEAD /d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txadc9b713bc2746cebd4d0-0055bf796c" "proxy-server 31531" 0.1095 "-" 31514 1
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ sudo grep "txe92b81e60fcc482c9a6b7-0055bf759e" /var/log/swift/all.log
Aug  3 14:23:50 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:50 +0000] "HEAD /d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.3386 "-" 31511 1
Aug  3 14:23:53 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:50 +0000] "HEAD /d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.0003 "-" 31522 1
Aug  3 14:23:54 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:54 +0000] "HEAD /d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.2519 "-" 31514 1
Aug  3 14:23:55 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:55 +0000] "HEAD /d1/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.1567 "-" 31522 1
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ sudo grep "tx6343e3f0e0d14880b3daa-0055bf796c" /var/log/swift/all.log
Aug  3 14:23:51 localhost proxy-server: Not enough object servers ack'ed (got 3) (txn: tx6343e3f0e0d14880b3daa-0055bf796c) (client_ip: 192.168.200.32)
Aug  3 14:23:51 localhost proxy-server: 192.168.200.32 192.168.200.32 03/Aug/2015/14/23/51 PUT /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 HTTP/1.0 503 - - demo%2CAUTH_tk9d40... 1048576 118 - tx6343e3f0e0d14880b3daa-0055bf796c - 11.1730 - - 1438611820.579284906 1438611831.752264023 1
Aug  3 14:23:54 localhost object-server: ERROR __call__ error with PUT /d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 : #012Traceback (most recent call last):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/usr/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/usr/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: tx6343e3f0e0d14880b3daa-0055bf796c)
Aug  3 14:23:55 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:54 +0000] "PUT /d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 500 1478 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "tx6343e3f0e0d14880b3daa-0055bf796c" "proxy-server 31534" 13.8574 "-" 31516 1
Aug  3 14:23:55 localhost object-server: ERROR __call__ error with PUT /d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 : #012Traceback (most recent call last):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/usr/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/usr/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: tx6343e3f0e0d14880b3daa-0055bf796c)
Aug  3 14:23:55 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:55 +0000] "PUT /d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 500 1478 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "tx6343e3f0e0d14880b3daa-0055bf796c" "proxy-server 31534" 14.2679 "-" 31516 1
Aug  3 14:23:55 localhost object-server: ERROR __call__ error with PUT /d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 : #012Traceback (most recent call last):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/usr/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/usr/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: tx6343e3f0e0d14880b3daa-0055bf796c)
Aug  3 14:23:55 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:55 +0000] "PUT /d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 500 1478 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "tx6343e3f0e0d14880b3daa-0055bf796c" "proxy-server 31534" 14.8341 "-" 31516 1
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ sudo grep "txe92b81e60fcc482c9a6b7-0055bf759e" /var/log/swift/all.log
Aug  3 14:23:50 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:50 +0000] "HEAD /d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.3386 "-" 31511 1
Aug  3 14:23:53 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:50 +0000] "HEAD /d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.0003 "-" 31522 1
Aug  3 14:23:54 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:54 +0000] "HEAD /d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.2519 "-" 31514 1
Aug  3 14:23:55 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:55 +0000] "HEAD /d1/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.1567 "-" 31522 1
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ sudo grep "txdda9016dd5ad417c963ef-0055bf75a1" /var/log/swift/all.log
Aug  3 14:24:02 localhost container-server: 192.168.200.21 - - [03/Aug/2015:14:24:02 +0000] "PUT /d0/14962/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txdda9016dd5ad417c963ef-0055bf75a1" "object-server 31511" 0.0006 "-" 31517 1
Aug  3 14:24:03 localhost container-server: 192.168.200.21 - - [03/Aug/2015:14:24:03 +0000] "PUT /d3/14962/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txdda9016dd5ad417c963ef-0055bf75a1" "object-server 31511" 0.0007 "-" 31519 1
Aug  3 14:24:03 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:24:03 +0000] "PUT /d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txdda9016dd5ad417c963ef-0055bf75a1" "proxy-server 3965" 11.5525 "-" 31511 1
Aug  3 14:24:04 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:24:03 +0000] "PUT /d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txdda9016dd5ad417c963ef-0055bf75a1" "proxy-server 3965" 11.4560 "-" 31511 1
Aug  3 14:24:04 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:24:03 +0000] "PUT /d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txdda9016dd5ad417c963ef-0055bf75a1" "proxy-server 3965" 11.3463 "-" 31511 1
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ 
vagrant@dev21:~$ sudo grep "txc04547dab44c4bc4b73f0-0055bf75a5" /var/log/swift/all.log
Aug  3 14:24:06 localhost container-server: 192.168.200.21 - - [03/Aug/2015:14:24:06 +0000] "PUT /d0/14962/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txc04547dab44c4bc4b73f0-0055bf75a5" "object-server 31520" 0.0011 "-" 31519 1
Aug  3 14:24:06 localhost container-server: 192.168.200.21 - - [03/Aug/2015:14:24:06 +0000] "PUT /d3/14962/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txc04547dab44c4bc4b73f0-0055bf75a5" "object-server 31520" 0.0007 "-" 31517 1
Aug  3 14:24:07 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:24:07 +0000] "PUT /d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txc04547dab44c4bc4b73f0-0055bf75a5" "proxy-server 3965" 10.6993 "-" 31520 1
Aug  3 14:24:07 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:24:07 +0000] "PUT /d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txc04547dab44c4bc4b73f0-0055bf75a5" "proxy-server 3965" 10.8916 "-" 31520 1
Aug  3 14:24:07 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:24:07 +0000] "PUT /d2/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txc04547dab44c4bc4b73f0-0055bf75a5" "proxy-server 3965" 10.5420 "-" 31520 1

Find 7 related transaction ids (dev23)

[vagrant@dev23 node]$ sudo grep "txbc4f2ef5f4ab48378fa68-0055bf7593" /var/log/swift/all.log
Aug  3 14:07:18 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:18 +0000] "HEAD /d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txbc4f2ef5f4ab48378fa68-0055bf7593" "proxy-server 3968" 0.0009 "-" 3953 1
Aug  3 14:07:18 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:18 +0000] "HEAD /d7/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txbc4f2ef5f4ab48378fa68-0055bf7593" "proxy-server 3968" 0.0005 "-" 3959 1
Aug  3 14:07:18 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:18 +0000] "HEAD /d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txbc4f2ef5f4ab48378fa68-0055bf7593" "proxy-server 3968" 0.0007 "-" 3953 1
Aug  3 14:07:18 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:18 +0000] "HEAD /d5/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txbc4f2ef5f4ab48378fa68-0055bf7593" "proxy-server 3968" 0.0006 "-" 3959 1
Aug  3 14:07:18 dev23 proxy-server: 192.168.200.32 192.168.200.32 03/Aug/2015/14/07/18 HEAD /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 HTTP/1.0 404 - - demo%2CAUTH_tk9d40... - - - txbc4f2ef5f4ab48378fa68-0055bf7593 - 2.4684 - - 1438610835.838107109 1438610838.306478977 1
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ sudo grep "txadc9b713bc2746cebd4d0-0055bf796c" /var/log/swift/all.log
Aug  3 14:07:24 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:24 +0000] "HEAD /d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txadc9b713bc2746cebd4d0-0055bf796c" "proxy-server 31531" 0.0261 "-" 3961 1
Aug  3 14:07:25 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:25 +0000] "HEAD /d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txadc9b713bc2746cebd4d0-0055bf796c" "proxy-server 31531" 0.0004 "-" 3958 1
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ sudo grep "txe92b81e60fcc482c9a6b7-0055bf759e" /var/log/swift/all.log
Aug  3 14:07:32 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:32 +0000] "HEAD /d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.0010 "-" 3953 1
Aug  3 14:07:32 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:32 +0000] "HEAD /d7/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.0054 "-" 3961 1
Aug  3 14:07:32 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:32 +0000] "HEAD /d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.0018 "-" 3958 1
Aug  3 14:07:32 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:32 +0000] "HEAD /d5/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.0007 "-" 3959 1
Aug  3 14:07:33 dev23 proxy-server: 192.168.200.31 192.168.200.31 03/Aug/2015/14/07/33 HEAD /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 HTTP/1.0 404 - - demo%2CAUTH_tk9d40... - - - txe92b81e60fcc482c9a6b7-0055bf759e - 6.7955 - - 1438610846.868506908 1438610853.663991928 1
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ sudo grep "tx6343e3f0e0d14880b3daa-0055bf796c" /var/log/swift/all.log
Aug  3 14:07:31 dev23 object-server: ERROR __call__ error with PUT /d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 : #012Traceback (most recent call last):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/opt/ss/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/opt/ss/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: tx6343e3f0e0d14880b3daa-0055bf796c)
Aug  3 14:07:31 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:31 +0000] "PUT /d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 500 1539 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "tx6343e3f0e0d14880b3daa-0055bf796c" "proxy-server 31534" 13.3888 "-" 3955 1
Aug  3 14:07:31 dev23 object-server: ERROR __call__ error with PUT /d7/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 : #012Traceback (most recent call last):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/opt/ss/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/opt/ss/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: tx6343e3f0e0d14880b3daa-0055bf796c)
Aug  3 14:07:31 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:31 +0000] "PUT /d7/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 500 1539 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "tx6343e3f0e0d14880b3daa-0055bf796c" "proxy-server 31534" 13.3875 "-" 3955 1
Aug  3 14:07:31 dev23 object-server: ERROR __call__ error with PUT /d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 : #012Traceback (most recent call last):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/opt/ss/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/opt/ss/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: tx6343e3f0e0d14880b3daa-0055bf796c)
Aug  3 14:07:31 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:31 +0000] "PUT /d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 500 1539 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "tx6343e3f0e0d14880b3daa-0055bf796c" "proxy-server 31534" 13.3993 "-" 3955 1
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ sudo grep "txe92b81e60fcc482c9a6b7-0055bf759e" /var/log/swift/all.log
Aug  3 14:07:32 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:32 +0000] "HEAD /d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.0010 "-" 3953 1
Aug  3 14:07:32 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:32 +0000] "HEAD /d7/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.0054 "-" 3961 1
Aug  3 14:07:32 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:32 +0000] "HEAD /d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.0018 "-" 3958 1
Aug  3 14:07:32 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:32 +0000] "HEAD /d5/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txe92b81e60fcc482c9a6b7-0055bf759e" "proxy-server 3963" 0.0007 "-" 3959 1
Aug  3 14:07:33 dev23 proxy-server: 192.168.200.31 192.168.200.31 03/Aug/2015/14/07/33 HEAD /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 HTTP/1.0 404 - - demo%2CAUTH_tk9d40... - - - txe92b81e60fcc482c9a6b7-0055bf759e - 6.7955 - - 1438610846.868506908 1438610853.663991928 1
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ sudo grep "txdda9016dd5ad417c963ef-0055bf75a1" /var/log/swift/all.log
Aug  3 14:07:39 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:39 +0000] "PUT /d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txdda9016dd5ad417c963ef-0055bf75a1" "proxy-server 3965" 9.7823 "-" 3953 1
Aug  3 14:07:39 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:39 +0000] "PUT /d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txdda9016dd5ad417c963ef-0055bf75a1" "proxy-server 3965" 9.8420 "-" 3953 1
Aug  3 14:07:39 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:39 +0000] "PUT /d7/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txdda9016dd5ad417c963ef-0055bf75a1" "proxy-server 3965" 9.8870 "-" 3953 1
Aug  3 14:07:40 dev23 proxy-server: 192.168.200.32 192.168.200.32 03/Aug/2015/14/07/40 PUT /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 HTTP/1.0 201 - - demo%2CAUTH_tk9d40... 1048576 - - txdda9016dd5ad417c963ef-0055bf75a1 - 10.4877 - - 1438610849.536551952 1438610860.024241924 1
Aug  3 14:07:40 dev23 container-server: 192.168.200.21 - - [03/Aug/2015:14:07:40 +0000] "PUT /d5/14962/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/d0/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txdda9016dd5ad417c963ef-0055bf75a1" "object-server 31511" 0.0012 "-" 3950 1
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ sudo grep "txc04547dab44c4bc4b73f0-0055bf75a5" /var/log/swift/all.log
Aug  3 14:07:43 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:43 +0000] "PUT /d7/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txc04547dab44c4bc4b73f0-0055bf75a5" "proxy-server 3965" 9.3701 "-" 3955 1
Aug  3 14:07:43 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:43 +0000] "PUT /d6/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txc04547dab44c4bc4b73f0-0055bf75a5" "proxy-server 3965" 9.4781 "-" 3958 1
Aug  3 14:07:43 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:43 +0000] "PUT /d4/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txc04547dab44c4bc4b73f0-0055bf75a5" "proxy-server 3965" 9.4510 "-" 3958 1
Aug  3 14:07:43 dev23 container-server: 192.168.200.21 - - [03/Aug/2015:14:07:43 +0000] "PUT /d5/14962/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" 201 - "PUT http://192.168.200.23/d3/279/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049" "txc04547dab44c4bc4b73f0-0055bf75a5" "object-server 31520" 0.0014 "-" 3950 1
Aug  3 14:07:43 dev23 proxy-server: 192.168.200.31 192.168.200.31 03/Aug/2015/14/07/43 PUT /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000048_ec42/1MB-PUT_000049 HTTP/1.0 201 - - demo%2CAUTH_tk9d40... 1048576 - - txc04547dab44c4bc4b73f0-0055bf75a5 - 10.1822 - - 1438610853.763317108 1438610863.945528984 1
Raw
4.2. check 2nd objects.md

object

├── 293
│   ├── 8c7
│   │   └── 494a8bd2e50c14706a6a340845eba8c7
│   │       ├── 1438611819.14361#5.data
│   │       └── 1438611823.58215#5.data

swift-object-info (1438611819.14361#5.data, missing .durable)

vagrant@dev21:/srv/node/d0/objects-1$ sudo -i swift-object-info /srv/node/d0/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7/1438611819.14361#5.data
Path: /AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044
  Account: AUTH_demo
  Container: ssbench_1MB-PUT_0803_142258_000043_ec42
  Object: 1MB-PUT_000044
  Object hash: 494a8bd2e50c14706a6a340845eba8c7
Content-Type: application/octet-stream
Timestamp: 2015-08-03T14:23:39.143610 (1438611819.14361)
System Metadata:
  X-Object-Sysmeta-Ec-Content-Length: 1048576
  X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
  X-Object-Sysmeta-Ec-Frag-Index: 5
  X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
  X-Object-Sysmeta-Ec-Segment-Size: 1048576
User Metadata:
  No metadata found
Other Metadata:
  No metadata found
ETag: ab4a36c2c0bd13a8f9eb01c1abefb1bc (valid)
Content-Length: 262224 (valid)
Partition	293
Hash     	494a8bd2e50c14706a6a340845eba8c7

Server:Port Device	192.168.200.23:6000 d7
Server:Port Device	192.168.200.21:6000 d3
Server:Port Device	192.168.200.23:6000 d4
Server:Port Device	192.168.200.21:6000 d2
Server:Port Device	192.168.200.23:6000 d6
Server:Port Device	192.168.200.21:6000 d0
Server:Port Device	192.168.200.23:6000 d5	 [Handoff]
Server:Port Device	192.168.200.21:6000 d1	 [Handoff]


curl -I -XHEAD "http://192.168.200.23:6000/d7/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.21:6000/d3/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d4/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.21:6000/d2/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d6/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.21:6000/d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d5/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]
curl -I -XHEAD "http://192.168.200.21:6000/d1/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]


Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d7/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d3/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d4/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d2/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d6/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d0/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d5/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7" # [Handoff]
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d1/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7" # [Handoff]

note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node.

swift-object-info (1438611823.58215#5.data, missing .durable)

vagrant@dev21:/srv/node/d0/objects-1$ sudo -i swift-object-info /srv/node/d0/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7/1438611823.58215#5.data
Path: /AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044
  Account: AUTH_demo
  Container: ssbench_1MB-PUT_0803_142258_000043_ec42
  Object: 1MB-PUT_000044
  Object hash: 494a8bd2e50c14706a6a340845eba8c7
Content-Type: application/octet-stream
Timestamp: 2015-08-03T14:23:43.582150 (1438611823.58215)
System Metadata:
  X-Object-Sysmeta-Ec-Content-Length: 1048576
  X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
  X-Object-Sysmeta-Ec-Frag-Index: 5
  X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
  X-Object-Sysmeta-Ec-Segment-Size: 1048576
User Metadata:
  No metadata found
Other Metadata:
  No metadata found
ETag: ab4a36c2c0bd13a8f9eb01c1abefb1bc (valid)
Content-Length: 262224 (valid)
Partition	293
Hash     	494a8bd2e50c14706a6a340845eba8c7

Server:Port Device	192.168.200.23:6000 d7
Server:Port Device	192.168.200.21:6000 d3
Server:Port Device	192.168.200.23:6000 d4
Server:Port Device	192.168.200.21:6000 d2
Server:Port Device	192.168.200.23:6000 d6
Server:Port Device	192.168.200.21:6000 d0
Server:Port Device	192.168.200.23:6000 d5	 [Handoff]
Server:Port Device	192.168.200.21:6000 d1	 [Handoff]


curl -I -XHEAD "http://192.168.200.23:6000/d7/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.21:6000/d3/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d4/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.21:6000/d2/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d6/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.21:6000/d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://192.168.200.23:6000/d5/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]
curl -I -XHEAD "http://192.168.200.21:6000/d1/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]


Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d7/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d3/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d4/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d2/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d6/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d0/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7"
ssh 192.168.200.23 "ls -lah ${DEVICE:-/srv/node*}/d5/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7" # [Handoff]
ssh 192.168.200.21 "ls -lah ${DEVICE:-/srv/node*}/d1/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7" # [Handoff]

note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node.

list all objects from disks

vagrant@dev21:/srv/node/d0/objects-1$ ls -lah /srv/node*/d?/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7
/srv/node/d0/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7:
total 532K
drwxr-xr-x 2 swift swift   66 Aug  3 14:23 .
drwxr-xr-x 3 swift swift   45 Aug  3 14:23 ..
-rw------- 1 swift swift 257K Aug  3 14:23 1438611819.14361#5.data
-rw------- 1 swift swift 257K Aug  3 14:23 1438611823.58215#5.data

/srv/node/d2/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7:
total 528K
drwxr-xr-x 2 swift swift   97 Aug  3 14:23 .
drwxr-xr-x 3 swift swift   45 Aug  3 14:23 ..
-rw------- 1 swift swift 257K Aug  3 14:23 1438611819.14361#3.data
-rw-r--r-- 1 swift swift    0 Aug  3 14:23 1438611819.14361.durable
-rw------- 1 swift swift 257K Aug  3 14:23 1438611823.58215#3.data

/srv/node/d3/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7:
total 532K
drwxr-xr-x 2 swift swift   97 Aug  3 14:23 .
drwxr-xr-x 3 swift swift   45 Aug  3 14:23 ..
-rw------- 1 swift swift 257K Aug  3 14:23 1438611819.14361#1.data
-rw-r--r-- 1 swift swift    0 Aug  3 14:23 1438611819.14361.durable
-rw------- 1 swift swift 257K Aug  3 14:23 1438611823.58215#1.data

[vagrant@dev23 node]$ ls -lah /srv/node*/d?/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7
/srv/node/d4/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7:
total 528K
drwxr-xr-x. 2 swift swift   97 Aug  3 14:07 .
drwxr-xr-x. 3 swift swift   45 Aug  3 14:07 ..
-rw-------. 1 swift swift 257K Aug  3 14:07 1438611819.14361#2.data
-rw-r--r--. 1 swift swift    0 Aug  3 14:07 1438611819.14361.durable
-rw-------. 1 swift swift 257K Aug  3 14:07 1438611823.58215#2.data

/srv/node/d6/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7:
total 528K
drwxr-xr-x. 2 swift swift   97 Aug  3 14:07 .
drwxr-xr-x. 3 swift swift   45 Aug  3 14:07 ..
-rw-------. 1 swift swift 257K Aug  3 14:07 1438611819.14361#4.data
-rw-r--r--. 1 swift swift    0 Aug  3 14:07 1438611819.14361.durable
-rw-------. 1 swift swift 257K Aug  3 14:07 1438611823.58215#4.data

/srv/node/d7/objects-1/293/8c7/494a8bd2e50c14706a6a340845eba8c7:
total 528K
drwxr-xr-x. 2 swift swift   97 Aug  3 14:07 .
drwxr-xr-x. 3 swift swift   45 Aug  3 14:07 ..
-rw-------. 1 swift swift 257K Aug  3 14:07 1438611819.14361#0.data
-rw-r--r--. 1 swift swift    0 Aug  3 14:07 1438611819.14361.durable
-rw-------. 1 swift swift 257K Aug  3 14:07 1438611823.58215#0.data

Retrieve all frag frmo object server (5 success, one failure because no .durable)

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.23:6000/d7/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438611819.14361
X-Object-Sysmeta-Ec-Frag-Index: 0
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:23:40 GMT
Etag: "dcde2e7caf1e874d80ad3c929b4cb1f7"
X-Timestamp: 1438611819.14361
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 14:44:53 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.21:6000/d3/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438611819.14361
X-Object-Sysmeta-Ec-Frag-Index: 1
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:23:40 GMT
Etag: "bfbd583afd01407bdf2e6dafd38fe4f8"
X-Timestamp: 1438611819.14361
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 15:02:23 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.23:6000/d4/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438611819.14361
X-Object-Sysmeta-Ec-Frag-Index: 2
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:23:40 GMT
Etag: "026634889036bde8a6b08036a4d3957d"
X-Timestamp: 1438611819.14361
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 14:44:53 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.21:6000/d2/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438611819.14361
X-Object-Sysmeta-Ec-Frag-Index: 3
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:23:40 GMT
Etag: "254c278788b0651a1e59c650c5227632"
X-Timestamp: 1438611819.14361
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 15:02:23 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.23:6000/d6/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 200 OK
Content-Length: 262224
X-Backend-Timestamp: 1438611819.14361
X-Object-Sysmeta-Ec-Frag-Index: 4
X-Object-Sysmeta-Ec-Content-Length: 1048576
X-Object-Sysmeta-Ec-Etag: e6065c4aa2ab1603008fc18410f579d4
Last-Modified: Mon, 03 Aug 2015 14:23:40 GMT
Etag: "491a7cae7d8f834d66ccba3d22543354"
X-Timestamp: 1438611819.14361
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_vand 4+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Mon, 03 Aug 2015 14:44:53 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.21:6000/d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Mon, 03 Aug 2015 15:02:25 GMT

No frag in handoff

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.23:6000/d5/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Mon, 03 Aug 2015 14:49:41 GMT

vagrant@dev21:/srv/node/d0/objects-1$ curl -I -XHEAD "http://192.168.200.21:6000/d1/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Mon, 03 Aug 2015 15:07:26 GMT

Find 5 related transaction ids (dev21)

vagrant@dev21:/srv/node/d0/objects-1$ sudo grep "txf23ae2ee17a14cc9b5e74-0055bf7593" /var/log/swift/all.log
Aug  3 14:23:37 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:37 +0000] "HEAD /d2/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txf23ae2ee17a14cc9b5e74-0055bf7593" "proxy-server 3965" 0.0002 "-" 31516 1
Aug  3 14:23:37 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:37 +0000] "HEAD /d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txf23ae2ee17a14cc9b5e74-0055bf7593" "proxy-server 3965" 0.0399 "-" 31514 1
Aug  3 14:23:38 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:38 +0000] "HEAD /d3/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txf23ae2ee17a14cc9b5e74-0055bf7593" "proxy-server 3965" 0.0587 "-" 31514 1
Aug  3 14:23:39 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:38 +0000] "HEAD /d1/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txf23ae2ee17a14cc9b5e74-0055bf7593" "proxy-server 3965" 0.2405 "-" 31514 1
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ sudo grep "tx529914b68d864dac9ecb2-0055bf796a" /var/log/swift/all.log
Aug  3 14:23:39 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:38 +0000] "HEAD /d2/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx529914b68d864dac9ecb2-0055bf796a" "proxy-server 31535" 0.1308 "-" 31514 1
Aug  3 14:23:40 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:39 +0000] "HEAD /d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx529914b68d864dac9ecb2-0055bf796a" "proxy-server 31535" 0.0003 "-" 31514 1
Aug  3 14:23:42 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:40 +0000] "HEAD /d3/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx529914b68d864dac9ecb2-0055bf796a" "proxy-server 31535" 0.0494 "-" 31514 1
Aug  3 14:23:43 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:42 +0000] "HEAD /d1/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx529914b68d864dac9ecb2-0055bf796a" "proxy-server 31535" 0.2560 "-" 31513 1
Aug  3 14:23:43 localhost proxy-server: 192.168.200.31 192.168.200.31 03/Aug/2015/14/23/43 HEAD /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 HTTP/1.0 404 - - demo%2CAUTH_tk9d40... - - - tx529914b68d864dac9ecb2-0055bf796a - 4.8388 - - 1438611818.586044073 1438611823.424827099 1
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ sudo grep "tx9eee5815960f4ac2be26e-0055bf796b" /var/log/swift/all.log
Aug  3 14:23:51 localhost proxy-server: 192.168.200.32 192.168.200.32 03/Aug/2015/14/23/51 PUT /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 HTTP/1.0 201 - - demo%2CAUTH_tk9d40... 1048576 - - tx9eee5815960f4ac2be26e-0055bf796b - 12.7657 - - 1438611819.059851885 1438611831.825532913 1
Aug  3 14:23:52 localhost container-server: 192.168.200.21 - - [03/Aug/2015:14:23:52 +0000] "PUT /d0/39337/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 201 - "PUT http://192.168.200.21/d2/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx9eee5815960f4ac2be26e-0055bf796b" "object-server 31522" 0.0009 "-" 31517 1
Aug  3 14:23:53 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:52 +0000] "PUT /d2/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 201 - "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx9eee5815960f4ac2be26e-0055bf796b" "proxy-server 31534" 13.5584 "-" 31522 1
Aug  3 14:23:54 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:23:53 +0000] "PUT /d3/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 201 - "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx9eee5815960f4ac2be26e-0055bf796b" "proxy-server 31534" 13.6425 "-" 31522 1
Aug  3 14:24:30 localhost object-server: ERROR __call__ error with PUT /d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 : #012Traceback (most recent call last):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/usr/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/usr/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: tx9eee5815960f4ac2be26e-0055bf796b)
Aug  3 14:24:30 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:24:30 +0000] "PUT /d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 500 1478 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx9eee5815960f4ac2be26e-0055bf796b" "proxy-server 31534" 51.4414 "-" 31520 1
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ sudo grep "tx677d22068a53484f8b9db-0055bf75a6" /var/log/swift/all.log
Aug  3 14:23:57 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:57 +0000] "PUT /d3/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 409 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx677d22068a53484f8b9db-0055bf75a6" "proxy-server 3968" 0.4353 "-" 31516 1
Aug  3 14:23:58 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:57 +0000] "PUT /d2/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 409 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx677d22068a53484f8b9db-0055bf75a6" "proxy-server 3968" 0.2182 "-" 31516 1
Aug  3 14:23:58 localhost object-server: ERROR __call__ error with PUT /d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 : #012Traceback (most recent call last):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 578, in PUT#012    _junk_hdrs, obj_input = next(mime_documents_iter)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 61, in iter_mime_headers_and_bodies#012    for file_like in mime_documents_iter:#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 3369, in iter_multipart_mime_documents#012    got = wsgi_input.readline(blen)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 193, in readline#012    return self._chunked_read(self.rfile, size, True)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: tx677d22068a53484f8b9db-0055bf75a6)
Aug  3 14:23:58 localhost object-server: 192.168.200.23 - - [03/Aug/2015:14:23:58 +0000] "PUT /d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 500 1134 "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx677d22068a53484f8b9db-0055bf75a6" "proxy-server 3968" 1.4567 "-" 31516 1
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ 
vagrant@dev21:/srv/node/d0/objects-1$ sudo grep "txce9fb25730b04ca9823f4-0055bf796f" /var/log/swift/all.log
Aug  3 14:23:56 localhost proxy-server: Not enough object servers ack'ed (got 3) (txn: txce9fb25730b04ca9823f4-0055bf796f) (client_ip: 192.168.200.31)
Aug  3 14:23:56 localhost proxy-server: 192.168.200.31 192.168.200.31 03/Aug/2015/14/23/56 PUT /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 HTTP/1.0 503 - - demo%2CAUTH_tk9d40... 1048576 118 - txce9fb25730b04ca9823f4-0055bf796f - 13.1230 - - 1438611823.544087887 1438611836.667085886 1
Aug  3 14:24:19 localhost object-server: ERROR __call__ error with PUT /d2/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 : #012Traceback (most recent call last):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/usr/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/usr/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: txce9fb25730b04ca9823f4-0055bf796f)
Aug  3 14:24:19 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:24:19 +0000] "PUT /d2/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 500 1478 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txce9fb25730b04ca9823f4-0055bf796f" "proxy-server 31527" 35.6704 "-" 31515 1
Aug  3 14:24:19 localhost object-server: ERROR __call__ error with PUT /d3/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 : #012Traceback (most recent call last):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/usr/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/usr/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: txce9fb25730b04ca9823f4-0055bf796f)
Aug  3 14:24:19 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:24:19 +0000] "PUT /d3/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 500 1478 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txce9fb25730b04ca9823f4-0055bf796f" "proxy-server 31527" 35.6019 "-" 31514 1
Aug  3 14:24:19 localhost object-server: ERROR __call__ error with PUT /d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 : #012Traceback (most recent call last):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/usr/lib/pymodules/python2.7/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/usr/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/usr/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/usr/lib/pymodules/python2.7/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: txce9fb25730b04ca9823f4-0055bf796f)
Aug  3 14:24:19 localhost object-server: 192.168.200.21 - - [03/Aug/2015:14:24:19 +0000] "PUT /d0/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 500 1478 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txce9fb25730b04ca9823f4-0055bf796f" "proxy-server 31527" 35.6807 "-" 31514 1

Find 5 related transaction ids (dev23)

[vagrant@dev23 node]$ sudo grep "txf23ae2ee17a14cc9b5e74-0055bf7593" /var/log/swift/all.log
Aug  3 14:07:16 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:16 +0000] "HEAD /d6/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txf23ae2ee17a14cc9b5e74-0055bf7593" "proxy-server 3965" 0.0006 "-" 3961 1
Aug  3 14:07:16 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:16 +0000] "HEAD /d7/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txf23ae2ee17a14cc9b5e74-0055bf7593" "proxy-server 3965" 0.0011 "-" 3958 1
Aug  3 14:07:16 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:16 +0000] "HEAD /d4/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txf23ae2ee17a14cc9b5e74-0055bf7593" "proxy-server 3965" 0.0005 "-" 3956 1
Aug  3 14:07:16 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:16 +0000] "HEAD /d5/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txf23ae2ee17a14cc9b5e74-0055bf7593" "proxy-server 3965" 0.0007 "-" 3956 1
Aug  3 14:07:16 dev23 proxy-server: 192.168.200.32 192.168.200.32 03/Aug/2015/14/07/16 HEAD /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 HTTP/1.0 404 - - demo%2CAUTH_tk9d40... - - - txf23ae2ee17a14cc9b5e74-0055bf7593 - 1.2058 - - 1438610835.603729963 1438610836.809545040 1
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ sudo grep "tx529914b68d864dac9ecb2-0055bf796a" /var/log/swift/all.log
Aug  3 14:07:20 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:20 +0000] "HEAD /d4/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx529914b68d864dac9ecb2-0055bf796a" "proxy-server 31535" 0.0008 "-" 3959 1
Aug  3 14:07:20 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:20 +0000] "HEAD /d6/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx529914b68d864dac9ecb2-0055bf796a" "proxy-server 31535" 0.0021 "-" 3961 1
Aug  3 14:07:20 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:20 +0000] "HEAD /d7/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx529914b68d864dac9ecb2-0055bf796a" "proxy-server 31535" 0.0023 "-" 3956 1
Aug  3 14:07:20 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:20 +0000] "HEAD /d5/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 404 - "HEAD http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx529914b68d864dac9ecb2-0055bf796a" "proxy-server 31535" 0.0004 "-" 3956 1
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ sudo grep "tx9eee5815960f4ac2be26e-0055bf796b" /var/log/swift/all.log
Aug  3 14:07:28 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:28 +0000] "PUT /d4/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 201 - "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx9eee5815960f4ac2be26e-0055bf796b" "proxy-server 31534" 11.6465 "-" 3961 1
Aug  3 14:07:28 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:28 +0000] "PUT /d7/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 201 - "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx9eee5815960f4ac2be26e-0055bf796b" "proxy-server 31534" 11.6495 "-" 3961 1
Aug  3 14:07:28 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:28 +0000] "PUT /d6/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 201 - "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx9eee5815960f4ac2be26e-0055bf796b" "proxy-server 31534" 11.6270 "-" 3961 1
Aug  3 14:07:30 dev23 container-server: 192.168.200.21 - - [03/Aug/2015:14:07:30 +0000] "PUT /d7/39337/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 201 - "PUT http://192.168.200.21/d3/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx9eee5815960f4ac2be26e-0055bf796b" "object-server 31522" 0.0016 "-" 3950 1
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ sudo grep "tx677d22068a53484f8b9db-0055bf75a6" /var/log/swift/all.log
Aug  3 14:07:34 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:34 +0000] "PUT /d4/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 409 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx677d22068a53484f8b9db-0055bf75a6" "proxy-server 3968" 0.0009 "-" 3958 1
Aug  3 14:07:34 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:34 +0000] "PUT /d6/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 409 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx677d22068a53484f8b9db-0055bf75a6" "proxy-server 3968" 0.0011 "-" 3958 1
Aug  3 14:07:34 dev23 object-server: 192.168.200.23 - - [03/Aug/2015:14:07:34 +0000] "PUT /d7/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 409 - "PUT http://192.168.200.23/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "tx677d22068a53484f8b9db-0055bf75a6" "proxy-server 3968" 0.0010 "-" 3958 1
Aug  3 14:07:35 dev23 proxy-server: ERROR Unhandled exception in request: #012Traceback (most recent call last):#012  File "/opt/ss/lib/python2.7/site-packages/swift/proxy/server.py", line 401, in handle_request#012    return handler(req)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/opt/ss/lib/python2.7/site-packages/swift/proxy/controllers/base.py", line 263, in wrapped#012    return func(*a, **kw)#012  File "/opt/ss/lib/python2.7/site-packages/swift/proxy/controllers/base.py", line 99, in wrapped#012    return func(*a, **kw)#012  File "/opt/ss/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line 844, in PUT#012    req, data_source, nodes, partition, outgoing_headers)#012  File "/opt/ss/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line 2546, in _store_object#012    self._check_failure_put_connections(putters, req, nodes, min_conns)#012  File "/opt/ss/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line 666, in _check_failure_put_connections#012    'timestamps': ', '.join(timestamps)})#012TypeError: sequence item 4: expected string, NoneType found (txn: tx677d22068a53484f8b9db-0055bf75a6) (client_ip: 192.168.200.31)
Aug  3 14:07:35 dev23 proxy-server: 192.168.200.31 192.168.200.31 03/Aug/2015/14/07/35 PUT /v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 HTTP/1.0 500 - - demo%2CAUTH_tk9d40... - 125 - tx677d22068a53484f8b9db-0055bf75a6 - 1.6363 - - 1438610854.348484993 1438610855.984777927 1
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ 
[vagrant@dev23 node]$ sudo grep "txce9fb25730b04ca9823f4-0055bf796f" /var/log/swift/all.log
Aug  3 14:07:55 dev23 object-server: ERROR __call__ error with PUT /d6/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 : #012Traceback (most recent call last):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/opt/ss/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/opt/ss/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: txce9fb25730b04ca9823f4-0055bf796f)
Aug  3 14:07:55 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:55 +0000] "PUT /d6/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 500 1539 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txce9fb25730b04ca9823f4-0055bf796f" "proxy-server 31527" 34.4499 "-" 3952 1
Aug  3 14:07:55 dev23 object-server: ERROR __call__ error with PUT /d7/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 : #012Traceback (most recent call last):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/opt/ss/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/opt/ss/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: txce9fb25730b04ca9823f4-0055bf796f)
Aug  3 14:07:55 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:55 +0000] "PUT /d7/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 500 1539 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txce9fb25730b04ca9823f4-0055bf796f" "proxy-server 31527" 34.4724 "-" 3952 1
Aug  3 14:07:55 dev23 object-server: ERROR __call__ error with PUT /d4/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044 : #012Traceback (most recent call last):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 908, in __call__#012    res = method(req)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 2667, in wrapped#012    return func(*a, **kw)#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 1207, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 637, in PUT#012    if not self._read_put_commit_message(mime_documents_iter):#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 377, in _read_put_commit_message#012    commit_hdrs, commit_iter = next(mime_documents_iter)#012  File "/opt/ss/lib/python2.7/site-packages/swift/obj/server.py", line 62, in iter_mime_headers_and_bodies#012    hdrs = HeaderKeyDict(rfc822.Message(file_like, 0))#012  File "/opt/ss/lib/python2.7/rfc822.py", line 108, in __init__#012    self.readheaders()#012  File "/opt/ss/lib/python2.7/rfc822.py", line 155, in readheaders#012    line = self.fp.readline()#012  File "/opt/ss/lib/python2.7/site-packages/swift/common/utils.py", line 3328, in readline#012    chunk = self.wsgi_input.read(self.read_chunk_size)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 188, in read#012    return self._chunked_read(self.rfile, length)#012  File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 178, in _chunked_read#012    self.chunk_length = int(rfile.readline().split(b";", 1)[0], 16)#012ValueError: invalid literal for int() with base 16: '' (txn: txce9fb25730b04ca9823f4-0055bf796f)
Aug  3 14:07:56 dev23 object-server: 192.168.200.21 - - [03/Aug/2015:14:07:56 +0000] "PUT /d4/293/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" 500 1539 "PUT http://192.168.200.21/v1/AUTH_demo/ssbench_1MB-PUT_0803_142258_000043_ec42/1MB-PUT_000044" "txce9fb25730b04ca9823f4-0055bf796f" "proxy-server 31527" 34.6451 "-" 3952 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment