Skip to content

Instantly share code, notes, and snippets.

@portante
Last active December 14, 2015 04:39
Show Gist options
  • Save portante/5029518 to your computer and use it in GitHub Desktop.
Save portante/5029518 to your computer and use it in GitHub Desktop.
strace of proxy-server, object-server, glusterfs with posix translator only.
0.000115 [proxy -server] 1361794636.177603 accept(4, {sa_family=AF_INET, sin_port=htons(56708), sin_addr=inet_addr("172.17.10.19")}, [16]) = 11
0.000058 [proxy -server] 1361794636.177894 fcntl(11, F_GETFL) = 0x2 (flags O_RDWR)
0.000028 [proxy -server] 1361794636.178021 fcntl(11, F_SETFL, O_RDWR|O_NONBLOCK) = 0
0.000172 [proxy -server] 1361794636.178319 recvfrom(11, "PUT /v1/AUTH_del0/gprfc019.pw32."..., 8192, 0, NULL, NULL) = 309
0.000013 [proxy -server] 1361794636.178717 getsockname(11, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("172.17.40.13")}, [16]) = 0
0.000014 [proxy -server] 1361794636.179056 read(12, "|R\242x\335\250\227\36P\352\221/\25\207\276\34", 16) = 16
0.000008 [proxy -server] 1361794636.179097 gettid() = 10464
0.000108 [proxy -server] 1361794636.179306 stat("/etc/swift/container.ring.gz", {st_mode=S_IFREG|0644, st_size=741, ...}) = 0
0.000068 [proxy -server] 1361794636.179553 sendto(13, "get 18f487541b6b4e33f36f1165f5ef"..., 38, 0, NULL, 0) = 38
0.000018 [proxy -server] 1361794636.179713 recvfrom(13, "VALUE 18f487541b6b4e33f36f1165f5"..., 8192, 0, NULL, NULL) = 145
0.000018 [proxy -server] 1361794636.179922 stat("/etc/swift/object.ring.gz", {st_mode=S_IFREG|0644, st_size=738, ...}) = 0
0.000044 [proxy -server] 1361794636.180500 socket(PF_NETLINK, SOCK_RAW, 0) = 14
0.000012 [proxy -server] 1361794636.180576 bind(14, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
0.000011 [proxy -server] 1361794636.180613 getsockname(14, {sa_family=AF_NETLINK, pid=10464, groups=00000000}, [12]) = 0
0.000041 [proxy -server] 1361794636.180650 sendto(14, "\24\0\0\0\26\0\1\3LV+Q\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
0.000039 [proxy -server] 1361794636.180728 recvmsg(14, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0LV+Q\340(\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 164
0.000011 [proxy -server] 1361794636.180815 recvmsg(14, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0LV+Q\340(\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 192
0.000010 [proxy -server] 1361794636.180860 recvmsg(14, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0LV+Q\340(\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
0.000019 [proxy -server] 1361794636.180899 close(14) = 0
0.000015 [proxy -server] 1361794636.180956 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 14
0.000007 [proxy -server] 1361794636.181003 fcntl(14, F_GETFL) = 0x2 (flags O_RDWR)
0.000007 [proxy -server] 1361794636.181027 fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0
0.000084 [proxy -server] 1361794636.181066 connect(14, {sa_family=AF_INET, sin_port=htons(6010), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
37.035748 [object-server] 1361794636.181175 <... poll resumed> ) = 1 ([{fd=8, revents=POLLIN}])
<unfinis> [proxy -server] 1361794636.181254 poll([{fd=14, events=POLLOUT|POLLERR|POLLHUP}], 1, 4999
<unfinis> [object-server] 1361794636.181276 accept(8,
0.000019 [proxy -server] 1361794636.181287 <... poll resumed> ) = 1 ([{fd=14, revents=POLLOUT}])
0.000022 [object-server] 1361794636.181301 <... accept resumed> {sa_family=AF_INET, sin_port=htons(53474), sin_addr=inet_addr("127.0.0.1")}, [16]) = 14
0.000011 [proxy -server] 1361794636.181323 getsockopt(14, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
<unfinis> [object-server] 1361794636.181349 fcntl(14, F_GETFL
<unfinis> [proxy -server] 1361794636.181362 connect(14, {sa_family=AF_INET, sin_port=htons(6010), sin_addr=inet_addr("127.0.0.1")}, 16
0.000016 [object-server] 1361794636.181371 <... fcntl resumed> ) = 0x2 (flags O_RDWR)
0.000011 [proxy -server] 1361794636.181379 <... connect resumed> ) = 0
0.000010 [object-server] 1361794636.181388 fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0
0.000042 [proxy -server] 1361794636.181408 sendto(14, "PUT /sdb1/65828/AUTH_del0/gprfc0"..., 562, 0, NULL, 0) = 562
<unfinis> [proxy -server] 1361794636.181605 recvfrom(14,
<unfinis> [object-server] 1361794636.181620 recvfrom(14,
0.000018 [proxy -server] 1361794636.181630 <... recvfrom resumed> 0x1112774, 1, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
0.000021 [object-server] 1361794636.181644 <... recvfrom resumed> "PUT /sdb1/65828/AUTH_del0/gprfc0"..., 8192, 0, NULL, NULL) = 562
<unfinis> [proxy -server] 1361794636.181729 poll([{fd=14, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 4998
0.000010 [object-server] 1361794636.181930 getsockname(14, {sa_family=AF_INET, sin_port=htons(6010), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
<unfinis> [object-server] 1361794636.182381 lstat("/mnt/gluster-object/AUTH_del0",
37.038598 [glusterfs-000] 1361794636.182538 <... readv resumed> [{"8\0\0\0\3\0\0\0+\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 80}, {"b26c4c8c3323b5f1a\0gluster_func.p"..., 131072}], 2) = 56
0.000084 [glusterfs-000] 1361794636.182715 open("/proc/10450/status", O_RDONLY) = 10
0.000033 [glusterfs-000] 1361794636.182888 fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
0.000050 [glusterfs-000] 1361794636.183024 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f718d386000
0.000045 [glusterfs-000] 1361794636.183108 read(10, "Name:\tswift-object-se\nState:\tS ("..., 1024) = 902
0.000016 [glusterfs-000] 1361794636.183181 close(10) = 0
0.000012 [glusterfs-000] 1361794636.183218 munmap(0x7f718d386000, 4096) = 0
0.000016 [glusterfs-000] 1361794636.183257 lstat("/brick/swift/posix-only/", {st_mode=S_IFDIR|0755, st_size=299, ...}) = 0
0.000012 [glusterfs-000] 1361794636.183311 writev(7, [{"x\0\0\0\0\0\0\0+\0\0\0\0\0\0\0", 16}, {"\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0+\1\0\0\0\0\0\0"..., 104}], 2) = 120
0.000959 [object-server] 1361794636.183348 <... lstat resumed> {st_mode=S_IFDIR|0755, st_size=299, ...}) = 0
<unfinis> [glusterfs-000] 1361794636.183369 readv(7,
0.000011 [object-server] 1361794636.183401 lstat("/mnt/gluster-object/AUTH_del0/..", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
<unfinis> [object-server] 1361794636.183471 stat("/mnt/gluster-object/AUTH_del0/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py",
0.000136 [glusterfs-000] 1361794636.183514 <... readv resumed> [{"f\0\0\0\1\0\0\0,\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 80}, {"r32-posix-only.clth30\0ter_func.p"..., 131072}], 2) = 102
0.000014 [glusterfs-000] 1361794636.183571 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
0.000015 [glusterfs-000] 1361794636.183651 lgetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", "trusted.gfid", "\xf6\x04"\xa14\xadF\xcd\x945\x9e,\x02\xb5\xe6\x02", 16) = 16
0.000008 [glusterfs-000] 1361794636.183693 lstat("/brick/swift/posix-only", {st_mode=S_IFDIR|0755, st_size=299, ...}) = 0
0.000010 [glusterfs-000] 1361794636.183732 writev(7, [{"\220\0\0\0\0\0\0\0,\0\0\0\0\0\0\0", 16}, {"T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 128}], 2) = 144
0.000008 [glusterfs-000] 1361794636.183773 readv(7, [{":\0\0\0\1\0\0\0-\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"r32-posix-only.clth30\0ter_func.p"..., 131072}], 2) = 58
0.000010 [glusterfs-000] 1361794636.183815 read(8, "\354L\332\272F \373Q\324\tG} ;\271\5", 16) = 16
0.000013 [glusterfs-000] 1361794636.183846 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe8f0) = -1 ENOENT (No such file or directory)
0.000008 [glusterfs-000] 1361794636.183884 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe940) = -1 ENOENT (No such file or directory)
0.000007 [glusterfs-000] 1361794636.183916 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe7f0) = -1 ENOENT (No such file or directory)
0.000008 [glusterfs-000] 1361794636.183946 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
0.000009 [glusterfs-000] 1361794636.183985 writev(7, [{"\20\0\0\0\376\377\377\377-\0\0\0\0\0\0\0", 16}], 1) = 16
0.000528 [object-server] 1361794636.184011 <... stat resumed> 0x7fff4d30b3d0) = -1 ENOENT (No such file or directory)
<unfinis> [glusterfs-000] 1361794636.184022 readv(7,
<unfinis> [object-server] 1361794636.184140 open("/mnt/gluster-object/AUTH_del0/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0777
0.000161 [glusterfs-000] 1361794636.184191 <... readv resumed> [{"\\\0\0\0\1\0\0\0.\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"8e5ba6fbd6e\0ly.clth30\0ter_func.p"..., 131072}], 2) = 92
0.000011 [glusterfs-000] 1361794636.184239 read(8, "g\\k~\323@U}\210\306~\2003<\270_", 16) = 16
#+
# posix_create()
# MAKE_ENTRY_HANDLE() - posix-handle.h
# posix_istat(), line 112 of MAKE_ENTRY_HANDLE()
0.000048 [glusterfs-000] 1361794636.184280 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0x7f7183ffe8b0) = -1 ENOENT (No such file or directory)
# MAKE_HANDLE_PATH()
0.000010 [glusterfs-000] 1361794636.184357 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0x7f7183ffe900) = -1 ENOENT (No such file or directory)
0.000009 [glusterfs-000] 1361794636.184394 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0x7f7183ffe770) = -1 ENOENT (No such file or directory)
# MAKE_HANDLE_PATH()
0.000008 [glusterfs-000] 1361794636.184429 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
# ... not sure of next three ...
0.000009 [glusterfs-000] 1361794636.184469 writev(7, [{"\20\0\0\0\376\377\377\377.\0\0\0\0\0\0\0", 16}], 1) = 16
0.000034 [glusterfs-000] 1361794636.184503 readv(7, [{"l\0\0\0#\0\0\0/\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"d37c4f0d239a2c468e5ba6fbd6e\0nc.p"..., 131072}], 2) = 108
0.000009 [glusterfs-000] 1361794636.184572 read(8, "6\355\365\212lD\263b!\264h\335Fx\353f", 16) = 16
# ... not sure, but probably part of MAKE_ENTRY_HANDLE() somehow
0.000009 [glusterfs-000] 1361794636.184606 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0x7f7183ffe820) = -1 ENOENT (No such file or directory)
# Line 1691 of posix_create() in posix.c - posix_pstat()
0.000007 [glusterfs-000] 1361794636.184641 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
# Line 1711 of posix_create() in posix.c - posix_pstat()
0.000008 [glusterfs-000] 1361794636.184679 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0x7f7183ffe820) = -1 ENOENT (No such file or directory)
# Line 1719 of posix_create() in posix.c
0.000117 [glusterfs-000] 1361794636.184713 open("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE|O_CLOEXEC, 0100755) = 10
#+
# posix_gfid_set() - called on Line 1733 of posix_create() in posix.c
# Line 488 of posix_gfid_set() in posix-helpers.c
0.000009 [glusterfs-000] 1361794636.184856 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
# Line 491 of posix_gfid_set() in posix-helpers.c
0.000009 [glusterfs-000] 1361794636.184901 lgetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "trusted.gfid", 0x7f7183ffe900, 16) = -1 ENODATA (No data available)
# Line 505 of posix_gfid_set() in posix-helpers.c
0.000013 [glusterfs-000] 1361794636.184940 lsetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16, XATTR_CREATE) = 0
#+
# posix_handle_hard()
# Line 214 of posix_handle_path() in posix-handle.c from macro MAKE_HANDLE_PATH, first invocation with length 0
0.046522 [glusterfs-000] 1361794636.184989 lstat("/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6", 0x7f7183ffe680) = -1 ENOENT (No such file or directory)
# Line 214 of posix_handle_path() in posix-handle.c from macro MAKE_HANDLE_PATH, first invocation with length previously generated
0.000061 [glusterfs-000] 1361794636.231594 lstat("/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6", 0x7f7183ffe620) = -1 ENOENT (No such file or directory)
# Line 441 of posix_handle_hard() in posix-handle.c
0.000099 [glusterfs-000] 1361794636.231722 lstat("/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6", 0x7f7183ffe790) = -1 ENOENT (No such file or directory) {{Line 214 of posix_handle_path in posix-handle.c from macro MAKE_HANDLE_PATH}}
# Line 449 of posix_handle_hard() in posix-handle.c, call to posix_handle_mkdir_hashes()
0.000041 [glusterfs-000] 1361794636.231937 mkdir("/brick/swift/posix-only/.glusterfs/1c", 0700) = -1 EEXIST (File exists)
0.000016 [glusterfs-000] 1361794636.232021 mkdir("/brick/swift/posix-only/.glusterfs/1c/0b", 0700) = -1 EEXIST (File exists)
# Line 457 of posix_handle_hard() in posix-handle.c
0.013147 [glusterfs-000] 1361794636.232072 link("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6") = 0
# Line 465 of posix_handle_hard() in posix-handle.c
0.000037 [glusterfs-000] 1361794636.245286 lstat("/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
# Line 474 of posix_handle_hard() in posix-handle.c
0.000063 [glusterfs-000] 1361794636.245434 lstat("/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
# posix_handle_hard()
#-
# posix_gfid_set()
#-
# Line 1740 of posix_create() in posix.c
0.000072 [glusterfs-000] 1361794636.245608 chown("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0, 0) = 0
# Line 1749 of posix_create() in posix.c, posix_acl_xattr_set()
0.000063 [glusterfs-000] 1361794636.245753 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
# Line 1764 of posix_create() in posix.c, posix_fdstat()
0.000011 [glusterfs-000] 1361794636.245888 fstat(10, {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000009 [glusterfs-000] 1361794636.245940 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
# Line 1772 of posix_create() in posix.c, posix_pstat()
0.000009 [glusterfs-000] 1361794636.245969 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=106, ...}) = 0
# posix_create()
#-
0.000011 [glusterfs-000] 1361794636.246014 writev(7, [{"\240\0\0\0\0\0\0\0/\0\0\0\0\0\0\0", 16}, {"|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 128}, {"\34\34\27\2\0\0\0\0\0\0\0\0\0\0\0\0", 16}], 3) = 160
<unfinis> [glusterfs-000] 1361794636.246062 readv(7,
0.061993 [object-server] 1361794636.246168 <... open resumed> ) = 15
<unfinis> [object-server] 1361794636.246331 fchown(15, 4294967295, 4294967295
0.000416 [glusterfs-000] 1361794636.246489 <... readv resumed> [{"D\0\0\0\26\0\0\0000\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"d37c4f0d239a2c468e5ba6fbd6e\0nc.p"..., 131072}], 2) = 68
0.000058 [glusterfs-000] 1361794636.246570 writev(7, [{"\20\0\0\0\303\377\377\3770\0\0\0\0\0\0\0", 16}], 1) = 16
0.000050 [glusterfs-000] 1361794636.246683 readv(7, [{"\200\0\0\0\4\0\0\0001\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072}], 2) = 128
0.000010 [glusterfs-000] 1361794636.246801 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000013 [glusterfs-000] 1361794636.246847 lchown("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 4294967295, 4294967295) = 0
0.000008 [glusterfs-000] 1361794636.246885 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000010 [glusterfs-000] 1361794636.246930 writev(7, [{"x\0\0\0\0\0\0\0001\0\0\0\0\0\0\0", 16}, {"\1\0\0\0\0\0\0\0\0\0\0\0ShIg\246\222\25k\347\373\322\237\0\0\0\0\0\0\0\0"..., 104}], 2) = 120
<unfinis> [glusterfs-000] 1361794636.246969 readv(7,
0.000635 [object-server] 1361794636.246981 <... fchown resumed> ) = 0
<unfinis> [object-server] 1361794636.247132 sendto(14, "HTTP/1.1 100 Continue\r\n\r\n", 25, 0, NULL, 0
0.065513 [proxy -server] 1361794636.247259 <... poll resumed> ) = 1 ([{fd=14, revents=POLLIN}])
0.000128 [object-server] 1361794636.247278 <... sendto resumed> ) = 25
<unfinis> [object-server] 1361794636.247393 recvfrom(14,
<unfinis> [proxy -server] 1361794636.247421 recvfrom(14,
0.000030 [object-server] 1361794636.247438 <... recvfrom resumed> 0x14ea4b4, 7040, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
0.000028 [proxy -server] 1361794636.247456 <... recvfrom resumed> "H", 1, 0, NULL, NULL) = 1
<unfinis> [proxy -server] 1361794636.247604 recvfrom(14,
<unfinis> [object-server] 1361794636.247621 poll([{fd=14, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 60000
0.000097 [proxy -server] 1361794636.247716 <... recvfrom resumed> "T", 1, 0, NULL, NULL) = 1
0.000095 [proxy -server] 1361794636.247870 recvfrom(14, "T", 1, 0, NULL, NULL) = 1
0.000040 [proxy -server] 1361794636.248093 recvfrom(14, "P", 1, 0, NULL, NULL) = 1
0.000031 [proxy -server] 1361794636.248190 recvfrom(14, "/", 1, 0, NULL, NULL) = 1
0.000029 [proxy -server] 1361794636.248255 recvfrom(14, "1", 1, 0, NULL, NULL) = 1
0.000030 [proxy -server] 1361794636.248332 recvfrom(14, ".", 1, 0, NULL, NULL) = 1
0.000011 [proxy -server] 1361794636.248412 recvfrom(14, "1", 1, 0, NULL, NULL) = 1
0.000009 [proxy -server] 1361794636.248447 recvfrom(14, " ", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248483 recvfrom(14, "1", 1, 0, NULL, NULL) = 1
0.000008 [proxy -server] 1361794636.248510 recvfrom(14, "0", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248537 recvfrom(14, "0", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248564 recvfrom(14, " ", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248590 recvfrom(14, "C", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248616 recvfrom(14, "o", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248643 recvfrom(14, "n", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248669 recvfrom(14, "t", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248700 recvfrom(14, "i", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248727 recvfrom(14, "n", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248753 recvfrom(14, "u", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248780 recvfrom(14, "e", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248806 recvfrom(14, "\r", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248833 recvfrom(14, "\n", 1, 0, NULL, NULL) = 1
0.000008 [proxy -server] 1361794636.248895 recvfrom(14, "\r", 1, 0, NULL, NULL) = 1
0.000008 [proxy -server] 1361794636.248924 recvfrom(14, "\n", 1, 0, NULL, NULL) = 1
0.000023 [proxy -server] 1361794636.249160 sendto(11, "HTTP/1.1 100 Continue\r\n\r\n", 25, 0, NULL, 0) = 25
0.000010 [proxy -server] 1361794636.249223 recvfrom(11, 0x1184384, 7040, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
0.000009 [proxy -server] 1361794636.249311 poll([{fd=11, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
0.000010 [proxy -server] 1361794636.249397 poll([{fd=11, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 4930) = 1 ([{fd=11, revents=POLLIN}])
0.000011 [proxy -server] 1361794636.249445 recvfrom(11, "#!/usr/bin/env python26\n#\n# func"..., 7040, 0, NULL, NULL) = 7040
0.000055 [proxy -server] 1361794636.249591 sendto(14, "#!/usr/bin/env python26\n#\n# func"..., 7040, 0, NULL, 0) = 7040
0.002032 [object-server] 1361794636.249666 <... poll resumed> ) = 1 ([{fd=14, revents=POLLIN}])
0.000013 [object-server] 1361794636.249749 recvfrom(14, "#!/usr/bin/env python26\n#\n# func"..., 7040, 0, NULL, NULL) = 7040
<unfinis> [object-server] 1361794636.249834 write(15, "#!/usr/bin/env python26\n#\n# func"..., 7040
0.000014 [proxy -server] 1361794636.249853 recvfrom(14, 0x11885e4, 8192, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
0.002921 [glusterfs-000] 1361794636.249896 <... readv resumed> [{"D\0\0\0\26\0\0\0002\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072}], 2) = 68
<unfinis> [proxy -server] 1361794636.249938 poll([{fd=14, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 4930
0.000016 [glusterfs-000] 1361794636.249954 writev(7, [{"\20\0\0\0\303\377\377\3772\0\0\0\0\0\0\0", 16}], 1) = 16
0.000012 [glusterfs-000] 1361794636.249998 readv(7, [{"\320\33\0\0\20\0\0\0003\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"#!/usr/bin/env python26\n#\n# func"..., 131072}], 2) = 7120
0.000007 [glusterfs-000] 1361794636.250046 fstat(10, {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000008 [glusterfs-000] 1361794636.250078 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000029 [glusterfs-000] 1361794636.250106 pwrite(10, "#!/usr/bin/env python26\n#\n# func"..., 7040, 0) = 7040
0.000006 [glusterfs-000] 1361794636.250153 fstat(10, {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000008 [glusterfs-000] 1361794636.250185 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000012 [glusterfs-000] 1361794636.250213 writev(7, [{"\30\0\0\0\0\0\0\0003\0\0\0\0\0\0\0", 16}, {"\200\33\0\0\0\0\0\0", 8}], 2) = 24
0.000400 [object-server] 1361794636.250244 <... write resumed> ) = 7040
<unfinis> [glusterfs-000] 1361794636.250253 readv(7,
0.000016 [object-server] 1361794636.250473 fadvise64(15, 0, 7040, POSIX_FADV_DONTNEED) = 0
<unfinis> [object-server] 1361794636.250517 fsync(15
0.000290 [glusterfs-000] 1361794636.250551 <... readv resumed> [{"8\0\0\0\24\0\0\0004\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"#!/usr/bin/env python26\n#\n# func"..., 131072}], 2) = 56
0.000010 [glusterfs-000] 1361794636.250596 fstat(10, {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000007 [glusterfs-000] 1361794636.250642 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000469 [glusterfs-000] 1361794636.250668 fsync(10) = 0
0.000009 [glusterfs-000] 1361794636.251167 fstat(10, {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000016 [glusterfs-000] 1361794636.251205 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
<unfinis> [glusterfs-000] 1361794636.251273 writev(7, [{"\20\0\0\0\0\0\0\0004\0\0\0\0\0\0\0", 16}], 1
0.000830 [object-server] 1361794636.251358 <... fsync resumed> ) = 0
0.000078 [glusterfs-000] 1361794636.251370 <... writev resumed> ) = 16
<unfinis> [object-server] 1361794636.251459 setxattr("/mnt/gluster-object/AUTH_del0/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "user.swift.metadata", "\x80\x02}q\x01(U\x0eContent-Lengthq\x02U\x047040q\x03U\x04ETagq\x04U 69722aba2bee114799fb6c64758da529q\x05U\x0bX-Timestampq\x06U\x101361794636.17999q\x07U\x0dX-Object-Typeq\x08U\x04fileq\x09U\x06X-Typeq\x0aU\x06Objectq\x0bU\x0cContent-Typeq\x0cU\x0dtext/x-pythonq\x0du.", 191, 0
0.000050 [glusterfs-000] 1361794636.251503 readv(7, [{"8\0\0\0\3\0\0\0005\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"#!/usr/bin/env python26\n#\n# func"..., 131072}], 2) = 56
0.000012 [glusterfs-000] 1361794636.251615 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=106, ...}) = 0
0.000010 [glusterfs-000] 1361794636.251662 writev(7, [{"x\0\0\0\0\0\0\0005\0\0\0\0\0\0\0", 16}, {"\1\0\0\0\0\0\0\0\0\0\0\0004\255F\315\2\346\265\2,\2365\224j\0\0\0\0\0\0\0"..., 104}], 2) = 120
0.000009 [glusterfs-000] 1361794636.251703 readv(7, [{"8\0\0\0\3\0\0\0006\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"#!/usr/bin/env python26\n#\n# func"..., 131072}], 2) = 56
0.000007 [glusterfs-000] 1361794636.251743 fstat(10, {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000008 [glusterfs-000] 1361794636.251774 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000010 [glusterfs-000] 1361794636.251803 writev(7, [{"x\0\0\0\0\0\0\0006\0\0\0\0\0\0\0", 16}, {"\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\246\222\25k\347\373\322\237\200\33\0\0\0\0\0\0"..., 104}], 2) = 120
0.000008 [glusterfs-000] 1361794636.251842 readv(7, [{"\3\1\0\0\25\0\0\0007\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"ent-Lengthq\2U\0047040q\3U\4ETagq\4U 69"..., 131072}], 2) = 259
0.000010 [glusterfs-000] 1361794636.251887 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000016 [glusterfs-000] 1361794636.251933 lsetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "user.swift.metadata", "\x80\x02}q\x01(U\x0eContent-Lengthq\x02U\x047040q\x03U\x04ETagq\x04U 69722aba2bee114799fb6c64758da529q\x05U\x0bX-Timestampq\x06U\x101361794636.17999q\x07U\x0dX-Object-Typeq\x08U\x04fileq\x09U\x06X-Typeq\x0aU\x06Objectq\x0bU\x0cContent-Typeq\x0cU\x0dtext/x-pythonq\x0du.", 191, 0) = 0
0.000009 [glusterfs-000] 1361794636.251993 writev(7, [{"\20\0\0\0\0\0\0\0007\0\0\0\0\0\0\0", 16}], 1) = 16
0.000522 [object-server] 1361794636.252019 <... setxattr resumed> ) = 0
<unfinis> [glusterfs-000] 1361794636.252028 readv(7,
<unfinis> [object-server] 1361794636.252097 rename("/mnt/gluster-object/AUTH_del0/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "/mnt/gluster-object/AUTH_del0/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py"
0.000129 [glusterfs-000] 1361794636.252165 <... readv resumed> [{":\0\0\0\1\0\0\0008\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"ent-Lengthq\2U\0047040q\3U\4ETagq\4U 69"..., 131072}], 2) = 58
0.000010 [glusterfs-000] 1361794636.252239 read(8, "q\20\267=\354n-\30\0\1\3633.\355\321\361", 16) = 16
0.000009 [glusterfs-000] 1361794636.252271 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe8f0) = -1 ENOENT (No such file or directory)
0.000008 [glusterfs-000] 1361794636.252305 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe940) = -1 ENOENT (No such file or directory)
0.000007 [glusterfs-000] 1361794636.252336 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe7f0) = -1 ENOENT (No such file or directory)
0.000007 [glusterfs-000] 1361794636.252367 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=106, ...}) = 0
0.000009 [glusterfs-000] 1361794636.252408 writev(7, [{"\20\0\0\0\376\377\377\3778\0\0\0\0\0\0\0", 16}], 1) = 16
0.000008 [glusterfs-000] 1361794636.252440 readv(7, [{"v\0\0\0\f\0\0\0009\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"239a2c468e5ba6fbd6e\0gluster_func"..., 131072}], 2) = 118
0.000008 [glusterfs-000] 1361794636.252481 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000009 [glusterfs-000] 1361794636.252525 lgetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000008 [glusterfs-000] 1361794636.252564 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe4b0) = -1 ENOENT (No such file or directory)
0.000008 [glusterfs-000] 1361794636.252595 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=106, ...}) = 0
0.000007 [glusterfs-000] 1361794636.252633 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=106, ...}) = 0
0.000007 [glusterfs-000] 1361794636.252671 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe4b0) = -1 ENOENT (No such file or directory)
0.000022 [glusterfs-000] 1361794636.252701 rename("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py") = 0
0.000008 [glusterfs-000] 1361794636.252755 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000009 [glusterfs-000] 1361794636.252795 lgetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000008 [glusterfs-000] 1361794636.252829 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=72, ...}) = 0
0.000008 [glusterfs-000] 1361794636.252867 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=72, ...}) = 0
0.000009 [glusterfs-000] 1361794636.252909 writev(7, [{"\20\0\0\0\0\0\0\0009\0\0\0\0\0\0\0", 16}], 1) = 16
0.000804 [object-server] 1361794636.252935 <... rename resumed> ) = 0
<unfinis> [glusterfs-000] 1361794636.252944 readv(7,
<unfinis> [object-server] 1361794636.253015 close(15
0.000097 [glusterfs-000] 1361794636.253049 <... readv resumed> [{"@\0\0\0\31\0\0\0:\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"239a2c468e5ba6fbd6e\0gluster_func"..., 131072}], 2) = 64
0.000009 [glusterfs-000] 1361794636.253097 writev(7, [{"\20\0\0\0\0\0\0\0:\0\0\0\0\0\0\0", 16}], 1) = 16
0.000102 [object-server] 1361794636.253123 <... close resumed> ) = 0
0.000008 [glusterfs-000] 1361794636.253132 readv(7, [{"@\0\0\0\22\0\0\0;\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"239a2c468e5ba6fbd6e\0gluster_func"..., 131072}], 2) = 64
0.000009 [glusterfs-000] 1361794636.253169 futex(0x216977c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x2169778, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
0.000009 [glusterfs-000] 1361794636.253200 writev(7, [{"\20\0\0\0\0\0\0\0;\0\0\0\0\0\0\0", 16}], 1) = 16
<unfinis> [glusterfs-000] 1361794636.253230 readv(7,
37.108957 [glusterfs-010] 1361794636.253308 <... futex resumed> ) = 0
<unfinis> [glusterfs-010] 1361794636.253413 futex(0x21697a8, FUTEX_WAKE_PRIVATE, 1
<unfinis> [object-server] 1361794636.253512 sendto(14, "HTTP/1.1 201 Created\r\nEtag: \"697"..., 301, 0, NULL, 0
0.000106 [glusterfs-010] 1361794636.253528 <... futex resumed> ) = 0
0.003600 [proxy -server] 1361794636.253547 <... poll resumed> ) = 1 ([{fd=14, revents=POLLIN}])
0.000037 [object-server] 1361794636.253560 <... sendto resumed> ) = 301
<unfinis> [proxy -server] 1361794636.253599 recvfrom(14,
<unfinis> [glusterfs-010] 1361794636.253610 close(10
0.000013 [proxy -server] 1361794636.253620 <... recvfrom resumed> "HTTP/1.1 201 Created\r\nEtag: \"697"..., 8192, 0, NULL, NULL) = 301
0.000020 [object-server] 1361794636.253655 shutdown(14, 2 /* send and receive */) = 0
0.000015 [object-server] 1361794636.253713 close(14) = 0
0.000132 [glusterfs-010] 1361794636.253745 <... close resumed> ) = 0
<unfinis> [glusterfs-010] 1361794636.253768 futex(0x216977c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 11, {1361795236, 0}, ffffffff
0.000013 [object-server] 1361794636.253815 accept(8, 0x7fff4d30ee50, [16]) = -1 EAGAIN (Resource temporarily unavailable)
0.000042 [proxy -server] 1361794636.253843 close(14) = 0
<unfinis> [object-server] 1361794636.253895 poll([{fd=8, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 60000
0.000015 [proxy -server] 1361794636.254176 sendto(11, "HTTP/1.1 201 Created\r\nContent-Le"..., 326, 0, NULL, 0) = 326
0.000058 [proxy -server] 1361794636.254697 sendto(10, "<142>proxy-server 172.17.10.19 1"..., 721, 0, NULL, 0) = 721
0.000009 [proxy -server] 1361794636.254859 recvfrom(11, "", 8192, 0, NULL, NULL) = 0
0.000012 [proxy -server] 1361794636.254910 shutdown(11, 2 /* send and receive */) = 0
0.000010 [proxy -server] 1361794636.254949 close(11) = 0
0.000012 [proxy -server] 1361794636.255026 accept(4, 0x7fff259ef030, [16]) = -1 EAGAIN (Resource temporarily unavailable)
0.000115 [proxy -server] 1361794636.177603 accept(4, {sa_family=AF_INET, sin_port=htons(56708), sin_addr=inet_addr("172.17.10.19")}, [16]) = 11
0.000058 [proxy -server] 1361794636.177894 fcntl(11, F_GETFL) = 0x2 (flags O_RDWR)
0.000028 [proxy -server] 1361794636.178021 fcntl(11, F_SETFL, O_RDWR|O_NONBLOCK) = 0
0.000172 [proxy -server] 1361794636.178319 recvfrom(11, "PUT /v1/AUTH_del0/gprfc019.pw32."..., 8192, 0, NULL, NULL) = 309
0.000013 [proxy -server] 1361794636.178717 getsockname(11, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("172.17.40.13")}, [16]) = 0
0.000014 [proxy -server] 1361794636.179056 read(12, "|R\242x\335\250\227\36P\352\221/\25\207\276\34", 16) = 16
0.000008 [proxy -server] 1361794636.179097 gettid() = 10464
0.000108 [proxy -server] 1361794636.179306 stat("/etc/swift/container.ring.gz", {st_mode=S_IFREG|0644, st_size=741, ...}) = 0
0.000068 [proxy -server] 1361794636.179553 sendto(13, "get 18f487541b6b4e33f36f1165f5ef"..., 38, 0, NULL, 0) = 38
0.000018 [proxy -server] 1361794636.179713 recvfrom(13, "VALUE 18f487541b6b4e33f36f1165f5"..., 8192, 0, NULL, NULL) = 145
0.000018 [proxy -server] 1361794636.179922 stat("/etc/swift/object.ring.gz", {st_mode=S_IFREG|0644, st_size=738, ...}) = 0
0.000044 [proxy -server] 1361794636.180500 socket(PF_NETLINK, SOCK_RAW, 0) = 14
0.000012 [proxy -server] 1361794636.180576 bind(14, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
0.000011 [proxy -server] 1361794636.180613 getsockname(14, {sa_family=AF_NETLINK, pid=10464, groups=00000000}, [12]) = 0
0.000041 [proxy -server] 1361794636.180650 sendto(14, "\24\0\0\0\26\0\1\3LV+Q\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
0.000039 [proxy -server] 1361794636.180728 recvmsg(14, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0LV+Q\340(\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 164
0.000011 [proxy -server] 1361794636.180815 recvmsg(14, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0LV+Q\340(\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 192
0.000010 [proxy -server] 1361794636.180860 recvmsg(14, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0LV+Q\340(\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
0.000019 [proxy -server] 1361794636.180899 close(14) = 0
0.000015 [proxy -server] 1361794636.180956 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 14
0.000007 [proxy -server] 1361794636.181003 fcntl(14, F_GETFL) = 0x2 (flags O_RDWR)
0.000007 [proxy -server] 1361794636.181027 fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0
0.000084 [proxy -server] 1361794636.181066 connect(14, {sa_family=AF_INET, sin_port=htons(6010), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
37.035748 [object-server] 1361794636.181175 <... poll resumed> ) = 1 ([{fd=8, revents=POLLIN}])
<unfinis> [proxy -server] 1361794636.181254 poll([{fd=14, events=POLLOUT|POLLERR|POLLHUP}], 1, 4999
<unfinis> [object-server] 1361794636.181276 accept(8,
0.000019 [proxy -server] 1361794636.181287 <... poll resumed> ) = 1 ([{fd=14, revents=POLLOUT}])
0.000022 [object-server] 1361794636.181301 <... accept resumed> {sa_family=AF_INET, sin_port=htons(53474), sin_addr=inet_addr("127.0.0.1")}, [16]) = 14
0.000011 [proxy -server] 1361794636.181323 getsockopt(14, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
<unfinis> [object-server] 1361794636.181349 fcntl(14, F_GETFL
<unfinis> [proxy -server] 1361794636.181362 connect(14, {sa_family=AF_INET, sin_port=htons(6010), sin_addr=inet_addr("127.0.0.1")}, 16
0.000016 [object-server] 1361794636.181371 <... fcntl resumed> ) = 0x2 (flags O_RDWR)
0.000011 [proxy -server] 1361794636.181379 <... connect resumed> ) = 0
0.000010 [object-server] 1361794636.181388 fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0
0.000042 [proxy -server] 1361794636.181408 sendto(14, "PUT /sdb1/65828/AUTH_del0/gprfc0"..., 562, 0, NULL, 0) = 562
<unfinis> [proxy -server] 1361794636.181605 recvfrom(14,
<unfinis> [object-server] 1361794636.181620 recvfrom(14,
0.000018 [proxy -server] 1361794636.181630 <... recvfrom resumed> 0x1112774, 1, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
0.000021 [object-server] 1361794636.181644 <... recvfrom resumed> "PUT /sdb1/65828/AUTH_del0/gprfc0"..., 8192, 0, NULL, NULL) = 562
<unfinis> [proxy -server] 1361794636.181729 poll([{fd=14, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 4998
0.000010 [object-server] 1361794636.181930 getsockname(14, {sa_family=AF_INET, sin_port=htons(6010), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
<unfinis> [object-server] 1361794636.182381 lstat("/mnt/gluster-object/AUTH_del0",
37.038598 [glusterfs-000] 1361794636.182538 <... readv resumed> [{"8\0\0\0\3\0\0\0+\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 80}, {"b26c4c8c3323b5f1a\0gluster_func.p"..., 131072}], 2) = 56
0.000084 [glusterfs-000] 1361794636.182715 open("/proc/10450/status", O_RDONLY) = 10
0.000033 [glusterfs-000] 1361794636.182888 fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
0.000050 [glusterfs-000] 1361794636.183024 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f718d386000
0.000045 [glusterfs-000] 1361794636.183108 read(10, "Name:\tswift-object-se\nState:\tS ("..., 1024) = 902
0.000016 [glusterfs-000] 1361794636.183181 close(10) = 0
0.000012 [glusterfs-000] 1361794636.183218 munmap(0x7f718d386000, 4096) = 0
0.000016 [glusterfs-000] 1361794636.183257 lstat("/brick/swift/posix-only/", {st_mode=S_IFDIR|0755, st_size=299, ...}) = 0
0.000012 [glusterfs-000] 1361794636.183311 writev(7, [{"x\0\0\0\0\0\0\0+\0\0\0\0\0\0\0", 16}, {"\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0+\1\0\0\0\0\0\0"..., 104}], 2) = 120
0.000959 [object-server] 1361794636.183348 <... lstat resumed> {st_mode=S_IFDIR|0755, st_size=299, ...}) = 0
<unfinis> [glusterfs-000] 1361794636.183369 readv(7,
0.000011 [object-server] 1361794636.183401 lstat("/mnt/gluster-object/AUTH_del0/..", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
<unfinis> [object-server] 1361794636.183471 stat("/mnt/gluster-object/AUTH_del0/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py",
0.000136 [glusterfs-000] 1361794636.183514 <... readv resumed> [{"f\0\0\0\1\0\0\0,\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 80}, {"r32-posix-only.clth30\0ter_func.p"..., 131072}], 2) = 102
0.000014 [glusterfs-000] 1361794636.183571 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
0.000015 [glusterfs-000] 1361794636.183651 lgetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", "trusted.gfid", "\xf6\x04"\xa14\xadF\xcd\x945\x9e,\x02\xb5\xe6\x02", 16) = 16
0.000008 [glusterfs-000] 1361794636.183693 lstat("/brick/swift/posix-only", {st_mode=S_IFDIR|0755, st_size=299, ...}) = 0
0.000010 [glusterfs-000] 1361794636.183732 writev(7, [{"\220\0\0\0\0\0\0\0,\0\0\0\0\0\0\0", 16}, {"T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 128}], 2) = 144
0.000008 [glusterfs-000] 1361794636.183773 readv(7, [{":\0\0\0\1\0\0\0-\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"r32-posix-only.clth30\0ter_func.p"..., 131072}], 2) = 58
0.000010 [glusterfs-000] 1361794636.183815 read(8, "\354L\332\272F \373Q\324\tG} ;\271\5", 16) = 16
0.000013 [glusterfs-000] 1361794636.183846 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe8f0) = -1 ENOENT (No such file or directory)
0.000008 [glusterfs-000] 1361794636.183884 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe940) = -1 ENOENT (No such file or directory)
0.000007 [glusterfs-000] 1361794636.183916 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe7f0) = -1 ENOENT (No such file or directory)
0.000008 [glusterfs-000] 1361794636.183946 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
0.000009 [glusterfs-000] 1361794636.183985 writev(7, [{"\20\0\0\0\376\377\377\377-\0\0\0\0\0\0\0", 16}], 1) = 16
0.000528 [object-server] 1361794636.184011 <... stat resumed> 0x7fff4d30b3d0) = -1 ENOENT (No such file or directory)
<unfinis> [glusterfs-000] 1361794636.184022 readv(7,
<unfinis> [object-server] 1361794636.184140 open("/mnt/gluster-object/AUTH_del0/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0777
0.000161 [glusterfs-000] 1361794636.184191 <... readv resumed> [{"\\\0\0\0\1\0\0\0.\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"8e5ba6fbd6e\0ly.clth30\0ter_func.p"..., 131072}], 2) = 92
0.000011 [glusterfs-000] 1361794636.184239 read(8, "g\\k~\323@U}\210\306~\2003<\270_", 16) = 16
0.000048 [glusterfs-000] 1361794636.184280 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0x7f7183ffe8b0) = -1 ENOENT (No such file or directory)
0.000010 [glusterfs-000] 1361794636.184357 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0x7f7183ffe900) = -1 ENOENT (No such file or directory)
0.000009 [glusterfs-000] 1361794636.184394 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0x7f7183ffe770) = -1 ENOENT (No such file or directory)
0.000008 [glusterfs-000] 1361794636.184429 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
0.000009 [glusterfs-000] 1361794636.184469 writev(7, [{"\20\0\0\0\376\377\377\377.\0\0\0\0\0\0\0", 16}], 1) = 16
0.000034 [glusterfs-000] 1361794636.184503 readv(7, [{"l\0\0\0#\0\0\0/\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"d37c4f0d239a2c468e5ba6fbd6e\0nc.p"..., 131072}], 2) = 108
0.000009 [glusterfs-000] 1361794636.184572 read(8, "6\355\365\212lD\263b!\264h\335Fx\353f", 16) = 16
0.000009 [glusterfs-000] 1361794636.184606 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0x7f7183ffe820) = -1 ENOENT (No such file or directory)
0.000007 [glusterfs-000] 1361794636.184641 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
0.000008 [glusterfs-000] 1361794636.184679 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0x7f7183ffe820) = -1 ENOENT (No such file or directory)
0.000117 [glusterfs-000] 1361794636.184713 open("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE|O_CLOEXEC, 0100755) = 10
0.000009 [glusterfs-000] 1361794636.184856 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000009 [glusterfs-000] 1361794636.184901 lgetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "trusted.gfid", 0x7f7183ffe900, 16) = -1 ENODATA (No data available)
0.000013 [glusterfs-000] 1361794636.184940 lsetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16, XATTR_CREATE) = 0
0.046522 [glusterfs-000] 1361794636.184989 lstat("/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6", 0x7f7183ffe680) = -1 ENOENT (No such file or directory)
0.000061 [glusterfs-000] 1361794636.231594 lstat("/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6", 0x7f7183ffe620) = -1 ENOENT (No such file or directory)
0.000099 [glusterfs-000] 1361794636.231722 lstat("/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6", 0x7f7183ffe790) = -1 ENOENT (No such file or directory)
0.000041 [glusterfs-000] 1361794636.231937 mkdir("/brick/swift/posix-only/.glusterfs/1c", 0700) = -1 EEXIST (File exists)
0.000016 [glusterfs-000] 1361794636.232021 mkdir("/brick/swift/posix-only/.glusterfs/1c/0b", 0700) = -1 EEXIST (File exists)
0.013147 [glusterfs-000] 1361794636.232072 link("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6") = 0
0.000037 [glusterfs-000] 1361794636.245286 lstat("/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000063 [glusterfs-000] 1361794636.245434 lstat("/brick/swift/posix-only/.glusterfs/1c/0b/1c0b1d78-5368-4967-9fd2-fbe76b1592a6", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000072 [glusterfs-000] 1361794636.245608 chown("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 0, 0) = 0
0.000063 [glusterfs-000] 1361794636.245753 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000011 [glusterfs-000] 1361794636.245888 fstat(10, {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000009 [glusterfs-000] 1361794636.245940 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000009 [glusterfs-000] 1361794636.245969 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=106, ...}) = 0
0.000011 [glusterfs-000] 1361794636.246014 writev(7, [{"\240\0\0\0\0\0\0\0/\0\0\0\0\0\0\0", 16}, {"|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 128}, {"\34\34\27\2\0\0\0\0\0\0\0\0\0\0\0\0", 16}], 3) = 160
<unfinis> [glusterfs-000] 1361794636.246062 readv(7,
0.061993 [object-server] 1361794636.246168 <... open resumed> ) = 15
<unfinis> [object-server] 1361794636.246331 fchown(15, 4294967295, 4294967295
0.000416 [glusterfs-000] 1361794636.246489 <... readv resumed> [{"D\0\0\0\26\0\0\0000\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"d37c4f0d239a2c468e5ba6fbd6e\0nc.p"..., 131072}], 2) = 68
0.000058 [glusterfs-000] 1361794636.246570 writev(7, [{"\20\0\0\0\303\377\377\3770\0\0\0\0\0\0\0", 16}], 1) = 16
0.000050 [glusterfs-000] 1361794636.246683 readv(7, [{"\200\0\0\0\4\0\0\0001\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072}], 2) = 128
0.000010 [glusterfs-000] 1361794636.246801 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000013 [glusterfs-000] 1361794636.246847 lchown("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", 4294967295, 4294967295) = 0
0.000008 [glusterfs-000] 1361794636.246885 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000010 [glusterfs-000] 1361794636.246930 writev(7, [{"x\0\0\0\0\0\0\0001\0\0\0\0\0\0\0", 16}, {"\1\0\0\0\0\0\0\0\0\0\0\0ShIg\246\222\25k\347\373\322\237\0\0\0\0\0\0\0\0"..., 104}], 2) = 120
<unfinis> [glusterfs-000] 1361794636.246969 readv(7,
0.000635 [object-server] 1361794636.246981 <... fchown resumed> ) = 0
<unfinis> [object-server] 1361794636.247132 sendto(14, "HTTP/1.1 100 Continue\r\n\r\n", 25, 0, NULL, 0
0.065513 [proxy -server] 1361794636.247259 <... poll resumed> ) = 1 ([{fd=14, revents=POLLIN}])
0.000128 [object-server] 1361794636.247278 <... sendto resumed> ) = 25
<unfinis> [object-server] 1361794636.247393 recvfrom(14,
<unfinis> [proxy -server] 1361794636.247421 recvfrom(14,
0.000030 [object-server] 1361794636.247438 <... recvfrom resumed> 0x14ea4b4, 7040, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
0.000028 [proxy -server] 1361794636.247456 <... recvfrom resumed> "H", 1, 0, NULL, NULL) = 1
<unfinis> [proxy -server] 1361794636.247604 recvfrom(14,
<unfinis> [object-server] 1361794636.247621 poll([{fd=14, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 60000
0.000097 [proxy -server] 1361794636.247716 <... recvfrom resumed> "T", 1, 0, NULL, NULL) = 1
0.000095 [proxy -server] 1361794636.247870 recvfrom(14, "T", 1, 0, NULL, NULL) = 1
0.000040 [proxy -server] 1361794636.248093 recvfrom(14, "P", 1, 0, NULL, NULL) = 1
0.000031 [proxy -server] 1361794636.248190 recvfrom(14, "/", 1, 0, NULL, NULL) = 1
0.000029 [proxy -server] 1361794636.248255 recvfrom(14, "1", 1, 0, NULL, NULL) = 1
0.000030 [proxy -server] 1361794636.248332 recvfrom(14, ".", 1, 0, NULL, NULL) = 1
0.000011 [proxy -server] 1361794636.248412 recvfrom(14, "1", 1, 0, NULL, NULL) = 1
0.000009 [proxy -server] 1361794636.248447 recvfrom(14, " ", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248483 recvfrom(14, "1", 1, 0, NULL, NULL) = 1
0.000008 [proxy -server] 1361794636.248510 recvfrom(14, "0", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248537 recvfrom(14, "0", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248564 recvfrom(14, " ", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248590 recvfrom(14, "C", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248616 recvfrom(14, "o", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248643 recvfrom(14, "n", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248669 recvfrom(14, "t", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248700 recvfrom(14, "i", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248727 recvfrom(14, "n", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248753 recvfrom(14, "u", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248780 recvfrom(14, "e", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248806 recvfrom(14, "\r", 1, 0, NULL, NULL) = 1
0.000007 [proxy -server] 1361794636.248833 recvfrom(14, "\n", 1, 0, NULL, NULL) = 1
0.000008 [proxy -server] 1361794636.248895 recvfrom(14, "\r", 1, 0, NULL, NULL) = 1
0.000008 [proxy -server] 1361794636.248924 recvfrom(14, "\n", 1, 0, NULL, NULL) = 1
0.000023 [proxy -server] 1361794636.249160 sendto(11, "HTTP/1.1 100 Continue\r\n\r\n", 25, 0, NULL, 0) = 25
0.000010 [proxy -server] 1361794636.249223 recvfrom(11, 0x1184384, 7040, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
0.000009 [proxy -server] 1361794636.249311 poll([{fd=11, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
0.000010 [proxy -server] 1361794636.249397 poll([{fd=11, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 4930) = 1 ([{fd=11, revents=POLLIN}])
0.000011 [proxy -server] 1361794636.249445 recvfrom(11, "#!/usr/bin/env python26\n#\n# func"..., 7040, 0, NULL, NULL) = 7040
0.000055 [proxy -server] 1361794636.249591 sendto(14, "#!/usr/bin/env python26\n#\n# func"..., 7040, 0, NULL, 0) = 7040
0.002032 [object-server] 1361794636.249666 <... poll resumed> ) = 1 ([{fd=14, revents=POLLIN}])
0.000013 [object-server] 1361794636.249749 recvfrom(14, "#!/usr/bin/env python26\n#\n# func"..., 7040, 0, NULL, NULL) = 7040
<unfinis> [object-server] 1361794636.249834 write(15, "#!/usr/bin/env python26\n#\n# func"..., 7040
0.000014 [proxy -server] 1361794636.249853 recvfrom(14, 0x11885e4, 8192, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
0.002921 [glusterfs-000] 1361794636.249896 <... readv resumed> [{"D\0\0\0\26\0\0\0002\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072}], 2) = 68
<unfinis> [proxy -server] 1361794636.249938 poll([{fd=14, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 4930
0.000016 [glusterfs-000] 1361794636.249954 writev(7, [{"\20\0\0\0\303\377\377\3772\0\0\0\0\0\0\0", 16}], 1) = 16
0.000012 [glusterfs-000] 1361794636.249998 readv(7, [{"\320\33\0\0\20\0\0\0003\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"#!/usr/bin/env python26\n#\n# func"..., 131072}], 2) = 7120
0.000007 [glusterfs-000] 1361794636.250046 fstat(10, {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
0.000008 [glusterfs-000] 1361794636.250078 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000029 [glusterfs-000] 1361794636.250106 pwrite(10, "#!/usr/bin/env python26\n#\n# func"..., 7040, 0) = 7040
0.000006 [glusterfs-000] 1361794636.250153 fstat(10, {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000008 [glusterfs-000] 1361794636.250185 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000012 [glusterfs-000] 1361794636.250213 writev(7, [{"\30\0\0\0\0\0\0\0003\0\0\0\0\0\0\0", 16}, {"\200\33\0\0\0\0\0\0", 8}], 2) = 24
0.000400 [object-server] 1361794636.250244 <... write resumed> ) = 7040
<unfinis> [glusterfs-000] 1361794636.250253 readv(7,
0.000016 [object-server] 1361794636.250473 fadvise64(15, 0, 7040, POSIX_FADV_DONTNEED) = 0
<unfinis> [object-server] 1361794636.250517 fsync(15
0.000290 [glusterfs-000] 1361794636.250551 <... readv resumed> [{"8\0\0\0\24\0\0\0004\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"#!/usr/bin/env python26\n#\n# func"..., 131072}], 2) = 56
0.000010 [glusterfs-000] 1361794636.250596 fstat(10, {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000007 [glusterfs-000] 1361794636.250642 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000469 [glusterfs-000] 1361794636.250668 fsync(10) = 0
0.000009 [glusterfs-000] 1361794636.251167 fstat(10, {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000016 [glusterfs-000] 1361794636.251205 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
<unfinis> [glusterfs-000] 1361794636.251273 writev(7, [{"\20\0\0\0\0\0\0\0004\0\0\0\0\0\0\0", 16}], 1
0.000830 [object-server] 1361794636.251358 <... fsync resumed> ) = 0
0.000078 [glusterfs-000] 1361794636.251370 <... writev resumed> ) = 16
<unfinis> [object-server] 1361794636.251459 setxattr("/mnt/gluster-object/AUTH_del0/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "user.swift.metadata", "\x80\x02}q\x01(U\x0eContent-Lengthq\x02U\x047040q\x03U\x04ETagq\x04U 69722aba2bee114799fb6c64758da529q\x05U\x0bX-Timestampq\x06U\x101361794636.17999q\x07U\x0dX-Object-Typeq\x08U\x04fileq\x09U\x06X-Typeq\x0aU\x06Objectq\x0bU\x0cContent-Typeq\x0cU\x0dtext/x-pythonq\x0du.", 191, 0
0.000050 [glusterfs-000] 1361794636.251503 readv(7, [{"8\0\0\0\3\0\0\0005\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"#!/usr/bin/env python26\n#\n# func"..., 131072}], 2) = 56
0.000012 [glusterfs-000] 1361794636.251615 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=106, ...}) = 0
0.000010 [glusterfs-000] 1361794636.251662 writev(7, [{"x\0\0\0\0\0\0\0005\0\0\0\0\0\0\0", 16}, {"\1\0\0\0\0\0\0\0\0\0\0\0004\255F\315\2\346\265\2,\2365\224j\0\0\0\0\0\0\0"..., 104}], 2) = 120
0.000009 [glusterfs-000] 1361794636.251703 readv(7, [{"8\0\0\0\3\0\0\0006\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"#!/usr/bin/env python26\n#\n# func"..., 131072}], 2) = 56
0.000007 [glusterfs-000] 1361794636.251743 fstat(10, {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000008 [glusterfs-000] 1361794636.251774 fgetxattr(10, "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000010 [glusterfs-000] 1361794636.251803 writev(7, [{"x\0\0\0\0\0\0\0006\0\0\0\0\0\0\0", 16}, {"\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\246\222\25k\347\373\322\237\200\33\0\0\0\0\0\0"..., 104}], 2) = 120
0.000008 [glusterfs-000] 1361794636.251842 readv(7, [{"\3\1\0\0\25\0\0\0007\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"ent-Lengthq\2U\0047040q\3U\4ETagq\4U 69"..., 131072}], 2) = 259
0.000010 [glusterfs-000] 1361794636.251887 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000016 [glusterfs-000] 1361794636.251933 lsetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "user.swift.metadata", "\x80\x02}q\x01(U\x0eContent-Lengthq\x02U\x047040q\x03U\x04ETagq\x04U 69722aba2bee114799fb6c64758da529q\x05U\x0bX-Timestampq\x06U\x101361794636.17999q\x07U\x0dX-Object-Typeq\x08U\x04fileq\x09U\x06X-Typeq\x0aU\x06Objectq\x0bU\x0cContent-Typeq\x0cU\x0dtext/x-pythonq\x0du.", 191, 0) = 0
0.000009 [glusterfs-000] 1361794636.251993 writev(7, [{"\20\0\0\0\0\0\0\0007\0\0\0\0\0\0\0", 16}], 1) = 16
0.000522 [object-server] 1361794636.252019 <... setxattr resumed> ) = 0
<unfinis> [glusterfs-000] 1361794636.252028 readv(7,
<unfinis> [object-server] 1361794636.252097 rename("/mnt/gluster-object/AUTH_del0/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "/mnt/gluster-object/AUTH_del0/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py"
0.000129 [glusterfs-000] 1361794636.252165 <... readv resumed> [{":\0\0\0\1\0\0\0008\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"ent-Lengthq\2U\0047040q\3U\4ETagq\4U 69"..., 131072}], 2) = 58
0.000010 [glusterfs-000] 1361794636.252239 read(8, "q\20\267=\354n-\30\0\1\3633.\355\321\361", 16) = 16
0.000009 [glusterfs-000] 1361794636.252271 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe8f0) = -1 ENOENT (No such file or directory)
0.000008 [glusterfs-000] 1361794636.252305 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe940) = -1 ENOENT (No such file or directory)
0.000007 [glusterfs-000] 1361794636.252336 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe7f0) = -1 ENOENT (No such file or directory)
0.000007 [glusterfs-000] 1361794636.252367 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=106, ...}) = 0
0.000009 [glusterfs-000] 1361794636.252408 writev(7, [{"\20\0\0\0\376\377\377\3778\0\0\0\0\0\0\0", 16}], 1) = 16
0.000008 [glusterfs-000] 1361794636.252440 readv(7, [{"v\0\0\0\f\0\0\0009\0\0\0\0\0\0\0T\301\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"239a2c468e5ba6fbd6e\0gluster_func"..., 131072}], 2) = 118
0.000008 [glusterfs-000] 1361794636.252481 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000009 [glusterfs-000] 1361794636.252525 lgetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000008 [glusterfs-000] 1361794636.252564 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe4b0) = -1 ENOENT (No such file or directory)
0.000008 [glusterfs-000] 1361794636.252595 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=106, ...}) = 0
0.000007 [glusterfs-000] 1361794636.252633 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=106, ...}) = 0
0.000007 [glusterfs-000] 1361794636.252671 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", 0x7f7183ffe4b0) = -1 ENOENT (No such file or directory)
0.000022 [glusterfs-000] 1361794636.252701 rename("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/.gluster_func.0.py.7425cd37c4f0d239a2c468e5ba6fbd6e", "/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py") = 0
0.000008 [glusterfs-000] 1361794636.252755 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", {st_mode=S_IFREG|0755, st_size=7040, ...}) = 0
0.000009 [glusterfs-000] 1361794636.252795 lgetxattr("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30/gluster_func.0.py", "trusted.gfid", "\x1c\x0b\x1dxShIg\x9f\xd2\xfb\xe7k\x15\x92\xa6", 16) = 16
0.000008 [glusterfs-000] 1361794636.252829 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=72, ...}) = 0
0.000008 [glusterfs-000] 1361794636.252867 lstat("/brick/swift/posix-only/gprfc019.pw32.ow32.mc1.dc2621440.pq10.thr32-posix-only.clth30", {st_mode=S_IFDIR|0755, st_size=72, ...}) = 0
0.000009 [glusterfs-000] 1361794636.252909 writev(7, [{"\20\0\0\0\0\0\0\0009\0\0\0\0\0\0\0", 16}], 1) = 16
0.000804 [object-server] 1361794636.252935 <... rename resumed> ) = 0
<unfinis> [glusterfs-000] 1361794636.252944 readv(7,
<unfinis> [object-server] 1361794636.253015 close(15
0.000097 [glusterfs-000] 1361794636.253049 <... readv resumed> [{"@\0\0\0\31\0\0\0:\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"239a2c468e5ba6fbd6e\0gluster_func"..., 131072}], 2) = 64
0.000009 [glusterfs-000] 1361794636.253097 writev(7, [{"\20\0\0\0\0\0\0\0:\0\0\0\0\0\0\0", 16}], 1) = 16
0.000102 [object-server] 1361794636.253123 <... close resumed> ) = 0
0.000008 [glusterfs-000] 1361794636.253132 readv(7, [{"@\0\0\0\22\0\0\0;\0\0\0\0\0\0\0|\302\331\210q\177\0\0\0\0\0\0\0\0\0\0"..., 80}, {"239a2c468e5ba6fbd6e\0gluster_func"..., 131072}], 2) = 64
0.000009 [glusterfs-000] 1361794636.253169 futex(0x216977c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x2169778, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
0.000009 [glusterfs-000] 1361794636.253200 writev(7, [{"\20\0\0\0\0\0\0\0;\0\0\0\0\0\0\0", 16}], 1) = 16
<unfinis> [glusterfs-000] 1361794636.253230 readv(7,
37.108957 [glusterfs-010] 1361794636.253308 <... futex resumed> ) = 0
<unfinis> [glusterfs-010] 1361794636.253413 futex(0x21697a8, FUTEX_WAKE_PRIVATE, 1
<unfinis> [object-server] 1361794636.253512 sendto(14, "HTTP/1.1 201 Created\r\nEtag: \"697"..., 301, 0, NULL, 0
0.000106 [glusterfs-010] 1361794636.253528 <... futex resumed> ) = 0
0.003600 [proxy -server] 1361794636.253547 <... poll resumed> ) = 1 ([{fd=14, revents=POLLIN}])
0.000037 [object-server] 1361794636.253560 <... sendto resumed> ) = 301
<unfinis> [proxy -server] 1361794636.253599 recvfrom(14,
<unfinis> [glusterfs-010] 1361794636.253610 close(10
0.000013 [proxy -server] 1361794636.253620 <... recvfrom resumed> "HTTP/1.1 201 Created\r\nEtag: \"697"..., 8192, 0, NULL, NULL) = 301
0.000020 [object-server] 1361794636.253655 shutdown(14, 2 /* send and receive */) = 0
0.000015 [object-server] 1361794636.253713 close(14) = 0
0.000132 [glusterfs-010] 1361794636.253745 <... close resumed> ) = 0
<unfinis> [glusterfs-010] 1361794636.253768 futex(0x216977c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 11, {1361795236, 0}, ffffffff
0.000013 [object-server] 1361794636.253815 accept(8, 0x7fff4d30ee50, [16]) = -1 EAGAIN (Resource temporarily unavailable)
0.000042 [proxy -server] 1361794636.253843 close(14) = 0
<unfinis> [object-server] 1361794636.253895 poll([{fd=8, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 60000
0.000015 [proxy -server] 1361794636.254176 sendto(11, "HTTP/1.1 201 Created\r\nContent-Le"..., 326, 0, NULL, 0) = 326
0.000058 [proxy -server] 1361794636.254697 sendto(10, "<142>proxy-server 172.17.10.19 1"..., 721, 0, NULL, 0) = 721
0.000009 [proxy -server] 1361794636.254859 recvfrom(11, "", 8192, 0, NULL, NULL) = 0
0.000012 [proxy -server] 1361794636.254910 shutdown(11, 2 /* send and receive */) = 0
0.000010 [proxy -server] 1361794636.254949 close(11) = 0
0.000012 [proxy -server] 1361794636.255026 accept(4, 0x7fff259ef030, [16]) = -1 EAGAIN (Resource temporarily unavailable)
@portante
Copy link
Author

portante commented Mar 2, 2013

From line 42 to line 52 is a stat() call that the object-server makes to check that we the account mapping is made to an actual mount point. This could be eliminated entirely, with a one time check by the DiskFile and DiskDir modules, or at least reduced to a periodic check once a minute or so. Coupled with line 54, which checks the mount point parent directory, almost 1.2ms could be eliminated per request on average, as well are reducing the work needed to be done by GlusterFS.

@portante
Copy link
Author

portante commented Mar 2, 2013

From line 55 to line 68 is a stat() call that the object-server makes to check to see if the file exists already. If it did exist, it would have been followed by two getxattr() calls to fetch the existing metadata. We could eliminate this pairing by attempting to read the metadata with getxattr() directly first, and checking for ENOENT to note the file does not exist.

For lines 57 and 58, why do we need to call lstat() and then lgetxattr() on the target object's parent directory, can't we just call lgetxattr() once and check for ENOENT for existence?

For line 59, why do we need to lstat() by name the mount point directory?

Lines 63, 64 and 65 file appear to be repeated lstat() system calls on the target object file name, is there are particular reason why we need to repeat this three times?

For line 66, we already called lstat() on the parent directory back on line 57, why do we need to do it again?

@portante
Copy link
Author

portante commented Mar 2, 2013

From line 70 through line 102 is the open() call that creates the temporary file. It took 62ms to do this call. Of that time, 46ms was spent on the lstat() call on line 87, and 13ms spent on the link() call on line 92. It is likely that the 46ms is a cache miss, and so too the link() call later. This is all work for handling of the .glusterfs tree.

Notice, the three stats of the target gluster internal file from lines 87-89.

Notice as well, that on lines 90 and 91, the mkdir() system calls return with EEXIST.

Given that fact, why don't we just first make the linkI() system call on line 92, and only if it returns ENOENT perform the mkdir() calls?

Also notice that on line 93 and 94 we call lstat() on the target of the link twice following its creation. Is that necessary?

So all told, could we not reduce lines 87-94 to be one call to link()? We might still pay the cache hit, but we'd pay it once rather than possibly twice, as evidenced by the 13ms hit, but in the fast case, we substantially reduce the overhead to perform this link creation.

Finally, when we look at lines 95 - 102, we see

  • a chown() of the new file on line 95 - why? Won't it already be owned by root?
  • an lstat() of it again on line 96 and an fstat() of the fd itself on line 97 - is there a need for these two? Are we comparing the results for a problem?
  • on line 98 we call fgetxattr() - is there a need to read these attributes for open semantics?
  • on line 99 we lstat() the parent directory of the target file again - do we need to do that?

@kalebskeithley
Copy link

Ugh, 11 months. Let me capture these comments while they're fresh in my mind.

Finally, when we look at lines 95 - 102, we see

  • a chown() of the new file on line 95 - why? Won't it already be owned by root?

In this case yes, because your test is running as root. We could eliminate the chown if we're only going to chown it to the same owner:group, i.e. root:root.

  • an lstat() of it again on line 96

The lstat here can probably be eliminated. This is in posix_acl_xattr_set(), which is called from multiple places, some that just have the path, or in this case we could pass the fd, omit the *stat(), and if there are acl attrs to set use fsetxattr instead of lsetxattr.

... and an fstat() of the fd itself on line 97 - is there a need for these two? Are we comparing the results for a problem?

No, this is used to fill in the iatt (inode attributes) of the file used later on. Well, perhaps maybe instead of no, let me do some more digging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment