gluster disconnect

gistfile1.txt

[2014-11-19 02:30:27.641372] C [client-handshake.c:127:rpc_client_ping_timer_expired] 0-integration_epa-client-0: server 192.168.30.107:49156 has not responded in the last 5 seconds, disconnecting.
[2014-11-19 02:30:27.641454] C [client-handshake.c:127:rpc_client_ping_timer_expired] 0-integration_epa-client-1: server 192.168.30.108:49156 has not responded in the last 5 seconds, disconnecting.
[2014-11-19 02:30:27.641521] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2014-11-19 02:30:18.906504 (xid=0x24173)
[2014-11-19 02:30:27.641533] W [client-rpc-fops.c:866:client3_3_writev_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected
[2014-11-19 02:30:27.641573] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:19.267308 (xid=0x24174)
[2014-11-19 02:30:27.641584] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.641606] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:19.768895 (xid=0x24175)
[2014-11-19 02:30:27.641612] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.641627] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:19.797737 (xid=0x24176)
[2014-11-19 02:30:27.641633] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.641647] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:19.910429 (xid=0x24177)
[2014-11-19 02:30:27.641653] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.641667] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:20.276212 (xid=0x24178)
[2014-11-19 02:30:27.641673] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.641687] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2014-11-19 02:30:22.641007 (xid=0x24179)
[2014-11-19 02:30:27.641692] W [client-handshake.c:276:client_ping_cbk] 0-integration_epa-client-0: timer must have expired
[2014-11-19 02:30:27.641715] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:24.032211 (xid=0x2417a)
[2014-11-19 02:30:27.641721] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.641736] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:24.088757 (xid=0x2417b)
[2014-11-19 02:30:27.641742] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.641751] I [client.c:2229:client_rpc_notify] 0-integration_epa-client-0: disconnected from 192.168.30.107:49156. Client process will keep trying to connect to glusterd until brick's port is available
[2014-11-19 02:30:27.641780] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2014-11-19 02:30:18.906556 (xid=0x24447)
[2014-11-19 02:30:27.641786] W [client-rpc-fops.c:866:client3_3_writev_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected
[2014-11-19 02:30:27.641810] W [client-rpc-fops.c:1812:client3_3_fxattrop_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected
[2014-11-19 02:30:27.644314] I [socket.c:3060:socket_submit_request] 0-integration_epa-client-1: not connected (priv->connected = 0)
[2014-11-19 02:30:27.644331] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-integration_epa-client-1: failed to submit rpc-request (XID: 0x24450 Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (integration_epa-client-1)
[2014-11-19 02:30:27.644340] W [client-rpc-fops.c:1812:client3_3_fxattrop_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected
[2014-11-19 02:30:27.644351] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected
[2014-11-19 02:30:27.649138] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-integration_epa-client-1: failed to submit rpc-request (XID: 0x24451 Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (integration_epa-client-1)
[2014-11-19 02:30:27.649155] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected
[2014-11-19 02:30:27.649181] W [fuse-bridge.c:1214:fuse_err_cbk] 0-glusterfs-fuse: 200684: FLUSH() ERR => -1 (Transport endpoint is not connected)
[2014-11-19 02:30:27.649265] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:19.267319 (xid=0x24448)
[2014-11-19 02:30:27.649276] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.649339] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:19.768907 (xid=0x24449)
[2014-11-19 02:30:27.649346] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.649385] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:19.797749 (xid=0x2444a)
[2014-11-19 02:30:27.649392] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.649436] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:19.910440 (xid=0x2444b)
[2014-11-19 02:30:27.649443] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.649479] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:20.276228 (xid=0x2444c)
[2014-11-19 02:30:27.649486] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.649545] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2014-11-19 02:30:22.641015 (xid=0x2444d)
[2014-11-19 02:30:27.649551] W [client-handshake.c:276:client_ping_cbk] 0-integration_epa-client-1: timer must have expired
[2014-11-19 02:30:27.649567] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:24.032228 (xid=0x2444e)
[2014-11-19 02:30:27.649574] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.649640] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-11-19 02:30:24.088768 (xid=0x2444f)
[2014-11-19 02:30:27.649648] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-11-19 02:30:27.649700] I [client.c:2229:client_rpc_notify] 0-integration_epa-client-1: disconnected from 192.168.30.108:49156. Client process will keep trying to connect to glusterd until brick's port is available
[2014-11-19 02:30:27.649709] E [afr-common.c:4168:afr_notify] 0-integration_epa-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2014-11-19 02:30:27.651250] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-integration_epa-client-1: changing port to 49156 (from 0)
[2014-11-19 02:30:27.669251] I [client-handshake.c:1677:select_server_supported_programs] 0-integration_epa-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-11-19 02:30:27.670603] I [client-handshake.c:1462:client_setvolume_cbk] 0-integration_epa-client-1: Connected to 192.168.30.108:49156, attached to remote volume '/bricks/integration/epa/brick'.
[2014-11-19 02:30:27.670616] I [client-handshake.c:1474:client_setvolume_cbk] 0-integration_epa-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-11-19 02:30:27.670622] I [client-handshake.c:1314:client_post_handshake] 0-integration_epa-client-1: 1 fds open - Delaying child_up until they are re-opened
[2014-11-19 02:30:27.674479] I [client-handshake.c:936:client_child_up_reopen_done] 0-integration_epa-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2014-11-19 02:30:27.674515] I [afr-common.c:4131:afr_notify] 0-integration_epa-replicate-0: Subvolume 'integration_epa-client-1' came back up; going online.
[2014-11-19 02:30:27.678443] I [client-handshake.c:450:client_set_lk_version_cbk] 0-integration_epa-client-1: Server lk version = 1
[2014-11-19 02:30:37.646662] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-integration_epa-client-0: changing port to 49156 (from 0)
[2014-11-19 02:30:37.650118] I [client-handshake.c:1677:select_server_supported_programs] 0-integration_epa-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-11-19 02:30:37.651097] I [client-handshake.c:1462:client_setvolume_cbk] 0-integration_epa-client-0: Connected to 192.168.30.107:49156, attached to remote volume '/bricks/integration/epa/brick'.
[2014-11-19 02:30:37.651110] I [client-handshake.c:1474:client_setvolume_cbk] 0-integration_epa-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-11-19 02:30:37.651116] I [client-handshake.c:1314:client_post_handshake] 0-integration_epa-client-0: 1 fds open - Delaying child_up until they are re-opened
[2014-11-19 02:30:37.651923] I [client-handshake.c:936:client_child_up_reopen_done] 0-integration_epa-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2014-11-19 02:30:37.652662] I [client-handshake.c:450:client_set_lk_version_cbk] 0-integration_epa-client-0: Server lk version = 1
[2014-11-19 02:30:38.790364] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:1a3c133e-caee-45c5-b45e-ab5aa4049ab7> (1a3c133e-caee-45c5-b45e-ab5aa4049ab7)
[2014-11-19 02:30:38.794665] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:a913445f-d112-487e-876e-527323879443> (a913445f-d112-487e-876e-527323879443)
[2014-11-19 02:30:38.794873] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:6271ab29-b14f-4620-92c4-1eae5ac719cf> (6271ab29-b14f-4620-92c4-1eae5ac719cf)
[2014-11-19 02:30:38.794967] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:4831b771-f999-4942-b494-c266db82d7bb> (4831b771-f999-4942-b494-c266db82d7bb)
[2014-11-19 02:30:38.795105] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:cef14f59-c407-4822-8677-6266f2e05147> (cef14f59-c407-4822-8677-6266f2e05147)
[2014-11-19 02:30:38.799679] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:29c191e0-d53f-4b41-a031-5313f29ca5e8> (29c191e0-d53f-4b41-a031-5313f29ca5e8)
[2014-11-19 02:30:38.799843] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:82c37f83-f3d2-456f-ac71-c1244990ceaa> (82c37f83-f3d2-456f-ac71-c1244990ceaa)
[2014-11-19 02:30:38.799957] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:faf4d98d-16cf-49b0-8992-bffa2e3e5827> (faf4d98d-16cf-49b0-8992-bffa2e3e5827)
[2014-11-19 02:30:38.800061] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:eecaca21-2d06-4096-9842-1f92d0b72636> (eecaca21-2d06-4096-9842-1f92d0b72636)
[2014-11-19 02:30:38.804172] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:82552ff5-3155-4c57-8852-fc7d2ee141e6> (82552ff5-3155-4c57-8852-fc7d2ee141e6)
[2014-11-19 02:30:38.804547] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:54590a0c-24d6-44cd-84de-3d6fc23ed3c8> (54590a0c-24d6-44cd-84de-3d6fc23ed3c8)
[2014-11-19 02:30:38.804613] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:37785cc8-1031-453d-a727-93e05201a317> (37785cc8-1031-453d-a727-93e05201a317)
[2014-11-19 02:30:38.804674] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:14a94523-301f-4b4e-bb3c-c700c0748686> (14a94523-301f-4b4e-bb3c-c700c0748686)
[2014-11-19 02:30:38.808984] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:9869a446-323a-4f46-898b-446dafa48f8b> (9869a446-323a-4f46-898b-446dafa48f8b)
[2014-11-19 02:30:38.809334] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: No such file or directory. Path: <gfid:3c207d98-0550-4fc0-bd73-6af67c1e5f22> (3c207d98-0550-4fc0-bd73-6af67c1e5f22)
[2014-11-19 02:30:38.812749] I [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-integration_epa-replicate-0:  metadata self heal  is successfully completed, entry self heal  is successfully completed,   metadata self heal from source integration_epa-client-1 to integration_epa-client-0,  metadata - Pending matrix:  [ [ 0 0 ] [ 1 0 ] ], on /messages/2014/11/18
[2014-11-19 02:31:11.799212] I [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-integration_epa-replicate-0:  metadata self heal  is successfully completed, backgroung data self heal  is successfully completed,  data self heal from integration_epa-client-1  to sinks  integration_epa-client-0, with 0 bytes on integration_epa-client-0, 6592 bytes on integration_epa-client-1,  data - Pending matrix:  [ [ 0 0 ] [ 5 0 ] ]  metadata self heal from source integration_epa-client-1 to integration_epa-client-0,  metadata - Pending matrix:  [ [ 0 0 ] [ 3 0 ] ], on /messages/2014/11/18/incoming_optum_2014-11-18T21:30:36-05:00.xml