gistfile1.txt

[2014-10-25 06:34:56.879706] C [client-handshake.c:127:rpc_client_ping_timer_expired] 0-integration_epa-client-0: server 192.168.30.107:49156 has not responded in the last 42 seconds, disconnecting.
[2014-10-25 06:34:56.879836] C [client-handshake.c:127:rpc_client_ping_timer_expired] 0-integration_epa-client-1: server 192.168.30.108:49156 has not responded in the last 42 seconds, disconnecting.
[2014-10-25 06:34:56.880026] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-10-25 06:34:14.499728 (xid=0x178df)
[2014-10-25 06:34:56.880042] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-10-25 06:34:56.880075] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-0: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2014-10-25 06:34:14.499773 (xid=0x178e0)
[2014-10-25 06:34:56.880084] W [client-handshake.c:276:client_ping_cbk] 0-integration_epa-client-0: timer must have expired
[2014-10-25 06:34:56.880094] I [client.c:2229:client_rpc_notify] 0-integration_epa-client-0: disconnected from 192.168.30.107:49156. Client process will keep trying to connect to glusterd until brick's port is available
[2014-10-25 06:34:56.880191] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-10-25 06:34:14.499807 (xid=0x13cc9)
[2014-10-25 06:34:56.880201] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-integration_epa-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2014-10-25 06:34:56.880287] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x361d80fced] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x361d80f833] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x361d80f74e]))) 0-integration_epa-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2014-10-25 06:34:14.499852 (xid=0x13cca)
[2014-10-25 06:34:56.880294] W [client-handshake.c:276:client_ping_cbk] 0-integration_epa-client-1: timer must have expired
[2014-10-25 06:34:56.880319] I [client.c:2229:client_rpc_notify] 0-integration_epa-client-1: disconnected from 192.168.30.108:49156. Client process will keep trying to connect to glusterd until brick's port is available
[2014-10-25 06:34:56.880326] E [afr-common.c:4168:afr_notify] 0-integration_epa-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2014-10-25 06:35:06.885139] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-integration_epa-client-0: changing port to 49156 (from 0)
[2014-10-25 06:35:06.889768] I [client-handshake.c:1677:select_server_supported_programs] 0-integration_epa-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-25 06:35:06.890127] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-integration_epa-client-1: changing port to 49156 (from 0)
[2014-10-25 06:35:06.892644] I [client-handshake.c:1462:client_setvolume_cbk] 0-integration_epa-client-0: Connected to 192.168.30.107:49156, attached to remote volume '/bricks/integration/epa/brick'.
[2014-10-25 06:35:06.892660] I [client-handshake.c:1474:client_setvolume_cbk] 0-integration_epa-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-25 06:35:06.892710] I [afr-common.c:4131:afr_notify] 0-integration_epa-replicate-0: Subvolume 'integration_epa-client-0' came back up; going online.
[2014-10-25 06:35:06.893438] I [client-handshake.c:450:client_set_lk_version_cbk] 0-integration_epa-client-0: Server lk version = 1
[2014-10-25 06:35:06.893665] I [client-handshake.c:1677:select_server_supported_programs] 0-integration_epa-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-25 06:35:06.894496] I [client-handshake.c:1462:client_setvolume_cbk] 0-integration_epa-client-1: Connected to 192.168.30.108:49156, attached to remote volume '/bricks/integration/epa/brick'.
[2014-10-25 06:35:06.894508] I [client-handshake.c:1474:client_setvolume_cbk] 0-integration_epa-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-25 06:35:06.895120] I [client-handshake.c:450:client_set_lk_version_cbk] 0-integration_epa-client-1: Server lk version = 1

gluster1.domain.local:/integration_epa /gluster/epa glusterfs _netdev,noatime,nodev,noexec,nosuid 0 0