Created a replica 2 volume:
# gluster volume info test1
Volume Name: test1
Type: Replicate
Volume ID: e1d66be2-a9d4-4073-9af9-4df4390dbd4e
Status: Created
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/bricks/brick1/brick
Brick2: gluster2:/bricks/brick1/brick
Options Reconfigured:
nfs.disable: onMounted that volume on a client; and started a loop to test writes while performing Gluster operations:
$ time for i in A B C D E F G H I J K L; do for j in $(seq 10 20); do mkdir -p /mnt/$i/$j; dd if=/dev/urandom of=/mnt/$i/$j/data bs=1M count=25; done; done;
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.48052 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45891 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.48406 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.50634 s, 10.5 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.5316 s, 10.4 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44455 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45351 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.4326 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.46512 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44994 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45568 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43247 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.47077 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43308 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43612 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.47728 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.47313 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.51108 s, 10.4 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45146 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.46933 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43313 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44823 s, 10.7 MB/sAt this point, on one of the gluster servers I performed this:
# gluster volume add-brick test1 replica 2 gluster1:/bricks/brick2/brick gluster2:/bricks/brick2/brickThe client writing data blew up:
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/10/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/11/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/12/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/13/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/14/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/15/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/16/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/17/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/18/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/19/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/20/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/10/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/11/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/12/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/13/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/14/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/15/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/16/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/17/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/18/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/19/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/20/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/10/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/11/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/12/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/13/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/14/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/15/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/16/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/17/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/18/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/19/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/20/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/10/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/11/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/12/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/13/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/14/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/15/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/16/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/17/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/18/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/19/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/20/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/10/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/11/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/12/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/13/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/14/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/15/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/16/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/17/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/18/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/19/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/20/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/10/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/11/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/12/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/13/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/14/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/15/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/16/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/17/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/18/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/19/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/20/data': No such file or directory
mkdir: cannot create directory `/mnt/I': Permission denied
dd: opening `/mnt/I/10/data': No such file or directory
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43536 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44204 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45516 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.47781 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.49671 s, 10.5 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.49074 s, 10.5 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.49009 s, 10.5 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44723 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43848 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.4368 s, 10.8 MB/s
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/10/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/11/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/12/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/13/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/14/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/15/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/16/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/17/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/18/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/19/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/20/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/10/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/11/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/12/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/13/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/14/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/15/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/16/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/17/data': No such file or directory
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.5286 s, 10.4 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.63556 s, 9.9 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44343 s, 10.7 MB/s
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/10/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/11/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/12/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/13/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/14/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/15/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/16/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/17/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/18/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/19/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/20/data': No such file or directory
real 1m27.789s
user 0m0.027s
sys 1m23.235s
[2014-08-11 17:56:59.382771] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: S
tarted running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs --volfile-
server=gluster1 --volfile-id=/test1 /mnt)
[2014-08-11 17:56:59.386313] I [socket.c:3561:socket_init] 0-glusterfs: SSL supp
ort is NOT enabled
[2014-08-11 17:56:59.386339] I [socket.c:3576:socket_init] 0-glusterfs: using sy
stem polling thread
[2014-08-11 17:56:59.393356] I [dht-shared.c:311:dht_init_regex] 0-test1-dht: us
ing regex rsync-hash-regex = ^.(.+).[^.]+$
[2014-08-11 17:56:59.394832] I [socket.c:3561:socket_init] 0-test1-client-1: SSL
support is NOT enabled
[2014-08-11 17:56:59.394843] I [socket.c:3576:socket_init] 0-test1-client-1: usi
ng system polling thread
[2014-08-11 17:56:59.395136] I [socket.c:3561:socket_init] 0-test1-client-0: SSL
support is NOT enabled
[2014-08-11 17:56:59.395146] I [socket.c:3576:socket_init] 0-test1-client-0: usi
ng system polling thread
[2014-08-11 17:56:59.395165] I [client.c:2294:notify] 0-test1-client-0: parent t
ranslators are ready, attempting connect on transport
[2014-08-11 17:56:59.397786] I [client.c:2294:notify] 0-test1-client-1: parent t
ranslators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume test1-client-0
2: type protocol/client
3: option remote-host gluster1
4: option remote-subvolume /bricks/brick1/brick
5: option transport-type socket
6: option send-gids true
7: end-volume
8:
9: volume test1-client-1
10: type protocol/client
11: option remote-host gluster2
12: option remote-subvolume /bricks/brick1/brick
13: option transport-type socket
14: option send-gids true
15: end-volume
16:
17: volume test1-replicate-0
18: type cluster/replicate
19: subvolumes test1-client-0 test1-client-1
20: end-volume
21:
22: volume test1-dht
23: type cluster/distribute
24: subvolumes test1-replicate-0
25: end-volume
26:
27: volume test1-write-behind
28: type performance/write-behind
29: subvolumes test1-dht
30: end-volume
31:
32: volume test1-read-ahead
33: type performance/read-ahead
34: subvolumes test1-write-behind
35: end-volume
36:
37: volume test1-io-cache
38: type performance/io-cache
39: subvolumes test1-read-ahead
40: end-volume
41:
42: volume test1-quick-read
43: type performance/quick-read
44: subvolumes test1-io-cache
45: end-volume
46:
47: volume test1-open-behind
48: type performance/open-behind
49: subvolumes test1-quick-read
50: end-volume
51:
52: volume test1-md-cache
53: type performance/md-cache
54: subvolumes test1-open-behind
55: end-volume
56:
57: volume test1
58: type debug/io-stats
59: option latency-measurement off
60: option count-fop-hits off
61: subvolumes test1-md-cache
62: end-volume
63:
+------------------------------------------------------------------------------+
[2014-08-11 17:56:59.401291] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49157 (from 0)
[2014-08-11 17:56:59.404171] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49157 (from 0)
[2014-08-11 17:56:59.407034] I [client-handshake.c:1677:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-08-11 17:56:59.407187] I [client-handshake.c:1677:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-08-11 17:56:59.407537] I [client-handshake.c:1462:client_setvolume_cbk] 0-test1-client-0: Connected to 192.168.30.107:49157, attached to remote volume '/bricks/brick1/brick'.
[2014-08-11 17:56:59.407570] I [client-handshake.c:1474:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-08-11 17:56:59.407661] I [afr-common.c:4131:afr_notify] 0-test1-replicate-0: Subvolume 'test1-client-0' came back up; going online.
[2014-08-11 17:56:59.407686] I [client-handshake.c:1462:client_setvolume_cbk] 0-test1-client-1: Connected to 192.168.30.108:49157, attached to remote volume '/bricks/brick1/brick'.
[2014-08-11 17:56:59.407693] I [client-handshake.c:1474:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-08-11 17:56:59.411264] I [fuse-bridge.c:4977:fuse_graph_setup] 0-fuse: switched to graph 0
[2014-08-11 17:56:59.411314] I [client-handshake.c:450:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1
[2014-08-11 17:56:59.411328] I [client-handshake.c:450:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1
[2014-08-11 17:56:59.411467] I [fuse-bridge.c:3914:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.13
[2014-08-11 17:56:59.412724] I [dht-layout.c:640:dht_layout_normalize] 0-test1-dht: found anomalies in /. holes=1 overlaps=0
[2014-08-11 17:58:52.290674] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-08-11 17:58:52.295585] I [dht-shared.c:311:dht_init_regex] 1-test1-dht: using regex rsync-hash-regex = ^.(.+).[^.]+$
[2014-08-11 17:58:52.301444] I [socket.c:3561:socket_init] 1-test1-client-3: SSL support is NOT enabled
[2014-08-11 17:58:52.301458] I [socket.c:3576:socket_init] 1-test1-client-3: using system polling thread
[2014-08-11 17:58:52.301610] I [socket.c:3561:socket_init] 1-test1-client-2: SSL support is NOT enabled
[2014-08-11 17:58:52.301618] I [socket.c:3576:socket_init] 1-test1-client-2: using system polling thread
[2014-08-11 17:58:52.301762] I [socket.c:3561:socket_init] 1-test1-client-1: SSL support is NOT enabled
[2014-08-11 17:58:52.301772] I [socket.c:3576:socket_init] 1-test1-client-1: using system polling thread
[2014-08-11 17:58:52.301912] I [socket.c:3561:socket_init] 1-test1-client-0: SSL support is NOT enabled
[2014-08-11 17:58:52.301922] I [socket.c:3576:socket_init] 1-test1-client-0: using system polling thread
[2014-08-11 17:58:52.301935] I [client.c:2294:notify] 1-test1-client-0: parent translators are ready, attempting connect on transport
[2014-08-11 17:58:52.304824] I [client.c:2294:notify] 1-test1-client-1: parent translators are ready, attempting connect on transport
[2014-08-11 17:58:52.309975] I [client.c:2294:notify] 1-test1-client-2: parent translators are ready, attempting connect on transport
[2014-08-11 17:58:52.312574] I [client.c:2294:notify] 1-test1-client-3: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume test1-client-0
2: type protocol/client
3: option remote-host gluster1
4: option remote-subvolume /bricks/brick1/brick
5: option transport-type socket
6: option send-gids true
7: end-volume
8:
9: volume test1-client-1
10: type protocol/client
11: option remote-host gluster2
12: option remote-subvolume /bricks/brick1/brick
13: option transport-type socket
14: option send-gids true
15: end-volume
16:
17: volume test1-replicate-0
18: type cluster/replicate
19: subvolumes test1-client-0 test1-client-1
20: end-volume
21:
22: volume test1-client-2
23: type protocol/client
24: option remote-host gluster1
25: option remote-subvolume /bricks/brick2/brick
26: option transport-type socket
27: option send-gids true
28: end-volume
29:
30: volume test1-client-3
31: type protocol/client
32: option remote-host gluster2
33: option remote-subvolume /bricks/brick2/brick
34: option transport-type socket
35: option send-gids true
36: end-volume
37:
38: volume test1-replicate-1
39: type cluster/replicate
40: subvolumes test1-client-2 test1-client-3
41: end-volume
42:
43: volume test1-dht
44: type cluster/distribute
45: subvolumes test1-replicate-0 test1-replicate-1
46: end-volume
47:
48: volume test1-write-behind
49: type performance/write-behind
50: subvolumes test1-dht
51: end-volume
52:
53: volume test1-read-ahead
54: type performance/read-ahead
55: subvolumes test1-write-behind
56: end-volume
57:
58: volume test1-io-cache
59: type performance/io-cache
60: subvolumes test1-read-ahead
61: end-volume
62:
63: volume test1-quick-read
64: type performance/quick-read
65: subvolumes test1-io-cache
66: end-volume
67:
68: volume test1-open-behind
69: type performance/open-behind
70: subvolumes test1-quick-read
71: end-volume
72:
73: volume test1-md-cache
74: type performance/md-cache
75: subvolumes test1-open-behind
76: end-volume
77:
78: volume test1
79: type debug/io-stats
80: option latency-measurement off
81: option count-fop-hits off
82: subvolumes test1-md-cache
83: end-volume
84:
+------------------------------------------------------------------------------+
[2014-08-11 17:58:52.321328] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 1-test1-client-0: changing port to 49157 (from 0)
[2014-08-11 17:58:52.323981] E [client-handshake.c:1760:client_query_portmap_cbk] 1-test1-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2014-08-11 17:58:52.324028] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 1-test1-client-1: changing port to 49157 (from 0)
[2014-08-11 17:58:52.324126] E [client-handshake.c:1760:client_query_portmap_cbk] 1-test1-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2014-08-11 17:58:52.324206] I [client.c:2229:client_rpc_notify] 1-test1-client-2: disconnected from 192.168.30.107:24007. Client process will keep trying to connect to glusterd until brick's port is available
[2014-08-11 17:58:52.328951] I [client.c:2229:client_rpc_notify] 1-test1-client-3: disconnected from 192.168.30.108:24007. Client process will keep trying to connect to glusterd until brick's port is available
[2014-08-11 17:58:52.328965] E [afr-common.c:4168:afr_notify] 1-test1-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2014-08-11 17:58:52.329245] I [client-handshake.c:1677:select_server_supported_programs] 1-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-08-11 17:58:52.331098] I [client-handshake.c:1677:select_server_supported_programs] 1-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-08-11 17:58:52.331149] I [client-handshake.c:1462:client_setvolume_cbk] 1-test1-client-0: Connected to 192.168.30.107:49157, attached to remote volume '/bricks/brick1/brick'.
[2014-08-11 17:58:52.331175] I [client-handshake.c:1474:client_setvolume_cbk] 1-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-08-11 17:58:52.331245] I [afr-common.c:4131:afr_notify] 1-test1-replicate-0: Subvolume 'test1-client-0' came back up; going online.
[2014-08-11 17:58:52.331420] I [client-handshake.c:450:client_set_lk_version_cbk] 1-test1-client-0: Server lk version = 1
[2014-08-11 17:58:52.331510] I [client-handshake.c:1462:client_setvolume_cbk] 1-test1-client-1: Connected to 192.168.30.108:49157, attached to remote volume '/bricks/brick1/brick'.
[2014-08-11 17:58:52.331520] I [client-handshake.c:1474:client_setvolume_cbk] 1-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-08-11 17:58:52.338465] I [fuse-bridge.c:4977:fuse_graph_setup] 0-fuse: switched to graph 1
[2014-08-11 17:58:52.338498] I [client-handshake.c:450:client_set_lk_version_cbk] 1-test1-client-1: Server lk version = 1
[2014-08-11 17:58:52.387143] I [fuse-bridge.c:4189:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x99d26c) to newfd (0xa6cc3c) (inode-gfid:6a3350fc-13d6-4e4e-865b-cd34097566e6)(old-subvolume:test1-0 new-subvolume:test1-1)
[2014-08-11 17:58:52.387189] I [client.c:2303:notify] 0-test1-client-0: current graph is no longer active, destroying rpc_client
[2014-08-11 17:58:52.387231] I [client.c:2303:notify] 0-test1-client-1: current graph is no longer active, destroying rpc_client
[2014-08-11 17:58:52.387446] I [client.c:2229:client_rpc_notify] 0-test1-client-0: disconnected from 192.168.30.107:49157. Client process will keep trying to connect to glusterd until brick's port is available
[2014-08-11 17:58:52.387467] I [client.c:2229:client_rpc_notify] 0-test1-client-1: disconnected from 192.168.30.108:49157. Client process will keep trying to connect to glusterd until brick's port is available
[2014-08-11 17:58:52.387480] E [afr-common.c:4168:afr_notify] 0-test1-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2014-08-11 17:58:53.977647] W [defaults.c:1381:default_release](-->/usr/lib64/glusterfs/3.5.2/xlator/protocol/client.so%28client3_3_finodelk_cbk+0x153%29 [0x7f06ff9408f3] %28-->/usr/lib64/glusterfs/3.5.2/xlator/protocol/client.so%28client_local_wipe+0x28%29 [0x7f06ff934b88] %28-->/usr/lib64/libglusterfs.so.0%28fd_unref+0x10e%29 [0x3129e3efbe]%29%29) 0-fuse: xlator does not implement release_cbk
[2014-08-11 17:58:53.989416] I [afr-common.c:4299:afr_local_init] 1-test1-replicate-1: no subvolumes up
[2014-08-11 17:58:53.989438] W [dht-diskusage.c:45:dht_du_info_cbk] 1-test1-dht: failed to get disk info from test1-replicate-1
[2014-08-11 17:58:53.990286] I [afr-common.c:4299:afr_local_init] 1-test1-replicate-1: no subvolumes up
[2014-08-11 17:58:55.406042] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 1-test1-client-2: changing port to 49158 (from 0)
[2014-08-11 17:58:55.412373] I [client-handshake.c:1677:select_server_supported_programs] 1-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-08-11 17:58:55.412663] I [client-handshake.c:1462:client_setvolume_cbk] 1-test1-client-2: Connected to 192.168.30.107:49158, attached to remote volume '/bricks/brick2/brick'.
[2014-08-11 17:58:55.412674] I [client-handshake.c:1474:client_setvolume_cbk] 1-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2014-08-11 17:58:55.412760] I [afr-common.c:4131:afr_notify] 1-test1-replicate-1: Subvolume 'test1-client-2' came back up; going online.
[2014-08-11 17:58:55.412847] I [client-handshake.c:450:client_set_lk_version_cbk] 1-test1-client-2: Server lk version = 1
[2014-08-11 17:58:55.519816] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 1-test1-client-3: changing port to 49158 (from 0)
[2014-08-11 17:58:55.525333] I [client-handshake.c:1677:select_server_supported_programs] 1-test1-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-08-11 17:58:55.525719] I [client-handshake.c:1462:client_setvolume_cbk] 1-test1-client-3: Connected to 192.168.30.108:49158, attached to remote volume '/bricks/brick2/brick'.
[2014-08-11 17:58:55.525730] I [client-handshake.c:1474:client_setvolume_cbk] 1-test1-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2014-08-11 17:58:55.525875] I [client-handshake.c:450:client_set_lk_version_cbk] 1-test1-client-3: Server lk version = 1
[2014-08-11 17:58:58.930719] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:58:58.930751] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:01.375442] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:01.375466] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.835809] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.835832] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.839280] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.839292] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.848317] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.848329] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.857268] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.857280] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.866239] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.866251] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht
: mismatching layouts for /
[2014-08-11 17:59:03.875264] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.875275] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.884272] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.884284] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.893250] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.893261] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.901921] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.901933] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.910704] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.910716] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.919363] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.919375] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.927938] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.927950] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.937307] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.937320] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.946126] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.946141] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.955060] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.955102] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.964025] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.964037] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.972859] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.972871] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.981330] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.981342] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.990172] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.990196] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:03.999173] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:03.999185] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.007705] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.007719] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.016427] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.016443] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.025242] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.025254] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.034742] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.034753] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.043297] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.043309] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.051966] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.051978] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.060658] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.060670] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.069314] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.069326] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.078004] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.078016] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.087185] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.087197] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.095958] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.095969] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.104899] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.104911] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.113736] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.113748] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.122413] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test
1-dht: / - disk layout missing
[2014-08-11 17:59:04.122425] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.131838] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.131850] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.140445] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.140457] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.149214] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.149226] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.158020] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.158042] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.166890] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.166902] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.175396] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.175408] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.183958] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.183969] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.192785] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.192796] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.201410] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.201422] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.210253] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.210265] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.218985] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.218996] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.228473] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.228485] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.237238] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:04.237249] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:04.245720] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
....
....
[2014-08-11 17:59:04.428495] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 1-tes
t1-client-2: remote operation failed: Permission denied. Path: /I
[2014-08-11 17:59:04.428597] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 1-tes
t1-client-3: remote operation failed: Permission denied. Path: /I
[2014-08-11 17:59:04.430107] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test
1-dht: / - disk layout missing
[2014-08-11 17:59:04.430118] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht
: mismatching layouts for /
[2014-08-11 17:59:04.431209] W [client-rpc-fops.c:2690:client3_3_opendir_cbk] 1-
test1-client-3: remote operation failed: Stale file handle. Path: /I (bdb7c9f0-8
b32-4092-8e53-e8a32f151148)
[2014-08-11 17:59:04.431228] W [client-rpc-fops.c:2690:client3_3_opendir_cbk] 1-
test1-client-2: remote operation failed: Stale file handle. Path: /I (bdb7c9f0-8
b32-4092-8e53-e8a32f151148)
[2014-08-11 17:59:04.434121] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-
test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:04.434148] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-
test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:04.434562] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-
test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:04.435029] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-
test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:04.435041] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test1-
replicate-1: unable to lock on even one child
[2014-08-11 17:59:04.435046] I [afr-transaction.c:1295:afr_post_blocking_entrylk
_cbk] 1-test1-replicate-1: Blocking entrylks failed.
[2014-08-11 17:59:06.876886] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test
1-dht: / - disk layout missing
[2014-08-11 17:59:06.876909] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:06.880942] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:06.880982] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:06.881401] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:06.881808] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:06.881819] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test1-replicate-1: unable to lock on even one child
[2014-08-11 17:59:06.881825] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test1-replicate-1: Blocking entrylks failed.
[2014-08-11 17:59:09.330419] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:09.330444] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:09.334365] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:09.337777] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:09.338250] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:09.338723] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:09.338734] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test1-replicate-1: unable to lock on even one child
[2014-08-11 17:59:09.338740] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test1-replicate-1: Blocking entrylks failed.
[2014-08-11 17:59:11.800575] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:11.800598] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:11.804541] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:11.804598] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:11.805036] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:11.805454] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:11.805466] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test1-replicate-1: unable to lock on even one child
[2014-08-11 17:59:11.805472] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test1-replicate-1: Blocking entrylks failed.
[2014-08-11 17:59:14.290260] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:14.290287] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:14.294250] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:14.294297] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:14.294725] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:14.295190] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-
test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:14.295204] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test1-replicate-1: unable to lock on even one child
[2014-08-11 17:59:14.295214] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test1-replicate-1: Blocking entrylks failed.
[2014-08-11 17:59:16.798397] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:16.798423] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:16.802402] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:16.802450] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:16.802869] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:16.803316] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:16.803327] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test1-replicate-1: unable to lock on even one child
[2014-08-11 17:59:16.803333] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test1-replicate-1: Blocking entrylks failed.
[2014-08-11 17:59:19.300746] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:19.300770] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:19.304845] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:19.304888] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:19.305299] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:19.305728] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:19.305742] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test1-replicate-1: unable to lock on even one child
[2014-08-11 17:59:19.305748] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test1-replicate-1: Blocking entrylks failed.
[2014-08-11 17:59:21.802573] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:21.802598] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /
[2014-08-11 17:59:21.806848] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:21.807421] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:21.807852] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-2: remote operation failed: Stale file handle
[2014-08-11 17:59:21.808314] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test1-client-3: remote operation failed: Stale file handle
[2014-08-11 17:59:21.808328] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test1-replicate-1: unable to lock on even one child
[2014-08-11 17:59:21.808335] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test1-replicate-1: Blocking entrylks failed.
[2014-08-11 17:59:24.262138] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test1-dht: / - disk layout missing
[2014-08-11 17:59:24.262162] I [dht-common.c:635:dht_revalidate_cbk] 1-test1-dht: mismatching layouts for /