gluster add bricks.md

Created a replica 2 volume:

# gluster volume info test1

Volume Name: test1
Type: Replicate
Volume ID: e1d66be2-a9d4-4073-9af9-4df4390dbd4e
Status: Created
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/bricks/brick1/brick
Brick2: gluster2:/bricks/brick1/brick
Options Reconfigured:
nfs.disable: on

Mounted that volume on a client; and started a loop to test writes while performing Gluster operations:

$ time for i in A B C D E F G H I J K L; do for j in $(seq 10 20); do mkdir -p /mnt/$i/$j; dd if=/dev/urandom of=/mnt/$i/$j/data bs=1M count=25; done; done;
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.48052 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45891 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.48406 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.50634 s, 10.5 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.5316 s, 10.4 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44455 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45351 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.4326 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.46512 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44994 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45568 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43247 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.47077 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43308 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43612 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.47728 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.47313 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.51108 s, 10.4 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45146 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.46933 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43313 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44823 s, 10.7 MB/s

At this point, on one of the gluster servers I performed this:

# gluster volume add-brick test1 replica 2 gluster1:/bricks/brick2/brick gluster2:/bricks/brick2/brick

The client writing data blew up:

mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/10/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/11/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/12/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/13/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/14/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/15/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/16/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/17/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/18/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/19/data': No such file or directory
mkdir: cannot create directory `/mnt/C': Permission denied
dd: opening `/mnt/C/20/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/10/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/11/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/12/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/13/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/14/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/15/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/16/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/17/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/18/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/19/data': No such file or directory
mkdir: cannot create directory `/mnt/D': Permission denied
dd: opening `/mnt/D/20/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/10/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/11/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/12/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/13/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/14/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/15/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/16/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/17/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/18/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/19/data': No such file or directory
mkdir: cannot create directory `/mnt/E': Permission denied
dd: opening `/mnt/E/20/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/10/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/11/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/12/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/13/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/14/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/15/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/16/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/17/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/18/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/19/data': No such file or directory
mkdir: cannot create directory `/mnt/F': Permission denied
dd: opening `/mnt/F/20/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/10/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/11/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/12/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/13/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/14/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/15/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/16/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/17/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/18/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/19/data': No such file or directory
mkdir: cannot create directory `/mnt/G': Permission denied
dd: opening `/mnt/G/20/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/10/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/11/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/12/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/13/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/14/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/15/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/16/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/17/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/18/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/19/data': No such file or directory
mkdir: cannot create directory `/mnt/H': Permission denied
dd: opening `/mnt/H/20/data': No such file or directory
mkdir: cannot create directory `/mnt/I': Permission denied
dd: opening `/mnt/I/10/data': No such file or directory
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43536 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44204 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45516 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.47781 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.49671 s, 10.5 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.49074 s, 10.5 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.49009 s, 10.5 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44723 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43848 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.4368 s, 10.8 MB/s
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/10/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/11/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/12/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/13/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/14/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/15/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/16/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/17/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/18/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/19/data': No such file or directory
mkdir: cannot create directory `/mnt/J': Permission denied
dd: opening `/mnt/J/20/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/10/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/11/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/12/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/13/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/14/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/15/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/16/data': No such file or directory
mkdir: cannot create directory `/mnt/K': Permission denied
dd: opening `/mnt/K/17/data': No such file or directory
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.5286 s, 10.4 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.63556 s, 9.9 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44343 s, 10.7 MB/s
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/10/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/11/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/12/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/13/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/14/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/15/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/16/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/17/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/18/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/19/data': No such file or directory
mkdir: cannot create directory `/mnt/L': Permission denied
dd: opening `/mnt/L/20/data': No such file or directory

real	1m27.789s
user	0m0.027s
sys	1m23.235s

[2014-08-12 14:43:09.714414] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs --volfile-server=gluster1 --volfile-id=/t1 /mnt)
[2014-08-12 14:43:09.717289] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
[2014-08-12 14:43:09.717314] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
[2014-08-12 14:43:09.723758] I [dht-shared.c:311:dht_init_regex] 0-t1-dht: using regex rsync-hash-regex = ^.(.+).[^.]+$
[2014-08-12 14:43:09.725716] I [socket.c:3561:socket_init] 0-t1-client-3: SSL support is NOT enabled
[2014-08-12 14:43:09.725726] I [socket.c:3576:socket_init] 0-t1-client-3: using system polling thread
[2014-08-12 14:43:09.725903] I [socket.c:3561:socket_init] 0-t1-client-2: SSL support is NOT enabled
[2014-08-12 14:43:09.725911] I [socket.c:3576:socket_init] 0-t1-client-2: using system polling thread
[2014-08-12 14:43:09.726112] I [socket.c:3561:socket_init] 0-t1-client-1: SSL support is NOT enabled
[2014-08-12 14:43:09.726119] I [socket.c:3576:socket_init] 0-t1-client-1: using system polling thread
[2014-08-12 14:43:09.726326] I [socket.c:3561:socket_init] 0-t1-client-0: SSL support is NOT enabled
[2014-08-12 14:43:09.726335] I [socket.c:3576:socket_init] 0-t1-client-0: using system polling thread
[2014-08-12 14:43:09.726351] I [client.c:2294:notify] 0-t1-client-0: parent translators are ready, attempting connect on transport
[2014-08-12 14:43:09.729041] I [client.c:2294:notify] 0-t1-client-1: parent translators are ready, attempting connect on transport
[2014-08-12 14:43:09.731439] I [client.c:2294:notify] 0-t1-client-2: parent translators are ready, attempting connect on transport
[2014-08-12 14:43:09.733859] I [client.c:2294:notify] 0-t1-client-3: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume t1-client-0
2: type protocol/client
3: option remote-host gluster1
4: option remote-subvolume /bricks/brick1/brick
5: option transport-type socket
6: option send-gids true
7: end-volume
8:
9: volume t1-client-1
10: type protocol/client
11: option remote-host gluster2
12: option remote-subvolume /bricks/brick1/brick
13: option transport-type socket
14: option send-gids true
15: end-volume
16:
17: volume t1-replicate-0
18: type cluster/replicate
19: subvolumes t1-client-0 t1-client-1
20: end-volume
21:
22: volume t1-client-2
23: type protocol/client
24: option remote-host gluster1
25: option remote-subvolume /bricks/brick2/brick
26: option transport-type socket
27: option send-gids true
28: end-volume
29:
30: volume t1-client-3
31: type protocol/client
32: option remote-host gluster2
33: option remote-subvolume /bricks/brick2/brick
34: option transport-type socket
35: option send-gids true
36: end-volume
37:
38: volume t1-replicate-1
39: type cluster/replicate
40: subvolumes t1-client-2 t1-client-3
41: end-volume
42:
43: volume t1-dht
44: type cluster/distribute
45: subvolumes t1-replicate-0 t1-replicate-1
46: end-volume
47:
48: volume t1-write-behind
49: type performance/write-behind
50: subvolumes t1-dht
51: end-volume
52:
53: volume t1-read-ahead
54: type performance/read-ahead
55: subvolumes t1-write-behind
56: end-volume
57:
58: volume t1-io-cache
59: type performance/io-cache
60: subvolumes t1-read-ahead
61: end-volume
62:
63: volume t1-quick-read
64: type performance/quick-read
65: subvolumes t1-io-cache
66: end-volume
67:
68: volume t1-open-behind
69: type performance/open-behind
70: subvolumes t1-quick-read
71: end-volume
72:
73: volume t1-md-cache
74: type performance/md-cache
75: subvolumes t1-open-behind
76: end-volume
77:
78: volume t1
79: type debug/io-stats
80: option latency-measurement off
81: option count-fop-hits off
82: subvolumes t1-md-cache
83: end-volume
84:
+------------------------------------------------------------------------------+
[2014-08-12 14:43:09.737082] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-t1-client-0: changing port to 49161 (from 0)
[2014-08-12 14:43:09.737125] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-t1-client-1: changing port to 49161 (from 0)
[2014-08-12 14:43:09.742083] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-t1-client-3: changing port to 49162 (from 0)
[2014-08-12 14:43:09.742127] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-t1-client-2: changing port to 49162 (from 0)
[2014-08-12 14:43:09.747365] I [client-handshake.c:1677:select_server_supported_programs] 0-t1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-08-12 14:43:09.747468] I [client-handshake.c:1677:select_server_supported_programs] 0-t1-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-08-12 14:43:09.747525] I [client-handshake.c:1677:select_server_supported_programs] 0-t1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-08-12 14:43:09.747579] I [client-handshake.c:1677:select_server_supported_programs] 0-t1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-08-12 14:43:09.747780] I [client-handshake.c:1462:client_setvolume_cbk] 0-t1-client-0: Connected to 192.168.30.107:49161, attached to remote volume '/bricks/brick1/brick'.
[2014-08-12 14:43:09.747791] I [client-handshake.c:1474:client_setvolume_cbk] 0-t1-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-08-12 14:43:09.747860] I [afr-common.c:4131:afr_notify] 0-t1-replicate-0: Subvolume 't1-client-0' came back up; going online.
[2014-08-12 14:43:09.747900] I [client-handshake.c:1462:client_setvolume_cbk] 0-t1-client-3: Connected to 192.168.30.108:49162, attached to remote volume '/bricks/brick2/brick'.
[2014-08-12 14:43:09.747907] I [client-handshake.c:1474:client_setvolume_cbk] 0-t1-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2014-08-12 14:43:09.747997] I [afr-common.c:4131:afr_notify] 0-t1-replicate-1: Subvolume 't1-client-3' came back up; going online.
[2014-08-12 14:43:09.748017] I [client-handshake.c:1462:client_setvolume_cbk] 0-t1-client-1: Connected to 192.168.30.108:49161, attached to remote volume '/bricks/brick1/brick'.
[2014-08-12 14:43:09.748023] I [client-handshake.c:1474:client_setvolume_cbk] 0-t1-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-08-12 14:43:09.748211] I [client-handshake.c:1462:client_setvolume_cbk] 0-t1-client-2: Connected to 192.168.30.107:49162, attached to remote volume '/bricks/brick2/brick'.
[2014-08-12 14:43:09.748221] I [client-handshake.c:1474:client_setvolume_cbk] 0-t1-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2014-08-12 14:43:09.751376] I [fuse-bridge.c:4977:fuse_graph_setup] 0-fuse: switched to graph 0
[2014-08-12 14:43:09.751414] I [client-handshake.c:450:client_set_lk_version_cbk] 0-t1-client-3: Server lk version = 1
[2014-08-12 14:43:09.751426] I [client-handshake.c:450:client_set_lk_version_cbk] 0-t1-client-1: Server lk version = 1
[2014-08-12 14:43:09.751433] I [client-handshake.c:450:client_set_lk_version_cbk] 0-t1-client-2: Server lk version = 1
[2014-08-12 14:43:09.751441] I [client-handshake.c:450:client_set_lk_version_cbk] 0-t1-client-0: Server lk version = 1
[2014-08-12 14:43:09.751582] I [fuse-bridge.c:3914:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.13
[2014-08-12 14:43:09.753480] I [dht-layout.c:727:dht_layout_dir_mismatch] 0-t1-dht: / - disk layout missing
[2014-08-12 14:43:09.753491] I [dht-common.c:635:dht_revalidate_cbk] 0-t1-dht: mismatching layouts for /
[2014-08-12 14:43:11.652867] I [dht-layout.c:727:dht_layout_dir_mismatch] 0-t1-dht: / - disk layout missing
[2014-08-12 14:43:11.652890] I [dht-common.c:635:dht_revalidate_cbk] 0-t1-dht: mismatching layouts for /
[2014-08-12 14:43:11.656312] I [dht-layout.c:727:dht_layout_dir_mismatch] 0-t1-dht: / - disk layout missing
[2014-08-12 14:43:11.656323] I [dht-common.c:635:dht_revalidate_cbk] 0-t1-dht: mismatching layouts for /
[2014-08-12 14:43:16.220594] I [dht-layout.c:727:dht_layout_dir_mismatch] 0-t1-dht: / - disk layout missing
[2014-08-12 14:43:16.220618] I [dht-common.c:635:dht_revalidate_cbk] 0-t1-dht: mismatching layouts for /
[2014-08-12 14:43:16.223796] I [dht-layout.c:727:dht_layout_dir_mismatch] 0-t1-dht: / - disk layout missing
[2014-08-12 14:43:16.223807] I [dht-common.c:635:dht_revalidate_cbk] 0-t1-dht: mismatching layouts for /
[2014-08-12 14:44:12.133686] I [dht-layout.c:754:dht_layout_dir_mismatch] 0-t1-dht: subvol: t1-replicate-0; inode layout - 0 - 4294967295; disk layout - 0 - 2147483646
[2014-08-12 14:44:12.133728] I [dht-common.c:635:dht_revalidate_cbk] 0-t1-dht: mismatching layouts for /
[2014-08-12 14:44:12.133765] I [dht-layout.c:754:dht_layout_dir_mismatch] 0-t1-dht: subvol: t1-replicate-1; inode layout - 0 - 0; disk layout - 2147483647 - 4294967295
[2014-08-12 14:44:12.133773] I [dht-common.c:635:dht_revalidate_cbk] 0-t1-dht: mismatching layouts for /
[2014-08-12 14:44:12.138224] W [client-rpc-fops.c:2225:client3_3_create_cbk] 0-t1-client-2: remote operation failed: Permission denied. Path: /X
[2014-08-12 14:44:12.138250] W [client-rpc-fops.c:2225:client3_3_create_cbk] 0-t1-client-3: remote operation failed: Permission denied. Path: /X
[2014-08-12 14:44:12.138269] W [fuse-bridge.c:1911:fuse_create_cbk] 0-glusterfs-fuse: 31: /X => -1 (Permission denied)

any tips about exec gluster volume status ?