Gluster Volume Checksum Mismatch

GlusterVolumeInfo

[root@ip-172-26-177-115 ~]# gluster volume info
 
Volume Name: supportgfs
Type: Distributed-Replicate
Volume ID: 695f6857-de4a-441f-bbf1-a57ec047eea6
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 172.26.178.4:/media/ephemeral0/supportgfs-readonly
Brick2: 172.26.177.115:/media/ephemeral0/supportgfs-readonly
Brick3: 172.26.178.254:/media/ephemeral0/supportgfs-readonly   # <--- Bad node
Brick4: 172.26.177.116:/media/ephemeral0/supportgfs-readonly
Options Reconfigured:
nfs.disable: on
performance.cache-size: 128MB
auth.allow: 172.26.*

GlusterVolumeStatus

[root@ip-172-26-177-115 ~]# gluster volume status
Status of volume: supportgfs
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick 172.26.178.4:/media/ephemeral0/supportgfs-readonl
y                                                       49152   Y       1913
Brick 172.26.177.115:/media/ephemeral0/supportgfs-reado
nly                                                     49152   Y       1972
Brick 172.26.177.116:/media/ephemeral0/supportgfs-reado
nly                                                     49152   Y       1917
Self-heal Daemon on localhost                           N/A     Y       1986
Self-heal Daemon on 172.26.177.116                      N/A     Y       1936
Self-heal Daemon on 172.26.178.4                        N/A     Y       1932
 
Task Status of Volume supportgfs
------------------------------------------------------------------------------
There are no active volume tasks

LogFromBadNode

[2014-06-17 04:21:11.275495] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: e15a8b47-259c-4bd1-a7df-aafe2ac0f9aa
[2014-06-17 04:21:11.275757] E [glusterd-utils.c:2373:glusterd_compare_friend_volume] 0-management: Cksums of volume supportgfs differ. local cksum = 2201279699, remote cksum = 52468988 on peer 172.26.177.115
[2014-06-17 04:21:11.275958] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 172.26.177.115 (0), ret: 0

LogFromGoodNode

[2014-06-17 04:21:11.266398] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 81857e74-a726-4f48-8d1b-c2a4bdbc094f
[2014-06-17 04:21:11.266485] E [glusterd-utils.c:2373:glusterd_compare_friend_volume] 0-management: Cksums of volume supportgfs differ. local cksum = 52468988, remote cksum = 2201279699 on peer 172.26.178.254
[2014-06-17 04:21:11.266542] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 172.26.178.254 (0), ret: 0
[2014-06-17 04:21:11.272206] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 81857e74-a726-4f48-8d1b-c2a4bdbc094f, host: 172.26.178.254, port: 0

Solution

The solution for this problem was to do the following, on the BAD node:

/etc/init.d/glusterd stop
rsync -havP --delete 172.26.177.115:/var/lib/glusterd/vols/ /var/lib/glusterd/vols/
/etc/init.d/glusterd start

After that, running a `gluster volume status` reported back properly on all nodes.

Thank you.