gluster split brain scenario troubleshooting

01.gluster_splitbrain_notes.md

resolving gfid mismatch

$ gluster vol heal volume_name info

Brick 10.140.99.14:/data/lv_gv0/brick
/docker/registry/v2/repositories/tom-test/toppages/_uploads/d8ec7ebe-3013-4378-98a8-b0f0fa1cf719/startedat
Status: Connected
Number of entries: 1

$ cd /data/lv_gv0/brick

$ cd docker/registry/v2/repositories/tom-test/toppages/_uploads/d8ec7ebe-3013-4378-98a8-b0f0fa1cf719/

$ stat startedat

File: ‘startedat’
 Size: 0           Blocks: 0          IO Block: 4096   regular empty file
Device: fd02h/64770d    Inode: 71728043    Links: 2
Access: (0644/-rw-r--r--)  Uid: (1000030000/ UNKNOWN)   Gid: (    0/    root)
Context: system_u:object_r:unlabeled_t:s0
Access: 2019-02-13 17:31:11.066957244 +0000
Modify: 2019-02-13 17:31:11.066957244 +0000
Change: 2019-07-23 17:03:56.345105626 +0100
Birth: -

$ getfattr -d -m. -ehex startedat
# file: startedat
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.gvol0-client-0=0x000000020000000000000000
trusted.afr.gvol0-client-1=0x000000020000000000000000
trusted.afr.gvol0-client-3=0x000000010000000200000000
trusted.afr.gvol0-client-5=0x000000010000000200000000
trusted.gfid=0x6b5ac7aff9744f84ac337c9f8630ffbb
trusted.gfid2path.bf7896d45c5d1180=0x34656132316335652d643031642d343331622d623833342d3832643361643430613232642f737461727465646174

$ setfattr -x trusted.afr.gvol0-client-0 startedat
$ setfattr -x trusted.afr.gvol0-client-1 startedat
$ setfattr -x trusted.afr.gvol0-client-3 startedat
$ setfattr -x trusted.afr.gvol0-client-5 startedat

repeat the same for other bricks to clear the gfid mismatch

More details can be found here: https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md

gfid_resolver.sh

#!/bin/bash
#  Glusterfs GFID Resolver Turns a GFID into a real path in the brick 

if [[ "$#" < "2" || "$#" > "3" ]]; then
  cat <<END
Glusterfs GFID resolver -- turns a GFID into a real file path
 
Usage: $0 <brick-path> <gfid> [-q]

  <brick-path> : the path to your glusterfs brick (required)
  
  <gfid> : the gfid you wish to resolve to a real path (required)
  
  -q : quieter output (optional)
       with this option only the actual resolved path is printed.
       without this option $0 will print the GFID, 
       whether it identifies a file or directory, and the resolved
       path to the real file or directory.
 
Theory:
The .glusterfs directory in the brick root has files named by GFIDs
If the GFID identifies a directory, then this file is a symlink to the
actual directory.  If the GFID identifies a file then this file is a
hard link to the actual file.
END
exit
fi

BRICK="$1"
 
GFID="$2"
GP1=`cut -c 1-2 <<<"$GFID"`
GP2=`cut -c 3-4 <<<"$GFID"`
GFIDPRE="$BRICK"/.glusterfs/"$GP1"/"$GP2"
GFIDPATH="$GFIDPRE"/"$GFID"
 
if [[ "$#" == "2" ]]; then
  echo -ne "$GFID\t==\t"
fi

 
if [[ -h "$GFIDPATH" ]]; then
  if [[ "$#" == "2" ]]; then
    echo -ne "Directory:\t"
  fi
  DIRPATH="$GFIDPRE"/`readlink "$GFIDPATH"`
  echo $(cd $(dirname "$DIRPATH"); pwd -P)/$(basename "$DIRPATH")
else
  if [[ "$#" == "2" ]]; then
    echo -ne "File:\t"
  fi
  INUM=`ls -i "$GFIDPATH" | cut -f 1 -d \ `  
  find "$BRICK" -inum "$INUM" ! -path \*.glusterfs/\*
fi