Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
gluster split brain scenario troubleshooting

resolving gfid mismatch

$ gluster vol heal volume_name info

Brick 10.140.99.14:/data/lv_gv0/brick
/docker/registry/v2/repositories/tom-test/toppages/_uploads/d8ec7ebe-3013-4378-98a8-b0f0fa1cf719/startedat
Status: Connected
Number of entries: 1

$ cd /data/lv_gv0/brick

$ cd docker/registry/v2/repositories/tom-test/toppages/_uploads/d8ec7ebe-3013-4378-98a8-b0f0fa1cf719/

$ stat startedat

File: ‘startedat’
 Size: 0           Blocks: 0          IO Block: 4096   regular empty file
Device: fd02h/64770d    Inode: 71728043    Links: 2
Access: (0644/-rw-r--r--)  Uid: (1000030000/ UNKNOWN)   Gid: (    0/    root)
Context: system_u:object_r:unlabeled_t:s0
Access: 2019-02-13 17:31:11.066957244 +0000
Modify: 2019-02-13 17:31:11.066957244 +0000
Change: 2019-07-23 17:03:56.345105626 +0100
Birth: -

$ getfattr -d -m. -ehex startedat
# file: startedat
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.gvol0-client-0=0x000000020000000000000000
trusted.afr.gvol0-client-1=0x000000020000000000000000
trusted.afr.gvol0-client-3=0x000000010000000200000000
trusted.afr.gvol0-client-5=0x000000010000000200000000
trusted.gfid=0x6b5ac7aff9744f84ac337c9f8630ffbb
trusted.gfid2path.bf7896d45c5d1180=0x34656132316335652d643031642d343331622d623833342d3832643361643430613232642f737461727465646174

$ setfattr -x trusted.afr.gvol0-client-0 startedat
$ setfattr -x trusted.afr.gvol0-client-1 startedat
$ setfattr -x trusted.afr.gvol0-client-3 startedat
$ setfattr -x trusted.afr.gvol0-client-5 startedat

repeat the same for other bricks to clear the gfid mismatch

More details can be found here: https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md

#!/bin/bash
# Glusterfs GFID Resolver Turns a GFID into a real path in the brick
if [[ "$#" < "2" || "$#" > "3" ]]; then
cat <<END
Glusterfs GFID resolver -- turns a GFID into a real file path
Usage: $0 <brick-path> <gfid> [-q]
<brick-path> : the path to your glusterfs brick (required)
<gfid> : the gfid you wish to resolve to a real path (required)
-q : quieter output (optional)
with this option only the actual resolved path is printed.
without this option $0 will print the GFID,
whether it identifies a file or directory, and the resolved
path to the real file or directory.
Theory:
The .glusterfs directory in the brick root has files named by GFIDs
If the GFID identifies a directory, then this file is a symlink to the
actual directory. If the GFID identifies a file then this file is a
hard link to the actual file.
END
exit
fi
BRICK="$1"
GFID="$2"
GP1=`cut -c 1-2 <<<"$GFID"`
GP2=`cut -c 3-4 <<<"$GFID"`
GFIDPRE="$BRICK"/.glusterfs/"$GP1"/"$GP2"
GFIDPATH="$GFIDPRE"/"$GFID"
if [[ "$#" == "2" ]]; then
echo -ne "$GFID\t==\t"
fi
if [[ -h "$GFIDPATH" ]]; then
if [[ "$#" == "2" ]]; then
echo -ne "Directory:\t"
fi
DIRPATH="$GFIDPRE"/`readlink "$GFIDPATH"`
echo $(cd $(dirname "$DIRPATH"); pwd -P)/$(basename "$DIRPATH")
else
if [[ "$#" == "2" ]]; then
echo -ne "File:\t"
fi
INUM=`ls -i "$GFIDPATH" | cut -f 1 -d \ `
find "$BRICK" -inum "$INUM" ! -path \*.glusterfs/\*
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.